Author Topic: been trying for 30 mins to post some pics I give up  (Read 8142 times)

0 Members and 1 Guest are viewing this topic.

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
I give up
too many dougli not enough time!

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #1 on: 28 Jun 2018 at 04:37 »
still trying
too many dougli not enough time!

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #2 on: 28 Jun 2018 at 04:39 »
it says file name already used here it is "IMG _2263 azxcfresbyjgyy;l,mnjeh" really I can't believe some random keyboard bashing can have the same file name
too many dougli not enough time!

Offline Doug

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4640
  • Location: Glen Mills, PA, USA
Re: been trying for 30 mins to post some pics I give up
« Reply #3 on: 28 Jun 2018 at 13:47 »
Daren,

Send me the file and I'll see if i cannot get it to upload. I suspect it is something in the file itself that is balking at, then when you try again (even preview?), the temporary name is already allocated on the server.

-Doug

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #4 on: 28 Jun 2018 at 22:19 »
Sorry Doug just venting I managed to upload them eventually in this post https://www.douglasmotorcycles.net/index.php?topic=7061.msg26798;topicseen#new its the fact that it just stalls and does not post I'm on many forus and this is the only one that does this with photo's  please delete the whole post  Daren
too many dougli not enough time!

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #5 on: 01 Jul 2018 at 21:51 »
Daren,

This 'error 500' is a really tough problem to solve as there are many situations that can trigger it. Also, it is generated by the host web server (rather than the forum software) and this is something over which we have little control.

On our forum, it seems to happen when images with certain characteristics are attached to a post. We have had various theories about what these characteristics might be, but often these theories are disproved by a subsequent successful posting of the same image. It may be that the cause is a certain combination of image characteristics, combined with current server conditions (eg memory usage). If this is the case, it could be really, really hard to solve.

Nevertheless, I would like to have a crack at recreating the error with your image. Was it 'IMG_2263' that caused the problem?

Could you email the original version of this image to the address shown in this topic please?

By 'original', I mean one as close as possible to the copy you tried to post the first time the error 500 occurred, without any re-saving or editing by any software. If that copy was straight off your iPhone 6, then the same would be good please.

Dave


Updated details for emailing photos - Dave, 11Feb2021
« Last Edit: 10 Feb 2021 at 20:05 by Dave »

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #6 on: 01 Jul 2018 at 22:51 »
Hi  Dave I was trying to load 4 pics at once all about 4 meg I kept renaming them with random letters it could also be our antiquated telstra copper network and it was raining it may have timed out ? what email address or do you want me to text then to your phone Daren
too many dougli not enough time!

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #7 on: 01 Jul 2018 at 23:00 »
Daren,

"I was trying to load 4 pics at once all about 4 meg..."

That's good info. Maybe we hit a size limit? I will try and duplicate the error.

Could you email all four images to the address shown in this topic please?

If there are any problems attaching all four to the one email, please try sending one image per message.

Thanks for your help, Dave

Updated details for emailing photos - Dave, 11Feb2021
« Last Edit: 10 Feb 2021 at 20:06 by Dave »

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #8 on: 02 Jul 2018 at 22:19 »
done
too many dougli not enough time!

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #9 on: 03 Jul 2018 at 01:05 »
All posts attempted using Firefox v61.0 browser.

Test Post 1 - Failed

This post attempts to attach all four photos from Daren saved with exactly the same names as supplied. The forum software generated this error message which although annoying, is understandable.




Test Post 2 - Failed

This post attempts to attach the same four files but they have been renamed in File Explorer. The images have not been opened yet or modified in any other way.

Result
No error message received. Modem lights flashed for about 90 seconds while images uploaded, then the browser screen went blank (white) and stayed that way.
« Last Edit: 03 Jul 2018 at 02:54 by Dave »

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #10 on: 03 Jul 2018 at 01:14 »
Test Post 3 - Failed

Attempted a re-run of Test Post 2 but received the error message below generated by the forum software.
So the images attached to Test Post 2 were stored in the Attachments directory on the server OK, even though the message itself was not stored.
When the same images with the same names arrived again attached to Test Post 3, the error was generated.


Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #11 on: 03 Jul 2018 at 01:32 »
Test Post 4 - Failed

Attempted to post the first three images with new names (thinking total size of the four images might be an issue). Modem lights indicated files were uploading for about 75 seconds, then the blank (white screen) appeared again.
Hitting the refresh button on the browser produced the error message below. Pressing 'Try Again' produced the same error message.


Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #12 on: 03 Jul 2018 at 01:44 »
Test Post 5 - Successful

Attempted to post images 1 and 4, the smallest two files. Both given new names. Total size of attachments approximately 2MB.
« Last Edit: 03 Jul 2018 at 01:50 by Dave »

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #13 on: 03 Jul 2018 at 01:51 »
Test Post 6 - Failed

Attempted to post images 1, 3 and 4, the smallest three files. All given new names. Total size of attachments approximately 4.5MB.
Screen went blank after uploading images finished. No reply from server.

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #14 on: 03 Jul 2018 at 01:52 »
ye its a pain some times I renamed the lot by adding random letters to the file name and it  kept doing it I thought it was my end timing out ??  Daren
too many dougli not enough time!

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #15 on: 03 Jul 2018 at 02:39 »
Daren,

Looks like the server timing out or choking for some other reason. I'll have to dig around in the server log files and see if there are any clues.
I'd like to try and find a repeatable way to generate that 'Error 500' message too if possible.

Dave

Offline Doug

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4640
  • Location: Glen Mills, PA, USA
Re: been trying for 30 mins to post some pics I give up
« Reply #16 on: 03 Jul 2018 at 14:07 »
Test Post 7 -

All four images were given a unique new name and uploaded via a fiber connection from Eastern U.S.A. to test whether the upload time is causing a server time-out.

Browser: FF v61.0 (64bit)

[add browser info. 03Jul18. -Doug]
« Last Edit: 03 Jul 2018 at 21:02 by Doug »

Offline Doug

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4640
  • Location: Glen Mills, PA, USA
Re: been trying for 30 mins to post some pics I give up
« Reply #17 on: 03 Jul 2018 at 14:09 »
Test post 7 - results.

Connection timed out during the attempt. Window went blank after 11 seconds. Attempting to refresh the connection resulted in the error message that the file(s) name already existed on the server.

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #18 on: 03 Jul 2018 at 20:47 »
Test Post 8 - Failed (but succeeded in generating error 500)

This test uses Google Chrome Version 67.0.3396.99 and is the same as Test 6 but using Chrome instead of Firefox.

Attempted to post images 1, 3 and 4, the smallest three files. All given new names. Total size of attachments approximately 4.5MB.
Chrome produced a more communicative result and showed a % count as the file uploaded. It reached 100% after about 30secs, then there was a short wait of about 10 secs but the screen didn't go blank. Then the error message below appeared. Error 500!


Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #19 on: 03 Jul 2018 at 20:48 »
Darren,

Could you tell me what kind of device you were using to post the images and which browser please?

Dave

Offline Daren W Australia

  • Master Member
  • ****
  • Join Date: Apr 2005
  • Posts: 629
  • Location: Sydney Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #20 on: 03 Jul 2018 at 21:59 »
Hi Windows 10 the computer was bought December last year and google Chrome Thanks Daren
too many dougli not enough time!

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #21 on: 04 Jul 2018 at 18:15 »
Test Post 9 - Success but with an error 500 message

This test uses Google Chrome Version 67.0.3396.99.

Each image was given a new name and attached to the post one image at a time.
So the first image was attached and the post saved.
Then the Modify button was used to open the post, attach the next image and then the post was saved again.
This process was repeated until all four images were attached.
There was an error 500 generated when attaching IMG_2263 but the server still managed to produce the 900 pixel wide 'thumbnail' and display it with the post. The browser's Back button was used to recover from the error 500 screen.

2021-01-30 - Large attachments removed to free up disk space. Result of this test was 1 + 3 attachment(s) posted OK.
« Last Edit: 29 Jan 2021 at 21:10 by Dave »

Offline Dave

  • Administrator
  • ****
  • Join Date: Mar 2004
  • Posts: 4353
  • Location: Australia
Re: been trying for 30 mins to post some pics I give up
« Reply #22 on: 09 Jul 2018 at 08:06 »
Daren,

This error 500 problem turned out to be a fairly complicated subject. I have attempted to explain what’s happening below, but it gets a bit wordy and technical. So my apologies if all this reads like mumbo-jumbo.

The important thing is that your posts have brought it to the forefront of our attention. So all the frustration you experienced trying to work around the problem has not just been wasted effort.

So here’s my current understanding of what is causing these error 500 messages when attaching the four images. Please note the exact specifications mentioned below have not been 100% confirmed by the forum software designers or engineers of the hosting company. However, I believe the general concept is on the right track and the numbers quoted are in the ballpark.

What happens when images are attached to a post

When an image is attached to a post, it is first uploaded and saved on the server’s hard drive. This part of the process has been working OK in our above tests.

Then software on the server processes each image and produces a large ‘thumbnail’ for display underneath the text of the post. It is this thumbnail that we see when we read a post.

The size of this thumbnail can be set by the forum’s administrator (us) and we currently have this set at 900 pixels wide. So what you see when you look at images attached to a post, is the 900 pixel wide thumbnail, not the original image that was uploaded.

There is a link contained within the thumbnail to the original image stored on the server. So if a reader wishes to see or download the original image, they can click on the thumbnail and view the full sized original image in another browser window or download it.

Memory requirements of the server

The producing of these thumbnails is a memory intensive process as each pixel of the original image has to be examined. So if an image is 1000 pixels wide x 1000 pixels high, then 1,000,000 pixels need to be processed.

As a rough guide to the server memory required to process an image, we can roughly calculate it as follows:
(height in pixels  x  width in pixels * 4 / 1,000, 000) MB.

So our example image requires (1000 x 1000 x 4 /1,000,000)MB or 4MB.

I say ’roughly’ because it is possible that the exact memory required may vary depending on the content of the pixels, but I haven’t investigated this aspect yet.                                                                                                                         

Avoiding ‘choked’ servers

Hosting companies configure their shared servers to limit the amount of memory that can be used by any one process or any one user at a time. This is to avoid situations where one website user running a memory intensive process can choke the server and deny other website users access.

Currently I believe the limit on our server is set to 128MB. This figure was produced by server testing software I ran on our server and has not been confirmed by the hosting engineers.

128MB sounds like sufficient for one process, but with today’s steadily increasing image resolutions, we are pushing these limits more frequently when we attach large photos to our posts.

How our posts tested the server’s limits

Applying the above calculations to the four images we have been testing above, we have:
easterncreek:  317 x 314 x 4 / 1,000,000 = 0.40MB
IMG_2262:  4032 x 3024 x 4 / 1,000,000 = 48.77MB
IMG_2263:  4032 x 3024 x 4 / 1,000,000 = 48.77MB
IMG_2264:  4032 x 3024 x 4 / 1,000,000 = 48.77MB

So to process all four images at once would require approximately 146.71MB of memory. This exceeds our 128MB limit.

To process images 1, 3, 4 as attempted in Test Posts 6 and 8, requires 97.94MB of memory (just to produce the thumbnails) which falls under the 128MB limit. So theoretically, this should have succeeded.
However, I think the actual code itself and/or other processes running at the same time, might have used up the remaining 30MB or so.

Total images stored by our Douglas forum

We have a total of approximately 10,200 images stored on the server of which approximately 6700 were pre-processed (reduced in size) by moderators and approximately 3500 were uploaded as attachments by members.

I made a spreadsheet containing the pixel dimensions of all 3500 attached images on the server. Your three iPhone 6 images we have been working with in this post, are in the top 50 (sized by megapixels) of those 3500 images at 12.19 megapixels each. So when these three are put together, we have a server processing challenge at the very top end of the historical range for our forum.

In the Douglas forum’s early years, we circumvented these sort of problems by having moderators manually produce ‘thumbnail’ images of each member’s original images on their own PC’s at home, before uploading them to the server. This is the 6700 number mentioned above.

This pre-processing avoided all these sort of server related problems and made the forum quite fast, even on a modestly specified server. However, as the number of Members and images increased, it has become too big a job for us to process all Members’ photos each day.

Short Term Solutions

As a temporary workaround, you could probably continue posting images of this size (4032 x 3024) by adding them one at a time to the same post as I did in Test Post 9.

Or, you could make consecutive posts in the one topic, each having one or two large images attached as you did successfully in your post 1927 600 manifold help.
 
Or you could select a lower resolution on the phone so that the originals are a bit smaller in the first place. They will still appear the same size when viewed in a post - ie 900 pixels wide.

Or, anytime you experience problems with attaching images, you could just send them to the address shown in this topic and we will produce the 900 wide ‘thumbnails’ and insert them in your post for you.

This last option is probably the simplest until we get some long term solutions in place.

Long Term Solutions

However, as camera image resolutions continue to increase, a long term solution needs to be found so that Members can still attach photos.

The easiest solution is to throw more money at the hosting end and rent a dedicated server where we can set whatever limits we like. This is how commercial websites typically solve these sort of problems.  Even though we are a privately funded website, this has been our general approach so far, where we have been increasing server capacity in order to make it as easy as possible for Members to post images straight from their cameras and phones. But the jump to a dedicated server increases hosting costs about five fold – a not insignificant change.

At the other end of the solution scale, we can ask Members to help with the photo processing effort by reducing the size and quantity of images they attach to each post. This is often how non-commercial websites solve the problem.
So a Member would have to either take smaller photos in the first instance, or pre-process images before uploading them. There is plenty of free software and online services available for this latter option, it just requires a bit more time.

So somewhere in that range, we need to find a long term solution that balances Member convenience and long term sustainability of the Douglas Forum. Working this out might take us a while to make sure we consider all the available options and allow for future developments.             

Error 500 in response to your uniquely named image files

In the first posts in this topic, you tried renaming some of the attached files, yet the error 500 continued to appear. To debug this accurately, I would have to be looking over your shoulder while you made the posts.

But broadly what happens is, that if an attempt to upload four large images fails due to the memory issue and an error 500 is generated, those four image files are still successfully stored on the server. They just haven’t been able to be shrunk to the 900 pixel wide thumbnails for display.

So for the next attempt to work, even though it may contain less images, all files have to be given new names. If that second attempt fails, then all files attached for the third attempt, have to be given new names again.

This is because the file uploading part of the process is working OK. It is the thumbnail producing part that is causing the problem.

In some tests I noticed that say, one or two thumbnails might be generated successfully and then the error 500 appears. This gives the impression that those one or two files are OK and don’t require new names for the next attempt. However if only the unsuccessful files are renamed and the first one or two left with their original names, then the error 500 is generated because the first files are already on the server.

So in summary, we have two things causing the error 500 message.
1. A memory error as explained under ‘How our posts tested the server’s limits’ above.
2. A file name clash where in a multiple file post, some files have already been successfully uploaded in an earlier attempt. (Or another member has previously uploaded an image with a straight-from-the-camera filename such as IMG_1234.)

Finally, I hope the above shines a little bit of light on the error 500 issues and some short term workarounds. Doug and I will now start working on longer term solutions.

For all Members, if you experience problems attaching images to your posts, please send them to the address shown in this topic and we will insert them for you.

Dave

Updated details for emailing photos - Dave, 11Feb2021
« Last Edit: 10 Feb 2021 at 20:09 by Dave »