Daren,
This error 500 problem turned out to be a fairly complicated subject. I have attempted to explain what’s happening below, but it gets a bit wordy and technical. So my apologies if all this reads like mumbo-jumbo.
The important thing is that your posts have brought it to the forefront of our attention. So all the frustration you experienced trying to work around the problem has not just been wasted effort.
So here’s my current understanding of what is causing these error 500 messages when attaching the four images. Please note the exact specifications mentioned below have not been 100% confirmed by the forum software designers or engineers of the hosting company. However, I believe the general concept is on the right track and the numbers quoted are in the ballpark.
What happens when images are attached to a postWhen an image is attached to a post, it is first uploaded and saved on the server’s hard drive. This part of the process has been working OK in our above tests.
Then software on the server processes each image and produces a large ‘thumbnail’ for display underneath the text of the post. It is this thumbnail that we see when we read a post.
The size of this thumbnail can be set by the forum’s administrator (us) and we currently have this set at 900 pixels wide. So what you see when you look at images attached to a post, is the 900 pixel wide thumbnail, not the original image that was uploaded.
There is a link contained within the thumbnail to the original image stored on the server. So if a reader wishes to see or download the original image, they can click on the thumbnail and view the full sized original image in another browser window or download it.
Memory requirements of the serverThe producing of these thumbnails is a memory intensive process as each pixel of the original image has to be examined. So if an image is 1000 pixels wide x 1000 pixels high, then 1,000,000 pixels need to be processed.
As a rough guide to the server memory required to process an image, we can roughly calculate it as follows:
(height in pixels x width in pixels * 4 / 1,000, 000) MB.
So our example image requires (1000 x 1000 x 4 /1,000,000)MB or 4MB.
I say ’roughly’ because it is possible that the exact memory required may vary depending on the content of the pixels, but I haven’t investigated this aspect yet.
Avoiding ‘choked’ serversHosting companies configure their shared servers to limit the amount of memory that can be used by any one process or any one user at a time. This is to avoid situations where one website user running a memory intensive process can choke the server and deny other website users access.
Currently I believe the limit on our server is set to 128MB. This figure was produced by server testing software I ran on our server and has not been confirmed by the hosting engineers.
128MB sounds like sufficient for one process, but with today’s steadily increasing image resolutions, we are pushing these limits more frequently when we attach large photos to our posts.
How our posts tested the server’s limitsApplying the above calculations to the four images we have been testing above, we have:
easterncreek: 317 x 314 x 4 / 1,000,000 = 0.40MB
IMG_2262: 4032 x 3024 x 4 / 1,000,000 = 48.77MB
IMG_2263: 4032 x 3024 x 4 / 1,000,000 = 48.77MB
IMG_2264: 4032 x 3024 x 4 / 1,000,000 = 48.77MB
So to process all four images at once would require approximately 146.71MB of memory. This exceeds our 128MB limit.
To process images 1, 3, 4 as attempted in Test Posts 6 and 8, requires 97.94MB of memory (just to produce the thumbnails) which falls under the 128MB limit. So theoretically, this should have succeeded.
However, I think the actual code itself and/or other processes running at the same time, might have used up the remaining 30MB or so.
Total images stored by our Douglas forumWe have a total of approximately 10,200 images stored on the server of which approximately 6700 were pre-processed (reduced in size) by moderators and approximately 3500 were uploaded as attachments by members.
I made a spreadsheet containing the pixel dimensions of all 3500 attached images on the server. Your three iPhone 6 images we have been working with in this post, are in the top 50 (sized by megapixels) of those 3500 images at 12.19 megapixels each. So when these three are put together, we have a server processing challenge at the very top end of the historical range for our forum.
In the Douglas forum’s early years, we circumvented these sort of problems by having moderators manually produce ‘thumbnail’ images of each member’s original images on their own PC’s at home, before uploading them to the server. This is the 6700 number mentioned above.
This pre-processing avoided all these sort of server related problems and made the forum quite fast, even on a modestly specified server. However, as the number of Members and images increased, it has become too big a job for us to process all Members’ photos each day.
Short Term SolutionsAs a temporary workaround, you could probably continue posting images of this size (4032 x 3024) by adding them one at a time to the same post as I did in Test Post 9.
Or, you could make consecutive posts in the one topic, each having one or two large images attached as you did successfully in your post
1927 600 manifold help.
Or you could select a lower resolution on the phone so that the originals are a bit smaller in the first place.
They will still appear the same size when viewed in a post - ie 900 pixels wide.Or, anytime you experience problems with attaching images, you could just send them to the address shown in
this topic and we will produce the 900 wide ‘thumbnails’ and insert them in your post for you.
This last option is probably the simplest until we get some long term solutions in place.
Long Term SolutionsHowever, as camera image resolutions continue to increase, a long term solution needs to be found so that Members can still attach photos.
The easiest solution is to throw more money at the hosting end and rent a dedicated server where we can set whatever limits we like. This is how commercial websites typically solve these sort of problems. Even though we are a privately funded website, this has been our general approach so far, where we have been increasing server capacity in order to make it as easy as possible for Members to post images straight from their cameras and phones. But the jump to a dedicated server increases hosting costs about five fold – a not insignificant change.
At the other end of the solution scale, we can ask Members to help with the photo processing effort by reducing the size and quantity of images they attach to each post. This is often how non-commercial websites solve the problem.
So a Member would have to either take smaller photos in the first instance, or pre-process images before uploading them. There is plenty of free software and online services available for this latter option, it just requires a bit more time.
So somewhere in that range, we need to find a long term solution that balances Member convenience and long term sustainability of the Douglas Forum. Working this out might take us a while to make sure we consider all the available options and allow for future developments.
Error 500 in response to your uniquely named image filesIn the first posts in this topic, you tried renaming some of the attached files, yet the error 500 continued to appear. To debug this accurately, I would have to be looking over your shoulder while you made the posts.
But broadly what happens is, that if an attempt to upload four large images fails due to the memory issue and an error 500 is generated, those four image files are still successfully stored on the server. They just haven’t been able to be shrunk to the 900 pixel wide thumbnails for display.
So for the next attempt to work, even though it may contain less images,
all files have to be given new names. If that second attempt fails, then
all files attached for the third attempt, have to be given new names again.
This is because the file uploading part of the process is working OK. It is the thumbnail producing part that is causing the problem.
In some tests I noticed that say, one or two thumbnails might be generated successfully and then the error 500 appears. This gives the impression that those one or two files are OK and don’t require new names for the next attempt. However if only the unsuccessful files are renamed and the first one or two left with their original names, then the error 500 is generated because the first files are already on the server.
So in summary, we have two things causing the error 500 message.
1. A memory error as explained under ‘How our posts tested the server’s limits’ above.
2. A file name clash where in a multiple file post, some files have already been successfully uploaded in an earlier attempt. (Or another member has previously uploaded an image with a straight-from-the-camera filename such as IMG_1234.)
Finally, I hope the above shines a little bit of light on the error 500 issues and some short term workarounds. Doug and I will now start working on longer term solutions.
For all Members, if you experience problems attaching images to your posts, please send them to the address shown in
this topic and we will insert them for you.
Dave
Updated details for emailing photos - Dave, 11Feb2021