Defect #3116

Uploading medium size file fails

Added by Thomas Jahns over 8 years ago. Updated over 8 years ago.

Status:ClosedStart date:2009-04-03
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Attachments
Target version:-
Resolution:Fixed Affected version:

Description

Hi,

our software requires ca. 1GiB initialization files, which I put in a tar archive I wanted to upload into the files section of our redmine setup1.

Unfortunately the following happens with my setup of redmine:

1. the file upload completes on the http level (at least the transferred amount of data I see in iftop matches the files size)
2. the system goes into a state of high load and starts swapping, this takes a few minutes
3. sometimes I get an error message (see attached file), sometimes the browser just times out on waitng for www server, there is no error message in the log
4. memory consumption stays high, unless I manually kill the ruby process and restart

[1] The setup is a debian lenny 5.0 i386 system on a P4 2.4GHz with 512MiB of RAM, following the installation of redmine in a users home directory recipe. redmine is started from a daemontools svscan.

redmine-error-message.html Magnifier (652 Bytes) Thomas Jahns, 2009-04-03 15:29

production.log (5.43 KB) Thomas Jahns, 2009-04-09 09:12


Related issues

Related to Redmine - Defect #3145: Upload of bigger files fails iff version is defined Closed 2009-04-09

Associated revisions

Revision 2670
Added by Jean-Philippe Lang over 8 years ago

Fixes memory consumption on file upload (#3116).

History

#1 Updated by Thomas Jahns over 8 years ago

Version is 0.8.2

#2 Updated by Eric Davis over 8 years ago

That is the generic "Something bad happened" error. Can you attach your log files that include the actual error? They are stored in log/production.log

#3 Updated by Thomas Jahns over 8 years ago

Eric Davis wrote:

Can you attach your log files that include the actual error? They are stored in log/production.log

good hint, another file to put into log rotation.

From the log I see that ruby got an out of memory error. From what I read on some other boards, I can see that redmine does some in-memory processing of uploaded files. Is that a sensible approach? Why doesn't the file simply get dumped into a staging directory inside the redmine directory and moved to the files directory when the upload finished successfully?

I hope the backtrace is of some use.

#4 Updated by Jean-Philippe Lang over 8 years ago

  • Resolution set to Fixed

Memory consumption on file upload is fixed in r2670.
Can you confirm please?

#5 Updated by Thomas Jahns over 8 years ago

Jean-Philippe Lang wrote:

Memory consumption on file upload is fixed in r2670.
Can you confirm please?

Only partially. Yes, memory consumption did go down, but was still high and upload again failed on another file. I then tried to search for similar reports and found <http://www.jedi.be/blog/2009/04/10/rails-and-large-large-file-uploads-looking-at-the-alternatives/&gt;. From there I decided to give mongrel a go and stopped webrick and used

mongrel_rails start -e production

to start the redmine server. That did significantly reduce memory footprint.

So I guess your fix does improve memory consumption and decrease the time for md5 computation but running mongrel is a worthwhile recommendation since redmine itself is innocent of most of the memory usage in file uploads.

Greetings, Thomas

#6 Updated by Thomas Jahns over 8 years ago

  • Status changed from New to Resolved

That should have been:

http://www.jedi.be/blog/2009/04/10/rails-and-large-large-file-uploads-looking-at-the-alternatives/

So perhaps parsing <URL> is another issue. ;-)

Greetings, Thomas

#7 Updated by Jean-Philippe Lang over 8 years ago

  • Category set to Attachments
  • Status changed from Resolved to Closed

I use mongrel for development. I didn't even try with webrick but anyone who runs rails applications should know that webrick is not suitable for production purpose :-)

The fix is merged in 0.8-stable branch in r2681.
Thanks for the feedback.

Also available in: Atom PDF