Redmine

Hi, Liang.

This is Mongrel issue, but maybe we could help you a little by locating the problem more precisely. I suppose you have an apache or whatever httpd server in front of your cluster: could you have a look into your httpd server logs: you should be able to discover which access failed. If it's a particular page, maybe it could be an indication of a too demanding content.

Thank you.

Thanks for the tip, Thomas.
I found out that the problem comes from the PDF generation for specific issues. It works on some issues, but not others. Once the error occurs, the Mongrel will consume 100% of the CPU.

Any further ideas?

-Liang

@Processing IssuesController#show (for 66.249.72.198 at 2008-02-12 06:00:27) [GET]
Session ID: d5e4cc350eecdb3ef689a4c84949f079
Parameters: {"format"=>"pdf", "action"=>"show", "id"=>"16", "controller"=>"issues"}
Rendering issues/show.rfpdf

ActionView::TemplateError (Mongrel timed out this thread: shutdown) in app/views/issues/_pdf.rfpdf:

/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:221:in `MultiCell'
    (erb):63:in `render'
    /var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:69:in `render'
    /var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `instance_eval'
    /var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `render'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:419:in `delegate_render'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:299:in `render_template'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:260:in `render_file'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:275:in `render'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/partials.rb:59:in `render_partial'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:30:in `benchmark'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/partials.rb:58:in `render_partial'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:287:in `render'
    (erb):7:in `render'
    /var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:69:in `render'
    /var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `instance_eval'
    /var/rails/redmine/vendor/plugins/rfpdf/lib/rfpdf/view.rb:62:in `render'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:419:in `delegate_render'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:299:in `render_template'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_view/base.rb:260:in `render_file'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:812:in `render_file'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:747:in `render_with_no_layout'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/layout.rb:256:in `render_without_benchmark'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:50:in `render'
    /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:50:in `render'
    /var/rails/redmine/app/controllers/issues_controller.rb:94:in `show'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:135:in `call'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:135:in `custom'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:167:in `call'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:167:in `respond'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:161:in `each'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:161:in `respond'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/mime_responds.rb:105:in `respond_to'
    /var/rails/redmine/app/controllers/issues_controller.rb:92:in `show'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:1101:in `send'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:1101:in `perform_action_without_filters'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/filters.rb:696:in `call_filters'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/filters.rb:688:in `perform_action_without_benchmark'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
    /usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:66:in `perform_action_without_rescue'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/rescue.rb:83:in `perform_action'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:435:in `send'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:435:in `process_without_filters'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/filters.rb:684:in `process_without_session_management_support'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/session_management.rb:114:in `process'
    /var/rails/redmine/vendor/rails/actionpack/lib/action_controller/base.rb:334:in `process'
    /var/rails/redmine/vendor/rails/railties/lib/dispatcher.rb:41:in `dispatch'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:76:in `process'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in `synchronize'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/rails.rb:74:in `process'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:159:in `process_client'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `each'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:158:in `process_client'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `initialize'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `new'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:285:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `initialize'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `new'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel.rb:268:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:282:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in `each'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/configurator.rb:281:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:128:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/lib/mongrel/command.rb:212:in `run'
    /usr/lib/ruby/gems/1.8/gems/mongrel-1.1.3/bin/mongrel_rails:281
    /usr/bin/mongrel_rails:19:in `load'
    /usr/bin/mongrel_rails:19

rfpdf is a nightmare... :(

The view crashed will rendering the content of a cell. According to your name, are you using redmine with a east charset? UTF-8? Could you try to reproduce it on a issue using only ASCII chars, just to be sure that's the charset that provoke the crash.

Thank you.

I have tried with English only chars and it works fine. The problem only occurs with issues with Chinese characters.
However, for some issues with Chinese characters, the PDF is rendered successfully. Although the characters are not correct in the PDF (font?).

It is a strange problem.

Since I know that it is the PDF export that is causing the crash. Would it make sense to add in an option in Redmine to allow or disallow PDF export globally, before the issue is solved? I don't see why many people would like the PDF function (it is nice to have, but not a show stopper).

I have disabled PDF export in the views. However, redmine crashed again with a different issue. It reported out of memory! One process of Mongrel was stuck and consumed 100% of CPU and 800MB of RAM!

This out of memory error only occurs in the repository viewing section:

Errno::ENOMEM (Cannot allocate memory - svn list --xml ...

Hum... This kind of ENOMEM could be the result of a bad process ending: how do you serve your mongrel_cluster? Is it thought lighttpd or other?

I have Nginx in front of mongrel_cluster.

Damn, nginx is not known as a great memory eater :-/

Could you run the svn list --xml and say how many lines it output?

First of all, this is running on a VPS with 1GB of total RAM (that is why I get the out of memory when Mongrel is using around 800MB). the "svn list --xml" command only outputed less than 40 lines, so I don't see this as a problem. However, I cannot find in the log what really caused the memory issue in the redmine instance. Is there any suggested way of debugging such problems? Thanks.

At that point, the best thing to do is to search on the mongrel side. But I find suspicious that your mongrels eat so much RAM: a mongrel eat a lot of it, but yours seem just over-weight! How many user do you have at once and how many mongrels instance have you setup in your cluster?

I have only two instances of mongrel, and this morning, one is stuck at 100%, but the other one is totally fine.

Well, I think this might be still related to the PDF bug. In the production.log file, I did find out that I haven't completely removed the PDF export feature. I am sure that there is no PDF export link on the pages anymore, but I guess some spiders or robots are accessing the links according to their caches. Again, is there a way to completely disable PDF export in RedMine? Thanks.

Here is the error in the log:
ActionView::TemplateError (Mongrel timed out this thread: shutdown) in app/views/issues/_pdf.rfpdf:

I can not help much since I don't use Mongrel.
This site in running Apache2 + fcgid without problems for now :-)

If you want to disable PDF exports, you can comment the 2 lines in app/controllers/issues_controller.rb that start with (lines 66 and 97 in latest revision):

format.pdf { send_data...

Spiders asking for the pdf will get the html version.

Hello,

I think we are experiencing the same problem.

• We are using a mongrel cluster behind apache
• We only use the Issues functionality
• We do not use PDF export (is there any PDF work taking place behind the scenes? I do not think so)
• There are just ~4-5 users

After some time after a restart the application becomes extremely irresponsive, sometimes leading to Apache not being able to wait till the output from mongrels. At this point a mongrel process is eating almost all CPU resources. This is definitely not an SMTP issue, I checked it, and it happens regularly after some time after a restart.

I installed a ruby-prof Rails plugin to see where this is happening, but so far none of these long requests has produced a ruby-prof report - request processing just does not end.

Hi, Yuri

Have you tried the JP's solution to disable PDF exports entirely by modifying app/controllers/issues_controller.rb ? Also, check your Ruby-MySQL version (discussion on the forum: http://rubyforge.org/forum/message.php?msg_id=25951)

-Liang

+1 - I am also seeing this problem on Mongrel Web Server 1.1.5 with a Apache as proxy on Debian etch. I am currently at r1725.

1. The memory footprint hardly ever droops once it raised and if it does very slowls
2. Highlighting a 1MB file will add about 100M for the first time you call it, it slowly goes down 10 or 20 MB. When you call it a second time adds another 50 MB - given the call was a few moments ago so Mongel does not provide it straight ahead form he cache.
3. It seems that re-highlighting a file causes the footprint to go down drastically before it raises again to it's old level.
4. Opening a diferent file does not cause the footprint to lower again.

So what I conclude of this:
The memory leak seems to be somewhere around the part where the code gets highlighted.

i have a similar problem, actually much more severe, which forced me to completely shut down redmine undefinetly.
i'm running a debian etch amd64 system btw.
so redmine has always had problems with memory leaking, killed my server twice until I discovered redmine/mongrel was to blame, so i made a cron to restart mongrel each night.
but at some point things worsened, and latelly memory was growing much faster, and mongrel processes would often hang and eat cpu power. very, very bad.
so i thought, mongrel is to blame, and i switched to passenger. not a good idea after all. after a few hours it eventually took down my whole apache and filled almost all memory but i was lucky enough to kill the stuff before losing the server again.
so redmine is really a no go for now :( am pondering to migrate the data back to mantis... somehow...
anyway, what i noticed with the process hanging was that it hangs when looking at the repositories. i have svn, btw. and search engines were looking at them, as the logs say.
so i modified robots.txt but engines were too slow to update. so i disabled the repository in redmine. but guess what: google's cache feature managed to fuck up redmine anyway, requesting things which should have been disabled.
btw sometime when trying to fix problems, i updated from 0.7.3 to svn, but i fear this has done more harm than good...
i'm desperate. what can i do?

oh, with filling memory i meant: my machine has 2 gb ram + 2 gb swap, and redmine takes about 500 MB when in "normal" operation, then grows, and grows, and grows...

The interesting point should be to find out what activity redmine was accomplishing: repository browsering? Fetching changeset? You should have a look into your logs to determine that, eventually turn log level to DEBUG.

When I ran Redmine with mongrel I used monit to watch my mongrels and restart them if they started to take too much RAM or CPU. It ran great and my two mongrels never went over 90MB of RAM each on my 1GB VPS server. If you want my monit config for Redmine, it's posted to gist.

(I've since switched to Passenger and it restarts processes as they are not used so monit isn't needed)

We get hit by this bug too very frequently. Commenting the format.pdf lines out has helped but still someone should fix this. Please ;)

I wanted to add another experiences about this issue and perhaps it might help you to solve. I've experienced same issue with both of passengers and mongrel. I had got an shared server and everything was fine. Later, i upgraded my server to VPS and it had 150 MB of RAM. And that was the point. The memory issue arose when i tried to access my svn repository on redmine. It wasn't matter which one of passenger or mongrel. Both of them had the same insufficient memory issue.

Anyway, later, i bumped up my RAM to 256 MB and now everything was fine. I didn't get an error about it. I saw that mongrel or passengers has been using the all of the memory when i had checked the my servers memory status. Perhasp, I'm not sure this issue is about redmine or passenger or mongrel. But i think, redmine may need a RAM usage improvements.

I've noticed this problem as-well - it seems to be repository related alright, but I haven't been able to consistently reproduce the problem.

What I can say is - these htaccess rules have significantly increased the time between issues:

RewriteCond %{HTTP_USER_AGENT} ^(msnbot|sogou|baiduspider) [NC]
RewriteRule .* - [F,L]

All three of those spiders had been indexing every revision of every file (and with a few k files in trunk .. 10 or so tags ... 4.5k revisions = 150 million or so total repository requests...)

All three of those spiders also seemed to have no idea what rate limiting is...

Also - we're using apache proxying to a thin cluster (so no passenger memory leaks) and we've kept fairly up to date with trunk since the issue began a few months back...

Hope this helps!