massive startup delays

Added by Gareth Evans over 6 years ago

Hi all

We are using redmine under apache2.
Whenever I go to use it and it hasn't been used for a while (like 30 min) it pushes the CPU load to 100% and sits there for a good few mins.
Our issues database is pretty small (maybe 300-400 issues) and all the CPU load seems to be on the ruby process.
I know very little about ruby as I'm a .net dev [sorry :)] but redmine is awesome when it works.
I've tried debugging - reading an article about inspecting ruby with gdb, and once I caught it in a select() but I couldn't trace it any more.
I've turned logging on and the best information I can find is this:

Processing IssuesController#index (for 10.0.0.58 at 2011-04-13 16:26:13) [GET]
Parameters: {"project_id"=>"aamobjob", "action"=>"index", "controller"=>"issues"}
Rendering template within layouts/base
Rendering issues/index.rhtml
Completed in 132043ms (View: 4368, DB: 5187) | 200 OK [http://ssaosvn/projects/aamobjob/issues]

We had SVN integration turned on, but I turned it off thinking this is what was slowing things down.

I've just updated the redmine to the latest version, as well as done a full update on the server, so everything is the latest ver, but the problem persists.
Immediately after I had upgraded redmine to the latest version, it loaded super fast, but a few mins later the issue had returned.

I'm not sure how to debug this issue.
The server is a ubuntu 10.4 LTS, happy to provide more debugging info.
The ssaosvn server is a VM with plenty (i think 1gb) of ram allocated to it, but the host machine is a quad core 3ghz xeon that's not doing much.
Locking the CPU at 100% for 2 mins seems like a LOT of work.
I notice the memory usage by ruby creep up (maybe 1mb every 3-4 secs) while this is happening

Once things start up, redmine is snappy and great, but that first load is really really bad.

Any help you can provide would be appreciated

n.b I didn't set up the redmine install originally

Thanks

Replies (12)

RE: massive startup delays - Added by Gareth Evans over 6 years ago

Rails 2.3.5
Apache/2.2.14 (Ubuntu)
redmine-1.1.2
mysql Ver 14.14 Distrib 5.1.41, for debian-linux-gnu (x86_64) using readline 6.1

root@ssaosvn:/opt/redmine-1.1.2# RAILS_ENV=production script/about
: No such file or directory

I read the FAQ and it looks like it could be the mysql ruby adapter, but I don't know how to update that - shouldn't an aptitude update && aptitude upgrade do it?

RE: massive startup delays - Added by Felix Schäfer over 6 years ago

The svn integration could at most slow down the repository views, and that even not by that much, so you can turn the on without worry.

Gareth Evans wrote:

I read the FAQ and it looks like it could be the mysql ruby adapter, but I don't know how to update that - shouldn't an aptitude update && aptitude upgrade do it?

It depends if it's been installed from apt or from rubygems (the "ruby" dependency and package manager). Have a look at the installed package list in apt and see if and which mysql-ruby or mysql-gem or something like that package has been installed.

While you're at it, have a look if passenger or mod_passenger or however it's called in ubuntu is installed. I guess the problem you are seeing is from the the default process management from the "rails apache"-module. Passenger spins up processes as needed (up to a certain limit), i.e. one process if you only have sequential requests, if two requests come in at the same time it spins up a second one, and so on, and those processes have a time to live of 30 minutes. Those processes make some heavy caching, i.e. once they're spun up, they run quite well, but it takes time to get them up. I think that's this phenomenon you're seeing after more that 30 minutes.

(oh, and getting the heavy and extremely narrow-width artillery like gdb or debugging to try to get behind performance problems is imho or at least in my experience tackling the problem from the wrong end, at least for ruby and especially rails)

RE: massive startup delays - Added by Gareth Evans over 6 years ago

I would say that it's been installed from rubygems, as this:
root@ssaosvn:~# dpkg --get-selections | grep ruby
libopenssl-ruby1.8 install
librack-ruby install
librack-ruby1.8 install
libreadline-ruby1.8 install
libruby1.8 install
ruby install
ruby1.8 install
rubygems install
rubygems1.8 install

shows nothing to do with mysql

yes, passenger is installed: libapache2-mod-passenger

I don't mind a spin-up time but 2-3 mins seems quite extreme

RE: massive startup delays - Added by Gareth Evans over 6 years ago

root@ssaosvn:~# gem list

  • LOCAL GEMS ***

actionmailer (2.3.5)
actionpack (2.3.5)
activerecord (2.3.5)
activeresource (2.3.5)
activesupport (2.3.5)
fastthread (1.0.7)
i18n (0.4.2)
mysql (2.8.1)
rack (1.0.1)
rails (2.3.5)
rake (0.8.7)

RE: massive startup delays - Added by H. Hol. over 6 years ago

I'm experiencing a similar problem. The first two requests after a about 3-5 hours of inactivity take about 30-60sec each. After these, redmine runs smooth.

The mysql-adapter can not be the reason, because I am PostgreSQL. In the beginning I used mod_passenger, after a while I switched to mongrel, thinking it might be passenger-specific. The problem remained, so this might also not be passenger-related.

I'm using ruby1.8 from Debian unstable, a manually installed rubygems 1.3.7 and all gems installed using gem install.

ii  libruby1.8                          1.8.7.334-1                  Libraries necessary to run Ruby 1.8
ii  ruby                                4.5                          An interpreter of object-oriented scripting language Ruby
ii  ruby-dev                            4.5                          Header files for compiling extension modules for Ruby
ii  ruby1.8                             1.8.7.334-1                  Interpreter of object-oriented scripting language Ruby 1.8
ii  ruby1.8-dev                         1.8.7.334-1                  Header files for compiling extension modules for the Ruby 1.8
*** LOCAL GEMS ***

actionmailer (2.3.5)
actionpack (2.3.5)
activerecord (2.3.5)
activeresource (2.3.5)
activesupport (2.3.5)
cgi_multipart_eof_fix (2.5.0)
daemons (1.1.0)
fastthread (1.0.7)
gem_plugin (0.2.3)
i18n (0.5.0, 0.4.2)
mongrel (1.1.5)
pg (0.10.1)
rack (1.0.1)
rails (2.3.5)
rake (0.8.7)

RE: massive startup delays - Added by H. Hol. over 6 years ago

forgot:
  • Redmine 1.0.5.stable.5168 (PostgreSQL)
    and
  • Redmine 1.1.2.stable.5168 (PostgreSQL)

RE: massive startup delays - Added by Felix Schäfer over 6 years ago

Gareth Evans wrote:

I don't mind a spin-up time but 2-3 mins seems quite extreme

That is much too much indeed. I can't say much about fixing this problem, but passenger can be configured to kill/restart processes after a certain number of requests instead of after an idle period, that way the startup penalty would happen only every so often, and you could probably reload the page if you it takes a little while and you would land with another process instead of the one spinning up. I'll try to post an example passenger config, feel free to up the thread if I forget :-)

RE: massive startup delays - Added by Gareth Evans over 6 years ago

Tried that :(
Passenger is up to date:

root@ssaosvn:~# dpkg -l | grep passenger
ii libapache2-mod-passenger 2.2.7debian-1 Rails and Rack support for Apache2

This is my apache site:

RailsEnv production
RailsBaseURI /redmine
PassengerAppRoot /opt/redmine-1.1.2
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
allow from all

RE: massive startup delays - Added by Gareth Evans over 6 years ago

and my passenger.conf:

  PassengerRoot /usr
  PassengerRuby /usr/bin/ruby
  PassengerDefaultUser redmine
  PassengerMaxPoolSize 6
  PassengerMaxRequests 0
  PassengerMinInstances 1
  PassengerPreStart url_to_redmine
  PassengerPoolIdleTime 0

Restarting apache2 gives:

Syntax error on line 7 of /etc/apache2/mods-enabled/passenger.conf:
Invalid command 'PassengerMinInstances', perhaps misspelled or defined by a module not included in the server configuration
.

RE: massive startup delays - Added by Gareth Evans over 6 years ago

I removed PassengerMinInstances and PassengerPreStart and apache starts, but the problem is persisting - though, I think the startup time is not as bad now after I haven't used it for a while

RE: massive startup delays - Added by Ronie Henrich over 6 years ago

I have an issue that seems to be the same as yours. Take a look on my post:
http://www.redmine.org/boards/2/topics/21122

If you do the following steps, can you use Redmine right away?
  1. Kill both Redmine/rails processes (kill -9 <pid #> );
  2. Open Redmine but not click on the Repository tab;
  3. Restart Redmine (touch /var/www/redmine/tmp/restart.txt);
  4. Open again Redmine but not click on the Repository tab;
  5. Now I can open the Repository tab;

(1-12/12)