Redmine (ruby?) is slow despite all efforts

Added by Andrew Winov almost 4 years ago

Hello!

I'm using redmine on quite old hardware (~1GHz CPU, 512Mb ram). My problem is that redmine runs too slow.
I do google about ruby/redmine performance, but still have to wait few seconds before redmine generate a page.
My OS is ubuntu 10.04 server.

Here is my list of performance tunings:
  • nginx as a frontend, it serves static data (PNG, JS, CSS, etc)
  • thin (1.8)
  • postgresql (9.0.2)
  • nginx-thin and thin-postgres comms via unix sockets
  • redmine internal caching turned on
I have also tried (but fail to improve anything):
  • phusion passenger and mongrel, instead of thin
  • ruby 1.9.1 (yes, I got problems with encoding, but NO performance wins)
  • nginx caching (useless, because most pages served only once)

Still, I have these lines in production.log:

Processing WelcomeController#index (for 127.0.0.1 at 2011-01-27 20:23:55) [GET]
  Parameters: {"action"=>"index", "controller"=>"welcome"}
Rendering template within layouts/base
Rendering welcome/index
Completed in 557ms (View: 423, DB: 99) | 200 OK [http://localhost:/]

In case of cache hits, times are better, but still slow:

Processing WelcomeController#index (for 127.0.0.1 at 2011-01-27 20:26:56) [GET]
  Parameters: {"action"=>"index", "controller"=>"welcome"}
Rendering template within layouts/base
Rendering welcome/index
Completed in 138ms (View: 86, DB: 18) | 200 OK [http://localhost:/]

Please advise is it possible to achieve reasonable performance on ruby, without "brute force", i.e. buying 4GHz CPU?

PS. I've searched this forums, but found nothing useful for my case.

Replies (9)

RE: Redmine (ruby?) is slow despite all efforts - Added by Felix Schäfer almost 4 years ago

Mmh, other than not to spin up more than 2 workers because of the little memory you have, I don't think I could give you good advice. Judging from your log, you are in production mode, so that's not the issue, it might really just be your processor or your hdd being too slow. If you had more memory, I'd say try using memcached, but that wouldn't change things much.

RE: Redmine (ruby?) is slow despite all efforts - Added by Andrew Winov almost 4 years ago

Felix, thank you for reply! So din't miss something important in my configs, and it just general slowness of ruby. Sad, it even worse than java, but redmine is best project manager…
Well, maybe it is possible to "compile" redmine code (like ruby gem)? Or new ruby vm (ruby 1.9+, rubinius) can help?

PS. Can someone post performance numbers of redmine running on VPS with similar virtual hw config.

RE: Redmine (ruby?) is slow despite all efforts - Added by Felix Schäfer almost 4 years ago

Andrew Winov wrote:

Felix, thank you for reply! So din't miss something important in my configs, and it just general slowness of ruby. Sad, it even worse than java, but redmine is best project manager…

I didn't say Ruby was the culprit ;-) But if the hardware is old, as you stated, it probably also has slow memory, slow hard drive, slow … and so on (slow by today's standards, that is), that all affect the performance a little, and all those littles tend so sum up.

Well, maybe it is possible to "compile" redmine code (like ruby gem)?

No. Gems don't have "compiled ruby code" either, some have ruby C extensions that need to be compiled, but it is C code bridged to ruby code, and only the C part is compiled.

Or new ruby vm (ruby 1.9+, rubinius) can help?

You could try Ruby EE and passenger, both are made by the same people and REE has some GC and memory improvements that reduce the memory footprint of passenger processes by up to a third (and REE is free, despite the name ;-) ). I don't know how much you had tweaked your passenger either, but there can be dramatic speed differences between the different options at your disposition.

PS. Can someone post performance numbers of redmine running on VPS with similar virtual hw config.

That won't help that much: a VM on modern hardware generally has fast memory and hard disk, most times even faster than current mainstream desktops.

I just had a look again at your numbers, and they are not that bad, here's what I get on decent-ish hardware with passenger on MRI-1.8:

Completed in 245ms (View: 94, DB: 145)
Completed in 156ms (View: 148, DB: 1)
Completed in 58ms (View: 49, DB: 1)

That's on the welcome page too, and looking at your second figure, ~130-150ms isn't bad at all once the different fragment caches and so on kick in. Even the first hit doesn't take much more than half a second to get out of redmine, so if you see load times of "a few" seconds, your problem is elsewhere. Do you maybe have reverse lookups on that could take that much time, maybe it's just the browser being slow rendering stuff, is it the page itself or all the small graphics that take their time being loaded, …?

RE: Redmine (ruby?) is slow despite all efforts - Added by Andrew Winov almost 4 years ago

damn redmine killed my reply as "spam" :(

Felix Schäfer wrote:

I didn't say Ruby was the culprit ;-) But if the hardware is old, as you stated, it probably also has slow memory, slow hard drive, slow … and so on (slow by today's standards, that is)

No, my hardware is OK. The bottleneck is cpu-to-memory bandwidth, and it seems that Ruby is memory and memory operations hungry. See below.

Or new ruby vm (ruby 1.9+, rubinius) can help?

You could try Ruby EE and passenger

OK, I'll try again.

I just had a look again at your numbers, and they are not that bad, here's what I get on decent-ish hardware with passenger on MRI-1.8:

[...]

That's on the welcome page too, and looking at your second figure, ~130-150ms isn't bad at all

Well, I wonder your "decent" hw specs :) 150ms is still slow for my taste, but let's do some calc.

Guys on modrails achive ~350 requests per second, it is about 3ms for single request, on Intel Core2 T5300.
Scaling down to single core and 1ghz, we'll get ~10ms.
Mem band for my chipset is about 1Gbyte/s, T5300's… google says ~5Gbytes/s, so 50ms
I don't know how to assess cpu architecture differences (caches, cores, etc), though they hardly excess 2-3 times — 100..150ms
This is comparable with my cached result.

RE: Redmine (ruby?) is slow despite all efforts - Added by Felix Schäfer almost 4 years ago

Andrew Winov wrote:

No, my hardware is OK. The bottleneck is cpu-to-memory bandwidth, and it seems that Ruby is memory and memory operations hungry. See below.

Ruby is indeed. As I said, REE+passenger is somewhat better with memory, might help in that case.

Well, I wonder your "decent" hw specs :)

Decent in that case is a not-so-recent (1-1,5 years old maybe?) quad-core+ht with enough memory, though I don't have enough passenger instances to have to worry about it being maxed out. I have another box with memcache and other tricks that's in the 40-ish ms range for the welcome page.

150ms is still slow for my taste, but let's do some calc.

Well, put a reverse proxy or whatnot in fron then ;-) You'll agree with me though that if you experience in your browser a "few seconds" whereas the rails logs tell you it's taken only 150ms to churn out the whole thing, that the most time is spent somewhere else than in rails. And regarding 150ms being slow: well, redmine's a complex app :-)

Guys on modrails achive ~350 requests per second, it is about 3ms for single request, on Intel Core2 T5300.

Let's not rush it here: Which app, and how many parallel processes? The measurement you have in your logs is how much time the one process took for one request, the measurement you have here is how many requests the whole thing can serve per second, which is a totally different metric. If I have 700 processes answering requests, but each takes 2 seconds, I'll average at 350 req/s too.

Scaling down to single core and 1ghz, we'll get ~10ms.
Mem band for my chipset is about 1Gbyte/s, T5300's… google says ~5Gbytes/s, so 50ms
I don't know how to assess cpu architecture differences (caches, cores, etc), though they hardly excess 2-3 times — 100..150ms
This is comparable with my cached result.

Well, you seem to come to the same conclusion as I: 150ms isn't that bad on your hardware ;-)

RE: Redmine (ruby?) is slow despite all efforts - Added by Andrew Winov almost 4 years ago

Felix Schäfer wrote:

As I said, REE+passenger is somewhat better with memory, might help in that case.

Yes! I've finally set up passenger as nginx module, and welcome page load times went down from 140ms to ~95ms. Almost 33% better! Thanks for the tip, Felix!

However, that setup was tricky. Passenger is messy about serving static data, as well as logs. So I have to proxy it. Lucky, nginx is the powerful tool.
Also, passenger module runs with root privileges. Potentially huge security hole.

Regarding memory usage, passenger appears to be more mem hungry (it spawns 3 extra processes), but at the end I did not found any measurable increase of memory usage on my server.

So, redmine is almost get rid of its slowness. And we have ruby 1.9 and rbx in reserve :)

I have another box with memcache and other tricks that's in the 40-ish ms range for the welcome page.

Well, I would love to hear that tricks :)

RE: Redmine (ruby?) is slow despite all efforts - Added by Felix Schäfer almost 4 years ago

Andrew Winov wrote:

However, that setup was tricky. Passenger is messy about serving static data,

Can't follow: static data is in the public directory in rails apps, the root URL of your app should serve those, it does take getting used to if coming from other web platforms (php *cough*), but nothing complicated or not documented.

as well as logs.

Same remark as above, those go in the log directory in rails apps, and that's handled by rails, not Redmine or Passenger per se.

So I have to proxy it. Lucky, nginx is the powerful tool.

I don't know what the logs have to do with a proxy, and why you would need to proxy anything if your setup is correct, but if it works, whatever.

Also, passenger module runs with root privileges. Potentially huge security hole.

False, and by a large margin. The passenger module needs to run one process as root if you have user switching enabled, if you don't care about that, you can turn it off and no passenger process should need to run as root. Hell, you can even run passenger as a standalone app (it has a lightweight nginx embedded IIRC) as whatever user of your choosing and reverse proxy to it.

Anyway, everything from the top down to here make me think you've either not read the passenger docs (which I have a hard time believing if you are savvy enough to handle nginx, concurrenty rubies and so on) or not read them attentively enough. I especially recommend this section regarding user switching: http://modrails.com/documentation/Users%20guide%20Nginx.html#user_switching

Regarding memory usage, passenger appears to be more mem hungry (it spawns 3 extra processes), but at the end I did not found any measurable increase of memory usage on my server.

Yes and no. The Passenger module manages a pool of rails processes to be able to accept and process concurrent connections, those can be tweaked in the config. You will want to read this passage of the already linked config: http://modrails.com/documentation/Users%20guide%20Nginx.html#_resource_control_and_optimization_options

I have another box with memcache and other tricks that's in the 40-ish ms range for the welcome page.

Well, I would love to hear that tricks :)

Ok, in no particular order, and I haven't measured which has how many impact on what part of the performance: the passenger config parts:

<IfDefine PASSENGER>
LoadModule passenger_module modules/mod_passenger.so

# "default" config
PassengerRoot /usr
PassengerLogLevel 0
PassengerRuby /usr/bin/ruby18
RailsAutoDetect On
PassengerMaxPoolSize 20
PassengerPoolIdleTime 0
PassengerMaxInstancesPerApp 0
PassengerUserSwitching On
PassengerDefaultUser apache

# Passenger "ricing" 
PassengerMaxRequests 3000
PassengerHighPerformance on
RailsSpawnMethod smart
RailsFrameworkSpawnerIdleTime 0
RailsAppSpawnerIdleTime 0
PassengerUseGlobalQueue on
</IfDefine>

I'll let you read up what does what exactly, but IIRC the RailsSpawnMethod smart allows multiple Redmine processes to share the same in-memory rails bits and even the same caches of the app files in memory and probably is the one option that saves memory the most.

The important apache config stuff:

PassengerMinInstances 3

# helpfull for jammit
<Location "/assets">
    Options MultiViews
</Location>

# Let rails handle etags/file deprecation/cache busting
FileETag None

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript

ExpiresActive On
<FilesMatch "\.(ico|gif|jpe?g|png|js|css|htc|swf)(\?\d+)?$">
    ExpiresDefault "access plus 1 year" 
</FilesMatch>

Ok, those were the parts I can't point you to a repo for, the rest is in my "deployment" repo on fachschaften.org. The version deployed is in the fsenorg branch, the important parts for performance are the memcache branch and the jammit branch. The former leverages memcache for session and controller cache and will show in the redmine logs, the later is a css and js minifier and might improve the browsing experience, but won't better the numbers in the redmine logs as it doesn't do anything there.

RE: Redmine (ruby?) is slow despite all efforts - Added by Andrew Winov almost 4 years ago

Felix Schäfer wrote:

Can't follow: static data is in the public directory in rails apps, the root URL of your app should serve those, it does take getting used to if coming from other web platforms (php *cough*), but nothing complicated or not documented.
Same remark as above, those go in the log directory in rails apps, and that's handled by rails, not Redmine or Passenger per se.

OK, I want to log only accesses to redmine-generated pages, and all static files should be served silently:

    location / {
        root /opt/redmine/public;
        try_files $uri @ruby;
        expires 30d;
        add_header Cache-Control public;
        access_log off;
    }

    location @ruby {
        proxy_pass h_t_t_p://unix:/tmp/phusion.socket;
        proxy_hide_header X-Runtime;
    }

Where unix:/tmp/phusion.socket - is separate nginx virtual server, solely for passenger.

Also, passenger module runs with root privileges. Potentially huge security hole.

False, and by a large margin. …

Anyway, everything from the top down to here make me think you've either not read the passenger docs (which I have a hard time believing if you are savvy enough to handle nginx, concurrenty rubies and so on) or not read them attentively enough. I especially recommend this section regarding user switching: …

It turns out that I've used passenger 2.2.14 (probably obsolete). Just upgraded to 3.0.2, and its do the job by default - no more ruby code under root. This upgrade doesn't improve perf alot, despite devs claim about v3 "up to 55% faster" vs v2 in nginx case. I got only ~2ms, and this might be just error margin.
And yes, I've missed user switching since I was fixed on performance :)

BTW, I forgot to mention my successful move to REE, in addition to passenger.

Well, I would love to hear that tricks :)

Ok, in no particular order, and I haven't measured which has how many impact on what part of the performance: the passenger config parts:

Ah, thanks for tips. I shall check them in final rewiew of my nginx config

Ok, those were the parts I can't point you to a repo for, the rest is in my "deployment" "repo on fachschaften.org"…

I'll try your code tomorrow, but I'm sure it should be merged into main redmine repo ASAP. The faster, the better!

PS. I have very hard time to make this post due to "Your submission was rejected because it appeared to be spam."

RE: Redmine (ruby?) is slow despite all efforts - Added by Felix Schäfer almost 4 years ago

Andrew Winov wrote:

OK, I want to log only accesses to redmine-generated pages, and all static files should be served silently:

Ah, that's nginx logs, not passenger logs, forget what I said :-)

It turns out that I've used passenger 2.2.14 (probably obsolete). Just upgraded to 3.0.2, and its do the job by default - no more ruby code under root. This upgrade doesn't improve perf alot, despite devs claim about v3 "up to 55% faster" vs v2 in nginx case. I got only ~2ms, and this might be just error margin.

Passenger 2 isn't new, but not that old yet either ;-) Anyway, the user switching was already part of passenger 2, so it could have worked there too.

Ah, thanks for tips. I shall check them in final rewiew of my nginx config

Glad I could help.

I'll try your code tomorrow, but I'm sure it should be merged into main redmine repo ASAP. The faster, the better!

Well, the jammit part is quite custom and adds a dependency, the memcache config also adds a dependency and all the config is at the rails level, not Redmine, I think if you need memcached, you can also read about it and make those config entries yourself ;-) All in all, I don't see much of it, if any, to the core.

PS. I have very hard time to make this post due to "Your submission was rejected because it appeared to be spam."

I have absolutely no clue whatsoever what or how this spam-swarting works, it's not part of the redmine core, and I have no way to look at the code. In other words: as you seem to have hit that several times already, open a new bug about it and assign it to JPLang.

(1-9/9)