Project

General

Profile

Redmine freezes often, finally getting errors in log

Added by John H over 9 years ago

Hi all,

I've been running Redmine 2.5.1 on a Tomcat7.0 webserver for 3 months now, and it freezes up about 4-5 times a day due to 100% CPU load, causing us to have to restart the whole Tomcat server each time.

Up to this point we have not received any logs in either the Tomcat or Redmine log directories, however we recently got some, and hopefully they will help point to why Redmine does not run on Tomcat.

Some info:
Redmine 2.5.1
MySql6.0 database
jruby 1.7.13
ruby 1.9.3p545

Errors:
---------------------------------------------------------------------------------------------------------------
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/redmine] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/redmine] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/redmine] appears to have started a thread named [JRubyWorker-1] but has failed to stop it. This is very likely to create a memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.jruby.util.Sprintf$1] (value [org.jruby.util.Sprintf$1@bc3aed]) and a value of type [java.util.HashMap] (value [{en_US=java.text.DecimalFormat@674dc}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.jruby.util.Sprintf$2] (value [org.jruby.util.Sprintf$2@1c0becf]) and a value of type [java.util.HashMap] (value [{en_US=java.text.DecimalFormatSymbols@1073a}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.joni.StackMachine$1] (value [org.joni.StackMachine$1@12f0afc]) and a value of type [java.lang.ref.WeakReference] (value [java.lang.ref.WeakReference@17a3f9e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.jruby.util.Sprintf$1] (value [org.jruby.util.Sprintf$1@bc3aed]) and a value of type [java.util.HashMap] (value [{en_US=java.text.DecimalFormat@674dc}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.jruby.util.Sprintf$2] (value [org.jruby.util.Sprintf$2@1c0becf]) and a value of type [java.util.HashMap] (value [{en_US=java.text.DecimalFormatSymbols@1073a}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.joni.StackMachine$1] (value [org.joni.StackMachine$1@12f0afc]) and a value of type [java.lang.ref.WeakReference] (value [java.lang.ref.WeakReference@e6a1e7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@18f7b85]) and a value of type [jnr.ffi.provider.jffi.TransientNativeMemory.Magazine] (value [jnr.ffi.provider.jffi.TransientNativeMemory$Magazine@f018cf]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.jruby.util.Sprintf$1] (value [org.jruby.util.Sprintf$1@bc3aed]) and a value of type [java.util.HashMap] (value [{en_US=java.text.DecimalFormat@674dc}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.jruby.util.Sprintf$2] (value [org.jruby.util.Sprintf$2@1c0becf]) and a value of type [java.util.HashMap] (value [{en_US=java.text.DecimalFormatSymbols@1073a}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/redmine] created a ThreadLocal with key of type [org.joni.StackMachine$1] (value [org.joni.StackMachine$1@12f0afc]) and a value of type [java.lang.ref.WeakReference] (value [java.lang.ref.WeakReference@1c5c58d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Oct 14, 2014 1:55:26 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
-------------------------------------------------------------------------------------------------------------------------
C:/Tomcat 7.0/webapps/redmine/WEB-INF/lib/redmine.rb:70 warning: already initialized constant FCSV
java.lang.NullPointerException

Please try help me solve this issue of Redmine causing Tomcat to hit 100% CPU load and freeze. We have had to endure this for 3 months and we would not like to have to switch to a different issue tracking software.


Replies (7)

RE: Redmine freezes often, finally getting errors in log - Added by Martin Denizet (redmine.org team member) over 9 years ago

Hello John,

Do you get an entry in the Redmine log itself? (logs/production.log)

Cheers,

RE: Redmine freezes often, finally getting errors in log - Added by Pavel Potcheptsov over 9 years ago

It looks like memory leak.
The same was happening with Mongrel.
Try to switch to Unicorn, Puma or Thin. I've never seen troubles with them.

RE: Redmine freezes often, finally getting errors in log - Added by John H over 9 years ago

Hi Martin,

No, there are no logs in the redmine webapps folder. Redmine is being deployed via a warble and being run on a Tomcat7 server, and I believe that has something to do with the fact that we have never received any logs in the redmine webapps directory.

We have had the 100%CPU load/freeze issue occur every single day for the past 3 months, and I have attempted multiple fresh installs. I have come to the conclusion that Redmine and Tomcat don't like each other very much. However, now that I have some logs to work with, hopefully someone can provide some insight to why Redmine works great until it hangs up and kills the server.

RE: Redmine freezes often, finally getting errors in log - Added by Tomasz B over 9 years ago

Hi John,
today I tried to set up Redmine (2.5.2) on Tomcat and of course I had the same problem as you. I started redmine and my Tomcat freeze with communique "redmine.rb:70 warning: already initialized constant FCSV". So I checked what is in redmine.rb and I found:

if RUBY_VERSION < '1.9'
require 'fastercsv'
else
require 'csv'
FCSV = CSV
end

We have version ruby greater than 1.9. So you have two options:
1. (I think this way it better, but I didn't try this) Downgrade your ruby to version below 1.9.
2. (Easy, unprofessional way, but it worked for me) Change RUBY_VERSION < '1.9' to RUBY_VERSION < '2.0'

I'm fresh redmine user and I didn't worked with Ruby. But it may help you until someone smarter will give us a better solution.

RE: Redmine freezes often, finally getting errors in log - Added by John H over 9 years ago

Hi Tomasz,

Thank you very much for that information! I will try your suggestions tomorrow and reply back with the results.

Thanks again for the help.

--JohnH

RE: Redmine freezes often, finally getting errors in log - Added by John H over 9 years ago

I just wanted to update that I did try your 2nd suggestion (changing <1.9 to <2.0), and although that does seem to help it run faster, we have still had Redmine pull 100% cpu load and freeze at least 2x today.

For some reason, when going to a new screen, or when adding an issue, redmine pulls huge cpu loads (85%+). We are running Redmine on a enterprise level server, one in which we could have 1000s of users hit simultaneously. None of our other code running through tomcat pulls even remotely close to 85% (more like <10%), even at our highest loads. I have no idea what Redmine is doing to require such processing power, but it obviously should not be doing so; if only Redmine would leave me some logs :/

RE: Redmine freezes often, finally getting errors in log - Added by John H over 9 years ago

Attached is a screen shot of what the server's performance looks like when switching url's in Redmine.... You cannot even switch between pages, let alone use the functions of Redmine.

As you can see, on my last switch (to redmine/activity), Redmine pulled 100% CPU load and froze. I had to restart Tomcat just because I attempted to view the activity page. Pretty crappy stuff... Might just have to move to a new issue tracker

redmine.png (16.8 KB) redmine.png 100% CPU Load
    (1-7/7)