Thin timing out Nginx cannot connect
Added by Andrew Parker over 9 years ago
I have been trying different configurations and cannot figure out why Thin is timing out. At least this is what I think is happening. Thin is not available after the server is idle for a while (i.e. overnight).
Environment:
Ubuntu 14.04 running on AWS t2.nano
Redmine: Redmine 2.6.10.stable.15251
Ruby: 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]
Rails: 3.2.22.2
Thin: 1.3.1 codename Triple Espresso
Database: mysql
No Thin error log. The error log in Nginx is:
2016/03/24 07:01:47 [error] 18432#0: *1376 connect() to unix:/ebs_001/redmine/run/thin/redmine.0.sock failed (111: Connection refused) while connecting to upstream, client: 184.166.153.12, server: pm.source3.com, request: "GET / HTTP/1.1", upstream: "http://unix:/ebs_001/redmine/run/thin/redmine.0.sock:/", host: "pm.source3.com"
I can restart Thin (more on this later) and connect without restarting Ngnix.
When I try to stop Thin I get the following message
user1@ip-172-31-18-58:/var/log/nginx$ sudo service thin stop
[stop] /etc/thin1.9.1/redmine.yml ...
Stopping server on /ebs_001/redmine/run/thin/redmine.0.sock ...
Sending QUIT signal to process 18385 ...
process not found!
Sending KILL signal to process 18385 ...
/usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:140:in `kill': No such process (Errno::ESRCH)
from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:140:in `force_kill'
from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:134:in `rescue in send_signal'
from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:118:in `send_signal'
from /usr/lib/ruby/vendor_ruby/thin/daemonizing.rb:107:in `kill'
from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:93:in `block in stop'
from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:134:in `tail_log'
from /usr/lib/ruby/vendor_ruby/thin/controllers/controller.rb:92:in `stop'
from /usr/lib/ruby/vendor_ruby/thin/runner.rb:185:in `run_command'
from /usr/lib/ruby/vendor_ruby/thin/runner.rb:151:in `run!'
from /usr/bin/thin:6:in `<main>'
After I stop Thin, I can then start Thin (sudo service Thin start) and connect to my redmine project without restarting nxinx.
I do not see any error logs in redmine or Thin.
My /etc/thin/redmine.yml file:
--- user: user1 group: group1 pid: /ebs_001/redmine/run/thin/redmine.pid timeout: 30 wait: 30 log: /ebs_001/redmine/logs/thin/redmine.log max_conns: 1024 require: [] environment: production max_persistent_conns: 512 servers: 1 daemonize: true socket: /ebs_001/redmine/run/thin/redmine.sock chdir: /ebs_001/redmine/redmine-2.6 tag: redmine
Portions of my /etc/nginx/sites-available/redmine.conf:
# Upstream Ruby process cluster for load balancing
upstream thin_cluster {
server unix:/ebs_001/redmine/run/thin/redmine.0.sock;
# server unix:/ebs_001/redmine/run/thin/redmine.1.sock max_fails=1 fail_timeout=15s;
# server unix:/ebs_001/redmine/run/thin/redmine.2.sock;
# server unix:/ebs_001/redmine/run/thin/redmine.3.sock;
}
### REDMINE - serve all pages via ssl (https)
server {
listen 80;
server_name pm.source3.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name pm.source3.com;
ssl on;
ssl_certificate /etc/nginx/ssl/redmine.crt;
ssl_certificate_key /etc/nginx/ssl/redmine.key;
include /etc/nginx/includes/redmine.include;
proxy_redirect off;
root /ebs_001/redmine/redmine-2.6;
# An alias to your upstream app
location @cluster {
proxy_pass http://thin_cluster;
# Define what a "failure" is, so it can try the next server
proxy_next_upstream error timeout http_502 http_503;
# If the upstream server doesn't respond within n seconds, timeout
proxy_read_timeout 60s;
}
location / {
try_files $uri/index.html $uri.html $uri @cluster;
}
}
...
And the ../includes/redmine.include
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
It is not a permission issue as I can restart Thin and connect to Redmine.
I only have about 15M of free memory. Maybe this is the issue. But if it was, then Redmine would crash as I am using it throughout the day.
Any help on figuring out the timeout is much appreciated.
Replies (1)
RE: Thin timing out Nginx cannot connect
-
Added by Andrew Parker over 9 years ago
The problem was out of memory on the server. I was running to many applications on a AWS t2.nano. Redmine was the only one crashing. No error logs was a hint it was a memory issue. And the "free -h" command was a big clue.
I was running:- One Django app (running Postgres vis AWS RDS)
- WordPress
- Redmine (running MySQL locally).
Migrate to a AWS t2.mircro for more memory and all is fine.