Defect #31241

intermittent 500s ActiveRecord::ConnectionTimeoutError

Added by bill auger over 1 year ago. Updated 4 months ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Database
Target version:-
Resolution:Invalid Affected version:

Description

after upgrading from redmine from 3.0-stable to 4.0-stable, i started getting intermittent failures:

ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.005 seconds); all pooled connections were in use):
  app/models/setting.rb:235:in `check_cache'
  app/controllers/application_controller.rb:90:in `user_setup'

using uwsgi/nginx - journalctl shows this error:

SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /issues/227>
uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /issues/>

i tried increasing the number of rails db pool connections and timeout, but that did not seem to help

$ bin/about

sh: hg: command not found
sh: cvs: command not found
sh: bzr: command not found
Environment:
  Redmine version                4.0.2.stable
  Ruby version                   2.6.1-p33 (2019-01-30) [x86_64-linux]
  Rails version                  5.2.2
  Environment                    production
  Database adapter               PostgreSQL
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                sendmail
SCM:
  Subversion                     1.11.1
  Git                            2.21.0
  Filesystem                     
Redmine plugins:
  parabola                       0.0.1

Related issues

Duplicated by Redmine - Defect #31363: 500 Internal Server Error Closed

History

#1 Updated by bill auger over 1 year ago

it often coincides with the firing of an ansynchronous outgoing mail event with 10 or so recipents

Performing ActionMailer::DeliveryJob (Job ID: N) from Async(mailers) 

#2 Updated by Go MAEDA 12 months ago

  • Category set to Database

#3 Updated by pasquale [:dedalus] 11 months ago

The same issue here.

Environment:
  Redmine version                4.0.5.stable
  Ruby version                   2.5.7-p206 (2019-10-01) [x64-mingw32]
  Rails version                  5.2.3
  Environment                    production
  Database adapter               Mysql2
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                smtp
SCM:
  Mercurial                      5.1.2
  Git                            2.19.1
  Filesystem                     

maybe there is related to #32314?

#4 Updated by Marius BALTEANU 5 months ago

  • Status changed from New to Needs feedback

pasquale [:dedalus] wrote:

The same issue here.

[...]

maybe there is related to #32314?

@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?

@bill auger, please see the notes from #32314#note-15 and #32314#note-16, you should not use in production the AsyncAdapter. Maybe your server remains without memory because of the mail queue (it's just a supposition).

#5 Updated by Marius BALTEANU 5 months ago

#6 Updated by pasquale [:dedalus] 5 months ago

Marius BALTEANU wrote:

pasquale [:dedalus] wrote:

The same issue here.

[...]

maybe there is related to #32314?

@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?

No, It works likes a charm!

#7 Updated by Marius BALTEANU 5 months ago

pasquale [:dedalus] wrote:

Marius BALTEANU wrote:

pasquale [:dedalus] wrote:

The same issue here.

[...]

maybe there is related to #32314?

@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?

No, It works likes a charm!

Thanks for your quick response.

@bill auger, please try with another adapter and let us how it works.

#8 Updated by jianmin Zhang 5 months ago

Hello,
The error dispaly that “all pooled connections were in use”,Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000,The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!

#9 Updated by Marius BALTEANU 5 months ago

  • Status changed from Needs feedback to Closed
  • Resolution set to Invalid

jianmin Zhang wrote:

Hello,
The error dispaly that “all pooled connections were in use”,Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000,The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!

You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).

I'm closing this for now because it seems to be a configuration problem.

#10 Updated by jianmin Zhang 5 months ago

Marius BALTEANU wrote:

jianmin Zhang wrote:

Hello,
The error dispaly that “all pooled connections were in use”,Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000,The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!

You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).

I'm closing this for now because it seems to be a configuration problem.

Thank you for your advice

#11 Updated by Robin Kluth 4 months ago

That seem to work, I added pool: 10 to the database.yml. So 5 seems a bit too low for redmine? In some cases?

Would be great, if one could change that via the official docker image variables.
For those with docker:

- "./data/database.yml:/usr/src/redmine/config/database.yml"

and put the changes inside the database.yml.

Also available in: Atom PDF