Project

General

Profile

Actions

Defect #31241

closed

intermittent 500s ActiveRecord::ConnectionTimeoutError

Added by bill auger about 5 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Database
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Resolution:
Invalid
Affected version:

Description

after upgrading from redmine from 3.0-stable to 4.0-stable, i started getting intermittent failures:

ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.005 seconds); all pooled connections were in use):
  app/models/setting.rb:235:in `check_cache'
  app/controllers/application_controller.rb:90:in `user_setup'

using uwsgi/nginx - journalctl shows this error:

SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /issues/227>
uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during GET /issues/>

i tried increasing the number of rails db pool connections and timeout, but that did not seem to help

$ bin/about

sh: hg: command not found
sh: cvs: command not found
sh: bzr: command not found
Environment:
  Redmine version                4.0.2.stable
  Ruby version                   2.6.1-p33 (2019-01-30) [x86_64-linux]
  Rails version                  5.2.2
  Environment                    production
  Database adapter               PostgreSQL
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                sendmail
SCM:
  Subversion                     1.11.1
  Git                            2.21.0
  Filesystem                     
Redmine plugins:
  parabola                       0.0.1

Related issues

Has duplicate Redmine - Defect #31363: 500 Internal Server ErrorClosed

Actions
Actions #1

Updated by bill auger about 5 years ago

it often coincides with the firing of an ansynchronous outgoing mail event with 10 or so recipents

Performing ActionMailer::DeliveryJob (Job ID: N) from Async(mailers) 
Actions #2

Updated by Go MAEDA almost 5 years ago

  • Category set to Database
Actions #3

Updated by pasquale [:dedalus] over 4 years ago

The same issue here.

Environment:
  Redmine version                4.0.5.stable
  Ruby version                   2.5.7-p206 (2019-10-01) [x64-mingw32]
  Rails version                  5.2.3
  Environment                    production
  Database adapter               Mysql2
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                smtp
SCM:
  Mercurial                      5.1.2
  Git                            2.19.1
  Filesystem                     

maybe there is related to #32314?

Actions #4

Updated by Marius BĂLTEANU about 4 years ago

  • Status changed from New to Needs feedback

pasquale [:dedalus] wrote:

The same issue here.

[...]

maybe there is related to #32314?

@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?

Andreas Bill auger, please see the notes from #32314#note-15 and #32314#note-16, you should not use in production the AsyncAdapter. Maybe your server remains without memory because of the mail queue (it's just a supposition).

Actions #5

Updated by Marius BĂLTEANU about 4 years ago

Actions #6

Updated by pasquale [:dedalus] about 4 years ago

Marius BALTEANU wrote:

pasquale [:dedalus] wrote:

The same issue here.

[...]

maybe there is related to #32314?

@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?

No, It works likes a charm!

Actions #7

Updated by Marius BĂLTEANU about 4 years ago

pasquale [:dedalus] wrote:

Marius BALTEANU wrote:

pasquale [:dedalus] wrote:

The same issue here.

[...]

maybe there is related to #32314?

@pasquale, after you changed the Mailer queue to Sidekiq in #32314, did this error reproduce again?

No, It works likes a charm!

Thanks for your quick response.

Andreas Bill auger, please try with another adapter and let us how it works.

Actions #8

Updated by jianmin Zhang about 4 years ago

Hello,
The error dispaly that “all pooled connections were in use”,Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000,The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!

Actions #9

Updated by Marius BĂLTEANU about 4 years ago

  • Status changed from Needs feedback to Closed
  • Resolution set to Invalid

jianmin Zhang wrote:

Hello,
The error dispaly that “all pooled connections were in use”,Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000,The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!

You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).

I'm closing this for now because it seems to be a configuration problem.

Actions #10

Updated by jianmin Zhang about 4 years ago

Marius BALTEANU wrote:

jianmin Zhang wrote:

Hello,
The error dispaly that “all pooled connections were in use”,Is there a concept about connection-pool in Redmine。This Problems arise when the number of users exceeds 1000,The number of people will increase in the future。If NO connection-pool in Redmine,Maybe I can change the maximum number of connections to the database。
Thanks!

You can change your connection pool from database.yml, please see https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html. Before doing this, I would recommend to change your mailer queue adapter if you're using the default one which stores in memory all the send email tasks (more users, more emails to send, more memory needed).

I'm closing this for now because it seems to be a configuration problem.

Thank you for your advice

Actions #11

Updated by Robin Kluth about 4 years ago

That seem to work, I added pool: 10 to the database.yml. So 5 seems a bit too low for redmine? In some cases?

Would be great, if one could change that via the official docker image variables.
For those with docker:

- "./data/database.yml:/usr/src/redmine/config/database.yml"

and put the changes inside the database.yml.

Actions #12

Updated by Elvis Stansvik almost 2 years ago

Hm, is not switching the mail queue adapter or increasing the database connection pool size just workarounds for an issue that should ideally be fixed?

Surely Redmine should not need to keep a bunch of database connections open at the same time just to send out e-mails e.g. when there are many subscribes to an issue, even if the "not for production" mail queue adapter is used? If a database connection needs to be held open while the e-mail is being sent out, then either the same connection should be used or the mails to be sent out should be batched so that they can be sent out with a reasonable number of db conns.

Actions

Also available in: Atom PDF