Defect #3295
closedMysql database getting crashed(Internal Error)
0%
Description
Dear All,
First, I started using redmine v 0.8.0 since last 4 months. Most of the time i received Internal error whenever i click any project after login to redmine. Sometimes, If i try click again on the specific project,then, it works fine. But, Its happening every day. Then, i read the redmine forum, Mr. Jean-Philippe Lang suggest to update settings.yml file under redmine directory. So last week, i upgraded with latest redmine v0.8.3 and everything was goes fine yesterday, but all of the sudden I get an internal error everytime whenever i click on any specific projects. On the background, i received INNODB DB error from mysql logs as summarized below,
InnoDB: Error: page n:o stored in the page read in is 2753267632, should be 1106!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 1106.
InnoDB: You may have to recover from a backup.
090505 9:14:49 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex f691ce33a41b87b0964eb1e0f80d0ab641b7dadef7daa7f1f98c36adbabf1e4d5ee4f48ae4e181cee561dc5b992024398dd6df19363faba7d1ac2e12d442a7
207f4240692729c5afe1e9bb08c944b2a0eb4f354aadb42178ee651f07b29939753f9d5a2ecbd76be96965e16dc76a1
len 16384; hex f691ce33a41b87b0964eb1e0f80d0ab641b7dadef7daa7f1f98c36adbabf1e4d5ee4f48ae4e181cee561dc5b992024398dd6df19363faba7d1ac2e12d442a7
% L 84U - J V A [ y bQ G * M E k PF , 9 k R a ,$ E 9 X f v [~ ?F c: _" w n 7
J\GLi zb \' v ! e : A q {O2R N H d o0Hos0) 1 L t+ s$io | cd 0C fM l J$ 3 % U , KZo wif w &: K
+ &k FR* H 4C ,]<a# $D ) j `? ?Q Gqq = ] : d k F C 3~ T _ % \g& b zv
c 5 $v % p^ U T * ]$ - { $ G: l Z 58AV NM.\ ? 4 z 'UO c .`0 r{6L k . O7 b X c 5 U [
yp0 H R x 1 &3 R$ X ;H H R q ;O * [{$ ? e G | K ls - a t(R* :1e E i 2R " * S Aro \ 7u
w [Q t 3 J F A D r b o j B ?H o?* * X*R *6 -# 7 :NQ )@< w 2 ' X m ) 1 +Jv*e W$P z :
O<6I E X x%x + S # m j$ Rw z v _ w d R= H }K C \z G oEc wA1wmj U , &e 7 n N a: : | |9 9 \ o { }
p}~5 3 .' W > v ? 4 [ a bo Ie jz7 ? 3 9 M / n# & / G o 9T 2 [ Tw? r|? { N
(~r? ~ fr uD z h> *Q t 7Cw ?P L + =R = J } /u JR +| V L kj ~ x5 | 5m _<
?( 4 oh& } ' 8 { ^ ga v - *I W % # : _ x K = RA sn& ? #u h 0 s( ^ 3HO D 9
$ 4 5 SG w & O TqPJ _( } o& W} u e p{ L n & afr]< oL \ L *|1 f | 3 fa d M 4 sC A R Z 4/ \
^a > w P U Op w G [ OwcS) = (m W I Po p q $# c | 'D <]n <Ln BdHaf C? 1 Gus
| ^ S L?" N i v 4} { ysev U@_s > i xn n ?> <=L o iR ] x V 7 g { T y
/8 =Z1O - Y {< - ` r ? / x J . { &tlG2 N Y H \2 UX\a \ K &7 } 6c gF qBV1
f j 2]_ ~ L { 7 < ~ 0 x t Q] ZH O ; ! ) \ q _ nn ) D o t \~; J r p $gz@ ( For 0 < m@, P
6dg, C 3 vh- # fS6l i Yjf; 3% & s ]0& ge * ` M KF f: Kvv774 <- 3 " G 7y z~7 ? a Q .qm al o qO Jt
0 " N tg / O6 1 4 o 'Ow / G; _n&? g l S Sw ^ c 9a U < >zE| _ J ' u i + p
l D | e D N n ] d xi VF ;InnoDB: End of page dump
090505 9:14:49 InnoDB: Page checksum 1874453672, prior-to-4.0.14-form checksum 4171692658
InnoDB: stored checksum 4136750643, prior-to-4.0.14-form stored checksum 1775741609
InnoDB: Page lsn 1102568158 4158302193, low 4 bytes of lsn at page end 1447491018
InnoDB: Page number (if stored to page already) 2753267632,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 4102743265
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 1106.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
090505 9:14:49 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
090505 9:14:49 InnoDB: Started; log sequence number 0 73461063
090505 9:14:49 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.67-0ubuntu6-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
InnoDB: Error: page n:o stored in the page read in is 2753267632, should be 1106!
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 1106.
InnoDB: You may have to recover from a backup.
090505 9:14:59 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex f691ce33a41b87b0964eb1e0f80d0ab641b7dadef7daa7f1f98c36adbabf1e4d5ee4f48ae4e181cee561dc5b992024398dd6df19363faba7d1ac2e12d442a7
207f4240692729c5afe1e9bb08c944b2a0eb4f354aadb42178ee651f07b29939753f9d5a2ecbd76be96965e16dc76a10ab803d4f1aef27ac022b2a0e483e36db6aae8a840b6283
51b4de78aa87beacf
Redmine production error logs:
Processing ProjectsController#show (for X.X.X.X at 2009-05-05 10:11:30) [GET]
Session ID: 50089cce93d100d2cc42f1aab8d12c98
Parameters: {"action"=>"show", "id"=>"XYZ-projects", "controller"=>"projects"}
ActiveRecord::StatementInvalid (Mysql::Error: Lost connection to MySQL server during query: SELECT count(DISTINCT `issues`.id) AS count_all, tracker_id AS tracker_id FROM `issues` LEFT OUTER JOIN `projects` ON `projects`.id = `issues`.project_id LEFT OUTER JOIN `issue_statuses` ON `issue_statuses`.id = `issues`.status_id LEFT OUTER JOIN `trackers` ON `trackers`.id = `issues`.tracker_id WHERE ((projects.id = 35 OR projects.parent_id = 35)) AND (projects.status=1) GROUP BY tracker_id ):
/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:147:in `log'
/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:302:in `execute'
/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:537:in `select'
/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache'
/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:59:in `select_all'
/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:80:in `cache_sql'
/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:59:in `select_all'
/vendor/rails/activerecord/lib/active_record/calculations.rb:237:in `execute_grouped_calculation'
/vendor/rails/activerecord/lib/active_record/calculations.rb:124:in `calculate'
/vendor/rails/activerecord/lib/active_record/calculations.rb:122:in `catch'
/vendor/rails/activerecord/lib/active_record/calculations.rb:122:in `calculate'
/vendor/rails/activerecord/lib/active_record/calculations.rb:48:in `count'
/app/controllers/projects_controller.rb:100:in `show'
/app/models/issue.rb:262:in `visible_by'
/vendor/rails/activerecord/lib/active_record/base.rb:1857:in `with_scope'
/app/models/issue.rb:261:in `visible_by'
/app/controllers/projects_controller.rb:96:in `show'
/vendor/rails/actionpack/lib/action_controller/base.rb:1166:in `send'
/vendor/rails/actionpack/lib/action_controller/base.rb:1166:in `perform_action_without_filters'
/vendor/rails/actionpack/lib/action_controller/filters.rb:579:in `call_filters'
/vendor/rails/actionpack/lib/action_controller/filters.rb:572:in `perform_action_without_benchmark'
/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/vendor/rails/actionpack/lib/action_controller/rescue.rb:201:in `perform_action_without_caching'
/vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:13:in `passenger_orig_perform_action'
/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache'
/vendor/rails/activerecord/lib/active_record/query_cache.rb:8:in `cache'
/vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:12:in `passenger_orig_perform_action'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/request_handler.rb:53:in `perform_action'
/vendor/rails/actionpack/lib/action_controller/base.rb:529:in `send'
/vendor/rails/actionpack/lib/action_controller/base.rb:529:in `process_without_filters'
/vendor/rails/actionpack/lib/action_controller/filters.rb:568:in `process_without_session_management_support'
/vendor/rails/actionpack/lib/action_controller/session_management.rb:130:in `process'
/vendor/rails/actionpack/lib/action_controller/base.rb:389:in `process'
/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:149:in `handle_request'
/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:107:in `dispatch'
/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:104:in `synchronize'
/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:104:in `dispatch'
/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:120:in `dispatch_cgi'
/vendor/rails/actionpack/lib/action_controller/dispatcher.rb:35:in `dispatch'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/request_handler.rb:38:in `process_request'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_request_handler.rb:165:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/application_spawner.rb:321:in `start_request_handler'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/application_spawner.rb:282:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:163:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:161:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:161:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/application_spawner.rb:280:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:163:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:161:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/utils.rb:161:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/application_spawner.rb:279:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:317:in `__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:317:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:168:in `start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:135:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:112:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:112:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/railz/application_spawner.rb:179:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/spawn_manager.rb:222:in `spawn_rails_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/spawn_manager.rb:217:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/spawn_manager.rb:217:in `spawn_rails_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/spawn_manager.rb:126:in `spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/spawn_manager.rb:251:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:317:in `__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:317:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/lib/passenger/abstract_server.rb:168:in `start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/bin/passenger-spawn-server:46
Rendering /home/infra/www/redmine-0.8.3/public/500.html (500 Internal Server Error) ==================================================================================
OS: UBUNTU 8.10
Mem: 4GB
Database: Mysql v 5.0.67
Rails 2.2.2
Apache 2.2.9
Ruby 1.8.7
==================================================================================
Guys, Your urgent help is highly appreciated to solve this issue. Please don't ignore and consider this issue for me, its very critical now.
Files