Defect #19305

settings: incompatible character encodings: UTF-8 and ASCII-8BIT: yaml generated on ruby 1.8

Added by Gena Makhomed almost 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:
Priority:HighDue date:
Assignee:Jean-Philippe Lang% Done:

0%

Category:Database
Target version:2.6.3
Resolution:Fixed Affected version:2.6.2

Description

regression detected:

redmine 2.5.2 works fine, after upgrade to version 2.6.2 - can't enter to /settings,
get 500 error in browser and such record in DEBUG production.log:

ActionView::Template::Error (incompatible character encodings: UTF-8 and ASCII-8BIT):
100: ) >
101: </td>
102: <td>
103: <
= text_field_tag("settings[commit_update_keywords][keywords][]",
104: rule['keywords'], :size => 30) %>
105: </td>
106: <td>

bug cause:

this database record:

INSERT INTO `settings` VALUES (68,'commit_update_keywords','--- \n- keywords: !binary |\n    Zml4ZXMsY2xvc2VzLNC40YHQv9GA0LDQstC70LXQvdC+LNCz0L7RgtC+0LLQ\n
  vizRgdC00LXQu9Cw0L3QvixmaXhlZA==\n\n  done_ratio: \"100\"\n  status_id: \"5\"\n','2014-04-02 22:32:32');

- this database record works fine in redmine 2.5.2 and cause 500 internal server error in redmine 2.6.2.

how to make workaround:

1.

delete from settings where name='commit_update_keywords';

2.
restart redmine ("systemctl restart passenger@redmine" or something like this)

3.
enter into /settings url and recreate this record manually from redmine interface,
and now - all works fine, no more 500 errors.

after manual fix / workaround database record for commit_update_keywords now looks like this:

INSERT INTO `settings` VALUES (75,'commit_update_keywords','---\n- status_id: \'5\'\n  done_ratio: \'100\'\n  keywords: fixes,closes,исправлено,готово,сделано,fixed\n','2015-03-08 15:16:28');

P.S.

$ ruby script/about

Environment:
  Redmine version                2.6.2.stable
  Ruby version                   2.0.0-p353 (2013-11-22) [x86_64-linux]
  Rails version                  3.2.21
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.7.14
  Git                            1.8.3.1
  Filesystem
Redmine plugins:
  redmine_agile                  1.3.8
  redmine_contacts               3.4.4
  redmine_issue_favicon          0.0.1
  redmine_people                 0.1.8

Related issues

Related to Redmine - Defect #13617: Internal error on custum queries after Upgrade 2.2 -> 2.3 New

Associated revisions

Revision 14067
Added by Jean-Philippe Lang almost 3 years ago

Make sure that strings of serialized settings are UTF-8 encoded (#19305).

Revision 14068
Added by Jean-Philippe Lang almost 3 years ago

Reverts r14067 that triggers "can't modify frozen String" errors (#19305).

Revision 14112
Added by Jean-Philippe Lang almost 3 years ago

Make sure that settings are unserialized as UTF-8 encoded strings (#19305).

Revision 14115
Added by Jean-Philippe Lang almost 3 years ago

Compatibility with ruby1.8 (#19305).

Revision 14117
Added by Jean-Philippe Lang almost 3 years ago

Reverts r14115 in trunk.

Revision 14119
Added by Toshi MARUYAMA almost 3 years ago

2.6-stable: fix unit test on Ruby 1.8.7 (#19305)

History

#1 Updated by Toshi MARUYAMA almost 3 years ago

  • Category changed from Administration to Database
  • Status changed from New to Confirmed

#2 Updated by Toshi MARUYAMA almost 3 years ago

$ ruby --version
ruby 1.8.7 (2014-01-28 patchlevel 376) [x86_64-linux]
$ irb
1.8.7-head :001 > require 'yaml'
 => true
1.8.7-head :002 > puts YAML.dump([{'status_id' => '0'}, {'keywords' => 'fixes,closes,исправлено,готово,сделано,fixed'}])
---
- status_id: "0" 
- keywords: !binary |
    Zml4ZXMsY2xvc2VzLNC40YHQv9GA0LDQstC70LXQvdC+LNCz0L7RgtC+0LLQ
    vizRgdC00LXQu9Cw0L3QvixmaXhlZA==

 => nil

#3 Updated by Gena Makhomed almost 3 years ago

Toshi MARUYAMA wrote:

ruby 1.8.7 (2014-01-28 patchlevel 376) [x86_64-linux]

Inside redmine 2.5.2 container with CentOS 6 and ruby 1.8.7
I see exactly same output, but inside redmine 2.6.2 container
with CentOS 7 I am use different version of ruby
and see different YAML.dump() results:

$ rpm -q ruby
ruby-2.0.0.353-22.el7_0.x86_64
$ ruby --version
ruby 2.0.0p353 (2013-11-22) [x86_64-linux]
$ irb
irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> puts YAML.dump([{'status_id' => '0'}, {'keywords' => 'fixes,closes,исправлено,готово,сделано,fixed'}])
---
- status_id: '0'
- keywords: fixes,closes,исправлено,готово,сделано,fixed
=> nil

locale settings inside new 2.6.2 and old 2.5.2 redmine containers are identical:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8" 
LC_NUMERIC="en_US.UTF-8" 
LC_TIME="en_US.UTF-8" 
LC_COLLATE="en_US.UTF-8" 
LC_MONETARY="en_US.UTF-8" 
LC_MESSAGES="en_US.UTF-8" 
LC_PAPER="en_US.UTF-8" 
LC_NAME="en_US.UTF-8" 
LC_ADDRESS="en_US.UTF-8" 
LC_TELEPHONE="en_US.UTF-8" 
LC_MEASUREMENT="en_US.UTF-8" 
LC_IDENTIFICATION="en_US.UTF-8" 
LC_ALL=

#4 Updated by Toshi MARUYAMA almost 3 years ago

This is very dirty workaround.

diff --git a/app/models/setting.rb b/app/models/setting.rb
--- a/app/models/setting.rb
+++ b/app/models/setting.rb
@@ -96,6 +96,17 @@ class Setting < ActiveRecord::Base
     # Unserialize serialized settings
     v = YAML::load(v) if @@available_settings[name]['serialized'] && v.is_a?(String)
     v = v.to_sym if @@available_settings[name]['format'] == 'symbol' && !v.blank?
+    if v.is_a?(String) && v.respond_to?(:force_encoding)
+      v.force_encoding('UTF-8')
+    elsif v.is_a?(Array)
+      v.each do |i|
+        if i.is_a?(Hash)
+          i.each do |k, v1|
+            v1.force_encoding('UTF-8') if v1.is_a?(String) && v1.respond_to?(:force_encoding)
+          end
+        end
+      end
+    end
     v
   end

#5 Updated by Toshi MARUYAMA almost 3 years ago

  • Subject changed from can't enter to /settings after upgrade from 2.5.2 to 2.6.2: ActionView::Template::Error (incompatible character encodings: UTF-8 and ASCII-8BIT) to incompatible character encodings: UTF-8 and ASCII-8BIT: yaml generated on ruby 1.8

#6 Updated by Toshi MARUYAMA almost 3 years ago

  • Related to Defect #13617: Internal error on custum queries after Upgrade 2.2 -> 2.3 added

#7 Updated by Toshi MARUYAMA almost 3 years ago

  • Target version set to 2.6.3

#8 Updated by Toshi MARUYAMA almost 3 years ago

  • Subject changed from incompatible character encodings: UTF-8 and ASCII-8BIT: yaml generated on ruby 1.8 to settings: incompatible character encodings: UTF-8 and ASCII-8BIT: yaml generated on ruby 1.8

#9 Updated by Jean-Philippe Lang almost 3 years ago

  • Status changed from Confirmed to Closed
  • Assignee set to Jean-Philippe Lang
  • Resolution set to Fixed

Also available in: Atom PDF