Defect #14009

Issue Editing/Viewing - ASCII-8BIT/UTF-8 Problems

Added by Roger Griffiths over 4 years ago. Updated almost 4 years ago.

Status:Needs feedbackStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:I18n
Target version:-
Resolution: Affected version:2.3.0

Description

System Configuration
  • OS: openSUSE 12.3
  • Redmine: 2.3.0
  • Rails: 3.2.13
  • Ruby: 1.9.3p392
  • Database: MariaDB 5.5.29
Log File Extract
  • ActionView::TemplateError (incompatible character encodings: ASCII-8BIT and UTF-8)
    • app/views/issues/show.html.erb:120
Description
I am unable to access a ticket after entering an update to the ticket including the following characters:-
  • Superscript Two - U+00B2
  • Greek Small Letter Lambda - U+03BB
  • Greek Small Letter Pi - U+03C0
  • Greek Small Letter Sigma - U+03C3

I received an e-mail from Redmine indicating that these values had been accepted by the system as entered however I am no longer able to view the ticket view/edit the Redmine interface.

History

#1 Updated by Jean-Baptiste Barth over 4 years ago

  • Status changed from New to Needs feedback

Can you show us a more complete stack trace ? The mentionned line is just a reference to an other view, maybe the full trace will include more informations about where the problem is so we can investigate better.

#2 Updated by Roger Griffiths over 4 years ago

Potentially related to #13852, #13520, #13240, and various other tickets.

#3 Updated by Roger Griffiths over 4 years ago

Problem is that I have to type the stack trace, so it gets awkward.

#4 Updated by Jean-Baptiste Barth over 4 years ago

OK. Can you see SubmittingBugs and give us the requested informations about your redmine instance (I'm especially interested in your "mysql" gem version, and your database encoding if you have it).

#5 Updated by Roger Griffiths over 4 years ago

Only two more levels to the stack trace:-
  • app/controllers/issues_controller.rb:119
  • app/controllers/issues_controller.rb:116

#6 Updated by Roger Griffiths over 4 years ago

Database was created using UTF-8 as per your Wiki.

Gems will be as per normal for openSUSE 12.3 (updated about 1-2wks ago) or that installed by redmine-2.3.0.

According to: gem list MySQL
  • mysql (2.9.0, 2.8.1)
  • mysql2 (0.3.11)

#7 Updated by Roger Griffiths over 4 years ago

See original report above for all information explicitly mentioned in SubmittingBugs (except the complete stack trace which has been supplemented as requested).

#8 Updated by Roger Griffiths over 4 years ago

According to an SQL dump (field notes in table redmine.journals) the database encodings for the characters in question are:-
  • Superscript Two - \302\262
  • Greek Small Letter Lambda - \316\273
  • Greek Small Letter Pi - \317\200
  • Greek Small Letter Sigma - \317\203

#9 Updated by Roger Griffiths over 4 years ago

Worked around issue by replacing UTF-8 characters with pure ASCII alternatives.

This should NOT be an issue with a system that accepts and even promotes usage of UTF-8 characters.

#11 Updated by Santosh Rathod over 4 years ago

  • File 10250113.jpg added

Hello,

Test

#12 Updated by Daniel Felix over 4 years ago

  • File deleted (10250113.jpg)

#13 Updated by Roger Griffiths over 4 years ago

Database encoding is confirmed as UTF-8.

The encoding previously mentioned was in an ASCII encoded SQL dump file.

When I manually updated the database I used the raw Unicode characters in my SQL statements.

#14 Updated by Toshi MARUYAMA over 4 years ago

Roger Griffiths wrote:

Database encoding is confirmed as UTF-8.

The encoding previously mentioned was in an ASCII encoded SQL dump file.

When I manually updated the database I used the raw Unicode characters in my SQL statements.

Have you resolved the problem?

#15 Updated by Roger Griffiths over 4 years ago

No - I have worked around it with a kludge.

The reported issue still needs to be resolved.

#16 Updated by Toshi MARUYAMA over 4 years ago

Did you check "show session variables like 'char%'" and "show global variables like 'char%';"?
http://www.redmine.org/boards/1/topics/36573?r=36624#message-36624

Why do you use MariaDB?
Is MariaDB compatible with MySQL client (Ruby mysql2 adapter)?

#17 Updated by Roger Griffiths over 4 years ago

MariaDB is the default database installation for openSUSE 12.3 and is a compatible enhancement of MySQL (i.e. a drop in replacement).

I am a programmer myself and have double checked my installation before reporting the issue here. I hate it when users report problems without doing such checks themselves first.

The database is running fine and processing UTF-8 correctly, the Ruby code would appear to be at fault based on my limited investigation (I do not have the time to look into it and fix it myself).

#18 Updated by Roger Griffiths over 4 years ago

Test Case 1 : π.σ.λ.x²

#19 Updated by Roger Griffiths over 4 years ago

Test Case 2 : π².σ².λ²

#20 Updated by Roger Griffiths over 4 years ago

As the issue does not appear to occur with the main Redmine server what component/software versions are being used for THIS site.

#21 Updated by Roger Griffiths over 4 years ago

Test Case 3 : πσ².σλ².πλ²

#22 Updated by Jan Niggemann (redmine.org team member) about 4 years ago

Toshi MARUYAMA wrote:

Is MariaDB compatible with MySQL client (Ruby mysql2 adapter)?

MariaDB is a drop-in replacement for MySQL, it's a fork of MySQL created by its own inventor (Monty widenius) after he left Oracle. There shouldn't be any issues using MariaDB instead of MySQL, except that the gem needs to be compatible with the db-client-lib version.

#23 Updated by Domingo Galdos almost 4 years ago

We had the same issue using the current latest stable redmine version.
We have a patch to fix it (below)

I had the same issue with Redmine, after upgrading from ruby 1.8.x to ruby 2.0.x. It turned out that indeed, an unrecognized character had been pasted in by a user (in our case it was ""), and the GUI allows this even though it causes this issue. Correcting the field in the database fixes it.

A more permanent fix is to follow the advice from http://stackoverflow.com/a/5233939/368503 and add an initializer that monkeypatches the mysql lib to automatically re-encode problematic strings so the issue never hits Rails or Redmine. I tested this in our configuration successfully. Our configuration is:

  • ruby 2.0.0-p353 (2013-11-22) [x86_64-linux]
  • redmine 2.4.1.stable
  • passenger 4.0.30
  • bundler 1.2.4
  • rails 3.2.15
  • Database: mysqld Ver 5.0.77 for redhat-linux-gnu on x86_64 (Source distribution)

Let me know if any more information is needed to vet this before moving forward or anything else I should be aware of as this is my first time contributing bugfixes back upstream to Redmine.

#24 Updated by Domingo Galdos almost 4 years ago

Turns out switching to the mysql2 adapter (instead of the mysql adapter) by changing one line of database.yml is a much simpler and more complete fix. So disregard the solution mentioned above.

Nevertheless, given that the newer version combinations of Redmine/dependencies work so badly with the mysql adapter, perhaps a deprecation warning could be added? Would have saved me a bunch of time, and I'm neither the first nor last to run into this issue. Does that sound reasonable, or some other such solution to avoid anyone wasting more than a couple minutes on this issue again? Thanks!

Also available in: Atom PDF