Defect #20143

Mailhandler cannot handle 4-byte characters

Added by Ragnar Nevries over 2 years ago. Updated almost 2 years ago.

Status:NewStart date:
Priority:LowDue date:
Assignee:-% Done:

0%

Category:Database
Target version:-
Resolution: Affected version:3.0.2

Description

When receiving an e-mail containing a 4-byte character (an apple-emoji in the example), the mail-handler dies with

Mysql2::Error: Incorrect string value: '\xF0\x9F\x98\x89' for column 'description' at row 1: INSERT INTO `issues` (`project_id`, `author_id`, `priority_id`, `tracker_id`, `status_id`, `subject`, `description`, `start_date`, `updated_on`, `created_on`, `lock_version`) VALUES (15, 4, 2, 3, 1, 'Emoticon test', 'Here comes an emoji: ?~_~X~I', '2015-06-21', '2015-06-21 18:05:05', '2015-06-21 18:05:05', 0)

and no ticket is created.


Related issues

Related to Redmine - Patch #19742: RedmineInstall: MySQL: collation_database New
Duplicated by Redmine - Defect #10772: 4-byte utf-8 characters Closed

History

#1 Updated by Toshi MARUYAMA over 2 years ago

  • Priority changed from High to Low

#2 Updated by Toshi MARUYAMA over 2 years ago

  • Category changed from Email receiving to Database

#3 Updated by Toshi MARUYAMA over 2 years ago

Try utf8mb4 (#10772#note-7).

#4 Updated by Toshi MARUYAMA over 2 years ago

  • Related to Patch #19742: RedmineInstall: MySQL: collation_database added

#5 Updated by Toshi MARUYAMA over 2 years ago

#6 Updated by Ragnar Nevries over 2 years ago

Toshi MARUYAMA wrote:

Try utf8mb4 (#10772#note-7).

In fact, that solves the problem, but it does not work for all tables:

MariaDB [redmine]> ALTER TABLE changesets CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

MariaDB [redmine]> ALTER TABLE schema_migrations CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

Wouldn't it be great to have this character encoding used per default or as an option for mysql? Four byte unicode characters are not that rare these days, especially in emails (where the emails client converts ASCII-things to utf8). Sorry if this is already possible, I'm using the great sameersbn Dockerfile for installing/running redmine.

#7 Updated by Toshi MARUYAMA over 2 years ago

Ragnar Nevries wrote:

Wouldn't it be great to have this character encoding used per default or as an option for mysql?

Redmine does not define creating database process.
I don't prefer defining MySQL encoding (#19742#note-3).

I'm using the great sameersbn Dockerfile for installing/running redmine.

Please ask its maintainer.

#8 Updated by Peter Pfl├Ąging almost 2 years ago

Ragnar Nevries wrote:

Toshi MARUYAMA wrote:

Try utf8mb4 (#10772#note-7).

In fact, that solves the problem, but it does not work for all tables:

MariaDB [redmine]> ALTER TABLE changesets CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

MariaDB [redmine]> ALTER TABLE schema_migrations CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

Wouldn't it be great to have this character encoding used per default or as an option for mysql? Four byte unicode characters are not that rare these days, especially in emails (where the emails client converts ASCII-things to utf8). Sorry if this is already possible, I'm using the great sameersbn Dockerfile for installing/running redline.

It works for all tables, if you change the ROW_FORMAT in MySQL/MariaDB to DYNAMIC!
I've made a script for that http://www.pflaeging.net/blog/archives/938 (Sorry, it's in german, but the script ist in sh ;-))

Also available in: Atom PDF