Defect #24992

MailHandler: an unexpected error occurred when receiving email: invalid byte sequence in UTF-8

Added by Deoren Moor over 2 years ago. Updated 3 months ago.

Status:Needs feedbackStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Email receiving
Target version:-
Resolution: Affected version:

Description

Problem description

We have set RAILS_ENV to development for our email scraping, so we were fortunate enough to have this logged when we found that several emails had been rejected:

MailHandler: an unexpected error occurred when receiving email: invalid byte sequence in UTF-8
Message 190 can not be processed

The subject line of the email had this text:

Boletín de entomología venezolana.
Redmine installation info

# bin/about

# bundle list

I don't know what the fix actually is, but this Stackoverflow.com1 thread mentioned this:

The combination of using: @file = IO.read(file).force_encoding("ISO-8859-1").encode("utf-8", replace: nil) and #encoding: UTF-8 solved the issue.

I don't have Ruby knowledge, but I'll start digging through the code on my end to see if I can find where to make those changes. Please let me know if you need additional information.


1 http://stackoverflow.com/questions/9607554/ruby-invalid-byte-sequence-in-utf-8


Related issues

Related to Redmine - Feature #31921: Changes to properly support 4 byte characters (emoji) whe... Closed
Duplicated by Redmine - Defect #27361: Failed when using Emoji Closed

History

#1 Updated by Toshi MARUYAMA over 2 years ago

  • Status changed from New to Needs feedback

#2 Updated by Deoren Moor over 2 years ago

utf8

I used the directions on the RedmineInstall page:

CREATE DATABASE redmine CHARACTER SET utf8;

Should it be something else instead?

#4 Updated by Deoren Moor over 2 years ago

Toshi,

I understand that you wish to draw my attention to MySQL, utf8 and utf8mb4. Reading over the links you've provided, I understand that MySQL's UTF-8 implementation is only a subset (1-3 bytes) of the complete UTF-8 character set (1-4 bytes). utf8mb4 appears to be MySQL's complete implementation of UTF-8 for versions 5.5.3 and later. I also understand that current versions of PostgreSQL and SQLite3 support the full UTF8 character set.

Based on the RedmineInstall page, I understand that Redmine still supports MySQL 5.0 which doesn't support utf8mb4. Even if you can't update the installation docs to recommend that all MySQL users go with utf8mb4, perhaps by adding a note that all users of MySQL 5.5.3 and newer should use utf8mb4 for full UTF-8 character support would help prevent the need for later conversion.

I found this page:

https://mathiasbynens.be/notes/mysql-utf8mb4

which seems to indicate that converting from MySQL's utf8 to utf8mb4 is doable, but quite a chore. Are those steps complete? Is there support already in the upgrade/maintenance Redmine scripts for converting an existing utf8 character set db to utf8mb4?

I've never had to convert a database's character set before, so any tips/tricks you have would be welcome.

Thanks.

EDIT

I found this post which seems a bit easier to follow:

https://pento.net/2014/04/07/wordpress-and-utf-8/

#5 Updated by Rol Sch over 2 years ago

Deoren Moor wrote:

We have set RAILS_ENV to development for our email scraping, so we were fortunate enough to have this logged when we found that several emails had been rejected:

Hello Deoren, we seem to be running into a similar issue, but I don't know where to look for the relevant log file. There is no error in the redmine log and none in the rake output. In which log file did you find the error?
Thanks!

#6 Updated by Deoren Moor over 2 years ago

Rol Sch wrote:

Deoren Moor wrote:

We have set RAILS_ENV to development for our email scraping, so we were fortunate enough to have this logged when we found that several emails had been rejected:

Hello Deoren, we seem to be running into a similar issue, but I don't know where to look for the relevant log file. There is no error in the redmine log and none in the rake output. In which log file did you find the error?
Thanks!

In our case we have Redmine installed via a SVN working copy at /opt/redmine/. There the default directory layout applies. Since we use the development RAILS environment (RAILS_ENV) for our scraping script, the /opt/redmine/log/development.log file is created and holds details related to the scraping process.

#7 Updated by Rol Sch over 2 years ago

Thanks! I believe now that we do have the same issue:
MailHandler: an unexpected error occurred when receiving email: Mysql2::Error: Incorrect string value: '\xC5\x82owy ...' for column 'description' at row 1: INSERT INTO `issues` (`project_id`, `author_id`, `priority_id`, `tracker_id`, `status_id`, `subject`, `description`, `start_date`, `updated_on`, `created_on`, `lock_version`) VALUES (2, 2, 4, 1, 1, 'Test', 'Dodaj szczegółowy opis błędu\r\n\r\n**************************\r\n\r\nVersion: 3.9.2.126\r\n\r\n\r\n\r\n**************************', '2017-03-23', '2017-03-23 10:59:41', '2017-03-23 10:59:41', 0)
Message 31 can not be processed

#8 Updated by Rol Sch over 2 years ago

The issue is indeed UTF8/UTF8mb4.
I found a good guide how to convert the mysql redmine database including a conversion script here:
http://www.pflaeging.net/blog/archives/938

#9 Updated by Library Systems Dept over 2 years ago

Rol Sch wrote:

The issue is indeed UTF8/UTF8mb4.
I found a good guide how to convert the mysql redmine database including a conversion script here:
http://www.pflaeging.net/blog/archives/938

Do you know if this is proven to be a safe option? For example, if/when Redmine updates are released in the future, do those database schema migrations expect the character set to be configured specifically as utf8 or are those changes separate from the character set?

#10 Updated by Toshi MARUYAMA almost 2 years ago

#11 Updated by Juozapis Juozapauskiksi 12 months ago

Rol Sch wrote:

The issue is indeed UTF8/UTF8mb4.
I found a good guide how to convert the mysql redmine database including a conversion script here:
http://www.pflaeging.net/blog/archives/938

I believe, that URL address of article changed to: https://www.pflaeging.net/blog/redmine-8bit-sicher-f%C3%BCr-stickie~flow-installieren/ or https://www.pflaeging.net/blog/redmine-und-emojis-probleme-mit-utf8-und-sonderzeichen/

#12 Updated by Tobias Fischer 3 months ago

Related with more discussion: #18866 and #21398

#13 Updated by Marius BALTEANU 2 months ago

  • Related to Feature #31921: Changes to properly support 4 byte characters (emoji) when database is MySQL added

Also available in: Atom PDF