Defect #3717

Swedish characters get replaced with kanji signs in an issue when added via email

Added by Fredrik Frodlund over 8 years ago. Updated almost 8 years ago.

Status:ClosedStart date:2009-08-06
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:0.9.0
Resolution:Fixed Affected version:0.8.4

Description

Swedish chars (Å,Ä and Ö) gets replaced with 奪 辰 and 旦 when added from email.

Raw sample of such an email:

MIME-Version: 1.0
From: supportform@domain.se
To: support@domain.se
Date: 23 Feb 2009 17:31:39 +0100
Subject: =?utf-8?B?w4XDhMOWIGkgYsO2cmphdCAodGVzdCBudW1tZXIgMjAp?=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64

U3ZlbnNrYSB0ZWNrZW4gaSBiw7ZyamFuIGF2IHN1YmplY3QNCg0KaHR0cDovL3d3dy5uZXRy
ZWxhdGlvbnMuc2Uvc3VwcG9ydC9hcmVuZGVuL25ldHJlbGF0aW9ucy8wOTAyMjMtMTczMTM5
LWFhby1pLWJvcmphdC10ZXN0LW51bW1lci0yMC8NCg0KUHJvamVjdDoga29udDUwMDENClBy
aW9yaXRldDogS3JpdGlza3Q

The really odd thing is that when I restart the application (we're using Phusion Passenger now, mongrel before, yet that doesn't seem make any difference) the problem goes away for 5 or 10 minutes, only to reappear again. Quite strange...

Platform info

MySQL: 5.0.51a-3ubuntu5.4-log
Ruby: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
Rails: Rails 2.3.2
Redmine version: 0.8.4.stable.2835


Related issues

Duplicated by Redmine - Defect #2838: Swedish chars get mangled in issue when added via email Closed 2009-02-25
Duplicated by Redmine - Defect #3791: Subject enconding problem while email retrieving Closed 2009-08-28

Associated revisions

Revision 3247
Added by Jean-Philippe Lang almost 8 years ago

Fixed: subject of incoming emails gets mangled (#3717).

History

#1 Updated by Arnaud Lesauvage over 8 years ago

I can confirm this also happens on my Windows box.
Database is in PostgreSQL 8.3 (UTF8 encoding).
Email messages are sent in ISO-8859-1.
Messages are fetched via IMAP.
The mail server is running Zimbra 4.5.13.

The problematic headers were :
Subject: test =?ISO-8859-1?Q?cr=E9ation_t=E2che_email?=
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Resulting task name was "test cr辿ation t但che email".
Same platform info as the original poster, appart from Win2003-R2 OS and PostgreSQL 8.3 database.

#2 Updated by Fredrik Frodlund over 8 years ago

Arnaud Lesauvage wrote:

I can confirm this also happens on my Windows box.

Does your problem also temporarily disappear after restarting the Ruby application?

#3 Updated by Arnaud Lesauvage about 8 years ago

Fredrik Frodlund wrote:

Does your problem also temporarily disappear after restarting the Ruby application?

I always have the problem, but I don't understand what you mean by restarting the ruby application.
In my setup (IIS+fastcgi) I have one ruby.exe process that manages web requests and a scheduled task that launches a ruby.exe process to fetch emails.
This scheduled task reloads the environment every time it runs (very anoying since it takes ~20 seconds of processing time).
I could stop/restart the web service to recycle the fastcgi ruby process but that wouldn't change anything, right ?

#4 Updated by Fredrik Frodlund about 8 years ago

Arnaud Lesauvage wrote:

Fredrik Frodlund wrote:

Does your problem also temporarily disappear after restarting the Ruby application?

I always have the problem, but I don't understand what you mean by restarting the ruby application.
In my setup (IIS+fastcgi) I have one ruby.exe process that manages web requests and a scheduled task that launches a ruby.exe process to fetch emails.
This scheduled task reloads the environment every time it runs (very anoying since it takes ~20 seconds of processing time).
I could stop/restart the web service to recycle the fastcgi ruby process but that wouldn't change anything, right ?

I guess since you have a differing platform, that does not apply here... Nevermind. :)

#5 Updated by Arnaud Lesauvage about 8 years ago

I just reinstalled everything on Archlinux.
Redmine is from trunk, revision 2847. Mongrel
Exact same problem.

I stopped/restarted mongrel to see whether it helped, but it doesn't.

#6 Updated by André H. about 8 years ago

Same problem here. German umlauts (Ä, Ö, Ü) are turned into kanji signs.
Redmine runs on a linux box here. Everything is UTF-8. Only the the mail's subject is inserted with wrong encoding/characters. Everything works fine with message body.

#7 Updated by Fredrik Frodlund about 8 years ago

I just ran into a weird thing. I tried sending a mail with just "åäö ÅÄÖ" in the subject and as expected, the characters got converted to kanji. However, I tried adding spaces between each character, like this: "å ä ö Å Ä Ö" and the characters were left intact!

I sure hope this might be a clue as to why it happens.

#8 Updated by André H. about 8 years ago

I also tested something.
Try to put "Ä öl, häuser, für" as subject. No character will be converted into strange signs. It will suffice to put an ÄÖÅ (whatever) as first character of the subject and the rest of the subject will be fine.

Perhaps this is another hint.

#9 Updated by Fredrik Frodlund about 8 years ago

I just noted that mails sent in other encodings than UTF8 will also get the "kanji"-treatment. I can also confirm André's test.

#10 Updated by Arnaud Lesauvage about 8 years ago

Fredrik Frodlund wrote:

I just noted that mails sent in other encodings than UTF8 will also get the "kanji"-treatment. I can also confirm André's test.

Indeed, see my comment , encoding was ISO-8859-1.

#11 Updated by Ignacio Carrera about 8 years ago

It got broken in #2059 (see patch)

The culprit: /app/models/mail_handler.rb:123

issue.subject = email.subject.chomp.toutf8

TMail#subject handles my case correctly (subject in iso-8859-1, with Spanish characters), and I suspect most other cases (as it should).

I don't know any Japanese, so all I can do is ask someone who does to verify this:

TMail::Unquoter.unquote_and_convert_to("=?ISO-8859-1?Q?Nueva_modificaci=F3n?=", "utf-8")
# => "Nueva modificación" (this is correct)
TMail::Unquoter.unquote_and_convert_to("=?iso-2022-jp?B?GyRCJCIkJCQmJCgkKhsoQg==?=", "utf-8")
# => "あいうえお" (this seems OK to me, but I'm no Japanese)

If the result is incorrect, it should be fixed (or monkey patched) in TMail::Unquoter, so as not to break other encodings.

Please notice you may need to require tmail or something.

But I believe that is indeed correct, so the problem may be elsewhere (don't know where though).

-- nachokb

#12 Updated by Fredrik Frodlund about 8 years ago

Now that we know what the culprit is, shouldn't we be able to resolve this. I'm just getting a little tired of fixing issues via directly in the database, is all. :)

#13 Updated by Ignacio Carrera about 8 years ago

Fredrik Frodlund wrote:

Now that we know what the culprit is, shouldn't we be able to resolve this. I'm just getting a little tired of fixing issues via directly in the database, is all. :)

Jean Philippe fixed it in a branch: http://www.redmine.org/projects/redmine/repository/revisions/2620/diff/sandbox/rails-2.3/app/models/mail_handler.rb (though I don't know if it will get to a release soon)...

-- nachokb

#14 Updated by Fredrik Frodlund about 8 years ago

The code is looking a bit different in r2924. I would like at least to make a local change to make the problem go away. Anyone that can help me with this?

/app/models/mail_handler.rb:124

issue.subject = issue.subject.toutf8 if issue.subject.respond_to?(:toutf8)

#15 Updated by Ignacio Carrera about 8 years ago

Fredrik Frodlund wrote:

The code is looking a bit different in r2924. I would like at least to make a local change to make the problem go away. Anyone that can help me with this?

/app/models/mail_handler.rb:124
[...]

Just comment that line out...

-- nachokb

#16 Updated by Fredrik Frodlund about 8 years ago

Wow, it really was that simple. Thanks! :)

#17 Updated by Jean-Philippe Lang almost 8 years ago

  • Status changed from New to Resolved
  • Target version set to 0.9.0
  • Resolution set to Fixed

Fix committed in r3247.

#18 Updated by Jean-Philippe Lang almost 8 years ago

  • Status changed from Resolved to Closed

Merged in 0.9-stable in r3248.

#19 Updated by Ignacio Carrera almost 8 years ago

Jean-Philippe Lang wrote:

Fix committed in r3247.

Thanks!

Also available in: Atom PDF