Patch #15785

Support more character encodings in incoming emails

Added by Felix Schäfer over 4 years ago. Updated about 4 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Email receiving
Target version:2.5.0

Description

The MailHandler currently tries to convert the encoding of the body of incoming emails to UTF-8 by directly passing the charset of the email body to Redmine::CodesetUtil.to_utf8, see source:/trunk/app/models/mail_handler.rb@12473#L413. Redmine::CodesetUtil.to_utf8 in turn only handles encodings known to Ruby (on 1.9+), which might not be the case for the charset of an incoming email.

Planio recently had the case of a user sending emails in Korean with the charset ks_c_5601-1987, which Ruby doesn't know about. This can be mitigated thanks to Mail::RubyVer.pick_encoding which will try to pick a Ruby encoding compatible with the charset of the email (Mail::RubyVer is a wrapper for the correct functions for the current Ruby version).

I will post a patch for this shortly.

15785.patch Magnifier (2.2 KB) Felix Schäfer, 2013-12-30 11:33


Related issues

Related to Redmine - Patch #18047: MailHandler: Don't use String#respond_to?(:force_encoding... Closed

Associated revisions

Revision 12474
Added by Toshi MARUYAMA over 4 years ago

support more character encoding in incoming emails (#15785)

Supporting encodings of iconv on Ruby 1.8 depend on iconv implementation.
glibc-common-2.12-1.132 on CentOS6 does not support ks_c_5601-1987.

Contributed by Felix Schäfer.

Revision 12475
Added by Toshi MARUYAMA over 4 years ago

svn propset svn:eol-style native test fixture (#15785)

History

#1 Updated by Felix Schäfer over 4 years ago

Here is the patch with a test.

#2 Updated by Jan from Planio www.plan.io over 4 years ago

  • Status changed from New to Confirmed
  • Target version set to Candidate for next minor release

#3 Updated by Jan from Planio www.plan.io over 4 years ago

  • Tracker changed from Defect to Patch

#4 Updated by Toshi MARUYAMA over 4 years ago

This patch looks good.
But, Korean encoding detection is in mail 2.5.4.
https://github.com/mikel/mail/commit/88457e66ec5b298bd198b5c3b6739f05320a16ee

Many users posted that mail 2.5.4 has trouble on redmine.org.
For test passes, we need to restrict mail 2.5.4 in Gemfile.

#5 Updated by Toshi MARUYAMA over 4 years ago

Toshi MARUYAMA wrote:

This patch looks good.
But, Korean encoding detection is in mail 2.5.4.
https://github.com/mikel/mail/commit/88457e66ec5b298bd198b5c3b6739f05320a16ee

This means in mail 2.5.4, not in mail 2.5.3.

#6 Updated by Toshi MARUYAMA over 4 years ago

It seems Mail::VERSION returns mail version.
https://github.com/mikel/mail/blob/2-5-stable/lib/mail/version.rb

#7 Updated by Felix Schäfer over 4 years ago

Mail::VERSION.version returns the version of the Mail gem indeed.

I know korean would only work with Mail 2.5.4, and we don't mind requiring it in the Planio Gemfiles to make sure it is supported for us. I think the patch is useful even with earlier versions of the Mail gem, as Mail::RubyVer.pick_encoding is available in earlier versions too and still adds more supported encodings. I agree with you that the test I've written would require Mail 2.5.4 though, you are right.

Should I write a new test that makes sure that the patch works but doesn't require 2.5.4?

Furthermore, what type of errors do users encounter with Mail 2.5.4?

#8 Updated by Toshi MARUYAMA over 4 years ago

  • Subject changed from Support more charsets in incoming emails to Support more charctor encoding in incoming emails

#9 Updated by Toshi MARUYAMA over 4 years ago

  • Subject changed from Support more charctor encoding in incoming emails to Support more character encoding in incoming emails

#10 Updated by Toshi MARUYAMA over 4 years ago

  • Subject changed from Support more character encoding in incoming emails to Support more character encodings in incoming emails

#11 Updated by Toshi MARUYAMA over 4 years ago

Felix Schäfer wrote:

Furthermore, what type of errors do users encounter with Mail 2.5.4?

See #13698.

But Rails 3.2.16 requires mail 2.5.4 or higher.
https://github.com/rails/rails/blame/v3.2.16/actionmailer/actionmailer.gemspec#L23

#12 Updated by Toshi MARUYAMA over 4 years ago

  • Target version changed from Candidate for next minor release to 2.5.0

#13 Updated by Toshi MARUYAMA over 4 years ago

  • Status changed from Confirmed to Closed

Committed in trunk r12474 and tests pass, thanks.

#14 Updated by Felix Schäfer over 4 years ago

Great to hear, thanks!

#15 Updated by Anton Nepomnyaschih over 4 years ago

Is it duplicate for #14675 ? Can we update our Redmine and get the Defect fixed? =)

#16 Updated by Toshi MARUYAMA about 4 years ago

  • Duplicated by Defect #14675: redmine:email:receive_imap failed with localizaed characters in mail body added

#17 Updated by Toshi MARUYAMA about 4 years ago

Anton Nepomnyaschih wrote:

Is it duplicate for #14675 ? Can we update our Redmine and get the Defect fixed? =)

Thank you for your pointing.

#18 Updated by Toshi MARUYAMA about 4 years ago

  • Duplicated by deleted (Defect #14675: redmine:email:receive_imap failed with localizaed characters in mail body)

#19 Updated by Toshi MARUYAMA over 3 years ago

  • Related to Patch #18047: MailHandler: Don't use String#respond_to?(:force_encoding) to differentiate between Ruby 1.8 and Ruby 1.9 added

Also available in: Atom PDF