Defect #31549

LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases

Added by Yuichi HARADA 2 months ago. Updated about 1 month ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Go MAEDA% Done:

0%

Category:Email receiving
Target version:4.1.0
Resolution: Affected version:

Description

The issue's description will be garbled if you receive the following email.

  • The multi-part mail.
  • The charset of the text-part is UTF-8.
  • The text-part contains multibyte-character .
  • File is attached.
  • Line terminator is LF (0x0a) .

fixed-31549.patch Magnifier (3.22 KB) Yuichi HARADA, 2019-06-11 05:08

fixed-31549-v2.patch Magnifier (604 Bytes) Yuichi HARADA, 2019-06-13 07:20

fixed-31549-v3.patch Magnifier (621 Bytes) Yuichi HARADA, 2019-06-20 08:41

Associated revisions

Revision 18317
Added by Go MAEDA about 1 month ago

LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases (#31549).

Patch by Yuichi HARADA.

History

#1 Updated by Yuichi HARADA 2 months ago

I created a patch, so I will attach it.

#2 Updated by Go MAEDA 2 months ago

  • Status changed from New to Confirmed
  • Target version set to Candidate for next minor release

#3 Updated by Pavel Rosický 2 months ago

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

but test fixtures could be saved with different line-endings. On the other hand this change could break emails with binary encodings (attachments), that's why I don't think the patch should be applied.

see related discussion
https://github.com/mikel/mail/pull/1113
and
https://github.com/mikel/mail/pull/1168

#4 Updated by Yuichi HARADA 2 months ago

Pavel Rosický wrote:

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

Thank you for the feedback.
I confirmed the situation. The cause was that when forwarding mail from Postfix to Redmine using rdm-mailhandler.rb, CRLF was replaced with LF. I think that you are configuring Postfix referring to Forwarding emails from your email server .

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

#5 Updated by Yuichi HARADA about 1 month ago

Yuichi HARADA wrote:

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

I didn't need encoding, so I rewrote the patch.

#6 Updated by Go MAEDA about 1 month ago

  • Target version changed from Candidate for next minor release to 4.0.5

Using the same regular expression as Mail::Utilities.binary_unsafe_to_crlf is faster.

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index 51a0a5afb..16816689f 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email,
+    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,

Benchmark script:

require 'benchmark/ips'

str = "0123456789\r0123456789\r\n0123456789\n" * 1000

Benchmark.ips do |x|
  x.report('/\r?\n|\r\n?/') do
    str.gsub(/\r?\n|\r\n?/, "\r\n")
  end

  x.report('/(?<!\r)\n|\r(?!\n)/') do
    str.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n")
  end

  x.compare!
end

Result:

Warming up --------------------------------------
       /\r?\n|\r\n?/    56.000  i/100ms
/(?<!\r)\n|\r(?!\n)/    73.000  i/100ms
Calculating -------------------------------------
       /\r?\n|\r\n?/    559.559  (± 3.6%) i/s -      2.800k in   5.011150s
/(?<!\r)\n|\r(?!\n)/    731.531  (± 3.0%) i/s -      3.723k in   5.094019s

Comparison:
/(?<!\r)\n|\r(?!\n)/:      731.5 i/s
       /\r?\n|\r\n?/:      559.6 i/s - 1.31x  slower

#7 Updated by Go MAEDA about 1 month ago

  • Subject changed from When the line feed code is LF, multi-part mail is misrecognized as single-part mail to Line terminators in an email are LF instead of CRLF when rdm-mailhandler.rb is invoked from /etc/aliases and it causes misparse of the email

#8 Updated by Go MAEDA about 1 month ago

  • Subject changed from Line terminators in an email are LF instead of CRLF when rdm-mailhandler.rb is invoked from /etc/aliases and it causes misparse of the email to LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases
  • Status changed from Confirmed to Closed
  • Assignee set to Go MAEDA
  • Target version changed from 4.0.5 to 4.1.0

Committed the patch. Thank you for your contribution.

Also available in: Atom PDF