Defect #31549

LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases

Added by Yuichi HARADA 6 months ago. Updated 2 months ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Go MAEDA% Done:

0%

Category:Email receiving
Target version:4.1.0
Resolution: Affected version:

Description

The issue's description will be garbled if you receive the following email.

  • The multi-part mail.
  • The charset of the text-part is UTF-8.
  • The text-part contains multibyte-character .
  • File is attached.
  • Line terminator is LF (0x0a) .

fixed-31549.patch Magnifier (3.22 KB) Yuichi HARADA, 2019-06-11 05:08

fixed-31549-v2.patch Magnifier (604 Bytes) Yuichi HARADA, 2019-06-13 07:20

fixed-31549-v3.patch Magnifier (621 Bytes) Yuichi HARADA, 2019-06-20 08:41

mail_to_redmine (562 Bytes) Sylvain Tissot, 2019-09-25 18:31

email_UTF8_with_emoji.eml (436 Bytes) Sylvain Tissot, 2019-09-26 12:14


Related issues

Related to Redmine - Defect #31900: Problem with tickets issued by rdm-mailhandler on redmine... New

Associated revisions

Revision 18317
Added by Go MAEDA 5 months ago

LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases (#31549).

Patch by Yuichi HARADA.

Revision 18544
Added by Go MAEDA 2 months ago

Incoming emails may cause "invalid byte sequence" error depending on the default external encoding, due to r18317 (#31549).

History

#1 Updated by Yuichi HARADA 6 months ago

I created a patch, so I will attach it.

#2 Updated by Go MAEDA 6 months ago

  • Status changed from New to Confirmed
  • Target version set to Candidate for next minor release

#3 Updated by Pavel Rosický 6 months ago

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

but test fixtures could be saved with different line-endings. On the other hand this change could break emails with binary encodings (attachments), that's why I don't think the patch should be applied.

see related discussion
https://github.com/mikel/mail/pull/1113
and
https://github.com/mikel/mail/pull/1168

#4 Updated by Yuichi HARADA 6 months ago

Pavel Rosický wrote:

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

Thank you for the feedback.
I confirmed the situation. The cause was that when forwarding mail from Postfix to Redmine using rdm-mailhandler.rb, CRLF was replaced with LF. I think that you are configuring Postfix referring to Forwarding emails from your email server .

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

#5 Updated by Yuichi HARADA 6 months ago

Yuichi HARADA wrote:

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

I didn't need encoding, so I rewrote the patch.

#6 Updated by Go MAEDA 6 months ago

  • Target version changed from Candidate for next minor release to 4.0.5

Using the same regular expression as Mail::Utilities.binary_unsafe_to_crlf is faster.

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index 51a0a5afb..16816689f 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email,
+    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,

Benchmark script:

require 'benchmark/ips'

str = "0123456789\r0123456789\r\n0123456789\n" * 1000

Benchmark.ips do |x|
  x.report('/\r?\n|\r\n?/') do
    str.gsub(/\r?\n|\r\n?/, "\r\n")
  end

  x.report('/(?<!\r)\n|\r(?!\n)/') do
    str.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n")
  end

  x.compare!
end

Result:

Warming up --------------------------------------
       /\r?\n|\r\n?/    56.000  i/100ms
/(?<!\r)\n|\r(?!\n)/    73.000  i/100ms
Calculating -------------------------------------
       /\r?\n|\r\n?/    559.559  (± 3.6%) i/s -      2.800k in   5.011150s
/(?<!\r)\n|\r(?!\n)/    731.531  (± 3.0%) i/s -      3.723k in   5.094019s

Comparison:
/(?<!\r)\n|\r(?!\n)/:      731.5 i/s
       /\r?\n|\r\n?/:      559.6 i/s - 1.31x  slower

#7 Updated by Go MAEDA 6 months ago

  • Subject changed from When the line feed code is LF, multi-part mail is misrecognized as single-part mail to Line terminators in an email are LF instead of CRLF when rdm-mailhandler.rb is invoked from /etc/aliases and it causes misparse of the email

#8 Updated by Go MAEDA 5 months ago

  • Subject changed from Line terminators in an email are LF instead of CRLF when rdm-mailhandler.rb is invoked from /etc/aliases and it causes misparse of the email to LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases
  • Status changed from Confirmed to Closed
  • Assignee set to Go MAEDA
  • Target version changed from 4.0.5 to 4.1.0

Committed the patch. Thank you for your contribution.

#9 Updated by Go MAEDA 3 months ago

  • Related to Defect #31900: Problem with tickets issued by rdm-mailhandler on redmine 4.0.4 added

#10 Updated by Sylvain Tissot 2 months ago

Using Postfix 3.3.0 alias to forward pipe the attached message using the latest rdm-mailhandler, I have the opposite problem: it used to work before the patch, but since the patch I have the error:

Status: 5.3.0
Diagnostic-Code: x-unix; /etc/postfix/rdm-mailhandler.rb:156:in `gsub': invalid
    byte sequence in US-ASCII (ArgumentError)  from
    /etc/postfix/rdm-mailhandler.rb:156:in `submit'  from
    /etc/postfix/rdm-mailhandler.rb:215:in `<main>'

The message piped by Postfix is UTF-8, with LF endings

If I manually convert the message from LF to CRLF before piping to rdm-mailhandler, it works.

cat mail_in_utf8_with_lf.eml | unix2dos | /etc/postfix/rdm-mailhandler.rb --project test --url https://redmine.example.com --key ***** --tracker=Maintenance --allow-override=project,tracker,category,priority,status

I've tried the following `/etc/aliases` config but it doesn't work (same error with gsub:156):

support: "|unix2dos | /etc/postfix/rdm-mailhandler.rb --project test --url https://redmine.example.com --key ***** --tracker=Maintenance --allow-override=project,tracker,category,priority,status" 

#11 Updated by Go MAEDA 2 months ago

  • Status changed from Closed to Reopened

#12 Updated by Go MAEDA 2 months ago

  • Status changed from Reopened to Needs feedback

Sylvain Tissot wrote:

Using Postfix 3.3.0 alias to forward pipe the attached message using the latest rdm-mailhandler, I have the opposite problem: it used to work before the patch, but since the patch I have the error:

[...]

Thank you for reporting the error.

Maybe it is because the value of LANG environment variable on your server is set to the locale which does not support UTF-8. A possible value is "C". Ruby's Encoding.default_external respects LANG (see https://docs.ruby-lang.org/en/2.6.0/Encoding.html#class-Encoding-label-External+encoding). If the LANG is "C", Ruby considers the encoding of the content of files as US-ASCII, not UTF-8.

Could you try again after applying the following patch against the latest rdm-mailhandler.rb?

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index b7284841d..62c658e09 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
+    data = { 'key' => key, 'email' => email.b.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,

#13 Updated by Sylvain Tissot 2 months ago

Hi Go MAEDA,

The LANG environment on the server is en_US.utf8

root@mail:/etc/postfix# echo $LANG
en_US.utf8

Thanks for your help, I applied the patch in on the latest rdm-mailhandler.rb and tried with a bunch of differents emails. Messages in UTF-8 are now working, as well as attachments.

Only the attached message containing an unicode pizza U+1F355) is bouncing with this error:

permission denied. Command output: Request was denied by your Redmine
    server. Possible reasons: email is sent from an invalid email address or is
    missing some information.

#14 Updated by Go MAEDA 2 months ago

  • Status changed from Needs feedback to Closed

Sylvain Tissot wrote:

Thanks for your help, I applied the patch in on the latest rdm-mailhandler.rb and tried with a bunch of differents emails. Messages in UTF-8 are now working, as well as attachments.

Thank you for your helpful feedback. I have just committed the fix in r18544.

Only the attached message containing an unicode pizza U+1F355) is bouncing with this error:

I think the error is irrelevant to this issue. Please open a new issue.

Also available in: Atom PDF