Project

General

Profile

Actions

Defect #31549

closed

LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases

Added by Yuichi HARADA almost 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Email receiving
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Resolution:
Affected version:

Description

The issue's description will be garbled if you receive the following email.

  • The multi-part mail.
  • The charset of the text-part is UTF-8.
  • The text-part contains multibyte-character .
  • File is attached.
  • Line terminator is LF (0x0a) .

Files

fixed-31549.patch (3.22 KB) fixed-31549.patch Yuichi HARADA, 2019-06-11 05:08
fixed-31549-v2.patch (604 Bytes) fixed-31549-v2.patch Yuichi HARADA, 2019-06-13 07:20
fixed-31549-v3.patch (621 Bytes) fixed-31549-v3.patch Yuichi HARADA, 2019-06-20 08:41
mail_to_redmine (562 Bytes) mail_to_redmine Sylvain Tissot, 2019-09-25 18:31
email_UTF8_with_emoji.eml (436 Bytes) email_UTF8_with_emoji.eml Sylvain Tissot, 2019-09-26 12:14

Related issues

Related to Redmine - Defect #31900: Problem with tickets issued by rdm-mailhandler on redmine 4.0.4New

Actions
Actions #1

Updated by Yuichi HARADA almost 5 years ago

I created a patch, so I will attach it.

Actions #2

Updated by Go MAEDA almost 5 years ago

  • Status changed from New to Confirmed
  • Target version set to Candidate for next minor release
Actions #3

Updated by Pavel Rosický almost 5 years ago

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

but test fixtures could be saved with different line-endings. On the other hand this change could break emails with binary encodings (attachments), that's why I don't think the patch should be applied.

see related discussion
https://github.com/mikel/mail/pull/1113
and
https://github.com/mikel/mail/pull/1168

Actions #4

Updated by Yuichi HARADA almost 5 years ago

Pavel Rosický wrote:

have you encountered with this problem for real? according to https://tools.ietf.org/html/rfc6152 line-endings should be always CRLF

Thank you for the feedback.
I confirmed the situation. The cause was that when forwarding mail from Postfix to Redmine using rdm-mailhandler.rb, CRLF was replaced with LF. I think that you are configuring Postfix referring to Forwarding emails from your email server .

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

Actions #5

Updated by Yuichi HARADA almost 5 years ago

Yuichi HARADA wrote:

I rewrote the patch of rdm-mailhandler.rb. Please delete the previous patch. ( fixed-31549.patch )

I didn't need encoding, so I rewrote the patch.

Actions #6

Updated by Go MAEDA almost 5 years ago

  • Target version changed from Candidate for next minor release to 4.0.5

Using the same regular expression as Mail::Utilities.binary_unsafe_to_crlf is faster.

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index 51a0a5afb..16816689f 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email,
+    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,

Benchmark script:

require 'benchmark/ips'

str = "0123456789\r0123456789\r\n0123456789\n" * 1000

Benchmark.ips do |x|
  x.report('/\r?\n|\r\n?/') do
    str.gsub(/\r?\n|\r\n?/, "\r\n")
  end

  x.report('/(?<!\r)\n|\r(?!\n)/') do
    str.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n")
  end

  x.compare!
end

Result:

Warming up --------------------------------------
       /\r?\n|\r\n?/    56.000  i/100ms
/(?<!\r)\n|\r(?!\n)/    73.000  i/100ms
Calculating -------------------------------------
       /\r?\n|\r\n?/    559.559  (± 3.6%) i/s -      2.800k in   5.011150s
/(?<!\r)\n|\r(?!\n)/    731.531  (± 3.0%) i/s -      3.723k in   5.094019s

Comparison:
/(?<!\r)\n|\r(?!\n)/:      731.5 i/s
       /\r?\n|\r\n?/:      559.6 i/s - 1.31x  slower

Actions #7

Updated by Go MAEDA almost 5 years ago

  • Subject changed from When the line feed code is LF, multi-part mail is misrecognized as single-part mail to Line terminators in an email are LF instead of CRLF when rdm-mailhandler.rb is invoked from /etc/aliases and it causes misparse of the email
Actions #8

Updated by Go MAEDA almost 5 years ago

  • Subject changed from Line terminators in an email are LF instead of CRLF when rdm-mailhandler.rb is invoked from /etc/aliases and it causes misparse of the email to LF line terminators cause misparse of a multi-part email when rdm-mailhandler.rb is invoked from /etc/aliases
  • Status changed from Confirmed to Closed
  • Assignee set to Go MAEDA
  • Target version changed from 4.0.5 to 4.1.0

Committed the patch. Thank you for your contribution.

Actions #9

Updated by Go MAEDA over 4 years ago

  • Related to Defect #31900: Problem with tickets issued by rdm-mailhandler on redmine 4.0.4 added
Actions #10

Updated by Sylvain Tissot over 4 years ago

Using Postfix 3.3.0 alias to forward pipe the attached message using the latest rdm-mailhandler, I have the opposite problem: it used to work before the patch, but since the patch I have the error:

Status: 5.3.0
Diagnostic-Code: x-unix; /etc/postfix/rdm-mailhandler.rb:156:in `gsub': invalid
    byte sequence in US-ASCII (ArgumentError)  from
    /etc/postfix/rdm-mailhandler.rb:156:in `submit'  from
    /etc/postfix/rdm-mailhandler.rb:215:in `<main>'

The message piped by Postfix is UTF-8, with LF endings

If I manually convert the message from LF to CRLF before piping to rdm-mailhandler, it works.

cat mail_in_utf8_with_lf.eml | unix2dos | /etc/postfix/rdm-mailhandler.rb --project test --url https://redmine.example.com --key ***** --tracker=Maintenance --allow-override=project,tracker,category,priority,status

I've tried the following `/etc/aliases` config but it doesn't work (same error with gsub:156):

support: "|unix2dos | /etc/postfix/rdm-mailhandler.rb --project test --url https://redmine.example.com --key ***** --tracker=Maintenance --allow-override=project,tracker,category,priority,status" 
Actions #11

Updated by Go MAEDA over 4 years ago

  • Status changed from Closed to Reopened
Actions #12

Updated by Go MAEDA over 4 years ago

  • Status changed from Reopened to Needs feedback

Sylvain Tissot wrote:

Using Postfix 3.3.0 alias to forward pipe the attached message using the latest rdm-mailhandler, I have the opposite problem: it used to work before the patch, but since the patch I have the error:

[...]

Thank you for reporting the error.

Maybe it is because the value of LANG environment variable on your server is set to the locale which does not support UTF-8. A possible value is "C". Ruby's Encoding.default_external respects LANG (see https://docs.ruby-lang.org/en/2.6.0/Encoding.html#class-Encoding-label-External+encoding). If the LANG is "C", Ruby considers the encoding of the content of files as US-ASCII, not UTF-8.

Could you try again after applying the following patch against the latest rdm-mailhandler.rb?

diff --git a/extra/mail_handler/rdm-mailhandler.rb b/extra/mail_handler/rdm-mailhandler.rb
index b7284841d..62c658e09 100644
--- a/extra/mail_handler/rdm-mailhandler.rb
+++ b/extra/mail_handler/rdm-mailhandler.rb
@@ -153,7 +153,7 @@ END_DESC

     headers = { 'User-Agent' => "Redmine mail handler/#{VERSION}" }

-    data = { 'key' => key, 'email' => email.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
+    data = { 'key' => key, 'email' => email.b.gsub(/(?<!\r)\n|\r(?!\n)/, "\r\n"),
                            'allow_override' => allow_override,
                            'unknown_user' => unknown_user,
                            'default_group' => default_group,
Actions #13

Updated by Sylvain Tissot over 4 years ago

Hi Go MAEDA,

The LANG environment on the server is en_US.utf8

root@mail:/etc/postfix# echo $LANG
en_US.utf8

Thanks for your help, I applied the patch in on the latest rdm-mailhandler.rb and tried with a bunch of differents emails. Messages in UTF-8 are now working, as well as attachments.

Only the attached message containing an unicode pizza U+1F355) is bouncing with this error:

permission denied. Command output: Request was denied by your Redmine
    server. Possible reasons: email is sent from an invalid email address or is
    missing some information.
Actions #14

Updated by Go MAEDA over 4 years ago

  • Status changed from Needs feedback to Closed

Sylvain Tissot wrote:

Thanks for your help, I applied the patch in on the latest rdm-mailhandler.rb and tried with a bunch of differents emails. Messages in UTF-8 are now working, as well as attachments.

Thank you for your helpful feedback. I have just committed the fix in r18544.

Only the attached message containing an unicode pizza U+1F355) is bouncing with this error:

I think the error is irrelevant to this issue. Please open a new issue.

Actions

Also available in: Atom PDF