Defect #27691

Issue created from mail_handler with description empty from

Added by Julien TEHERY 13 days ago. Updated 3 days ago.

Status:NewStart date:
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Email receiving
Target version:-
Resolution: Affected version:3.2.8

Description

Env : redmine 3.2.8 - PRODUCTION
OS: Debian 8.9
Mailhandler: https://github.com/redmine/redmine/blob/master/extra/mail_handler/rdm-mailhandler.rb

Migrating our production redmine from 3.2.1 to 3.2.8 , we noticed that some emails with french sepcial characters were creating empty issues.

Here are the details:

- mail is sent to
- on mailserver, we use latest version of rdm-mailhandler.rb which forwards email to our redmine in specific projects.
- on redmine server, we see both succeeding emails and some others who are not correctly transformed into issues.

On production.log, we don't see any particular problem
Here is an example of an email becoming an empty issue:

Started POST "/mail_handler" for 10.123.1.3 at 2017-12-01 11:49:52 +0100
Processing by MailHandlerController#index as HTML
  Parameters: {"key"=>"XIQlZmuj8wzsyWBC4ucB", "email"=>"From julinux@mycompany.fr  Fri Dec  1 11:49:52 2017\nReturn-Path: <julinux@mycompany.fr>\nX-Original-To: ticket@mycompany.fr\nDelivered-To: ticket@mycompany.fr\nReceived: from localhost (unknown [127.0.0.1])\n\tby mail.mycompany.fr (Postfix::smtpd) with ESMTP id 5E1A7559\n\tfor <ticket@mycompany.fr>; Fri,  1 Dec 2017 11:49:52 +0100 (CET)\nDKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=mycompany.fr;\n\ts=mycompany.fr; t=1512125392;\n\tbh=+cEL8GCaNKPGfAl6U7yCNuKL4UTN2UkhX+jOBsmT7iY=;\n\th=To:From:Subject:Date:From;\n\tb=QuPhO2+UuQM8K5Bc8AMNRXC/bC7pTaXq42ambTZz89BZcyzAFKiSZzLdAr99T+o/o\n\t thGNDzH26zgG4oxQK3szqxKzsJgsXijhJPftgDECFWIb5HOiCnb77hbYvBJ5dQ1ECz\n\t BlyasjpiMHrsALRjNAEettV58VMlDhqQaj8J7rSc=\nDKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=mycompany.fr;\n\ts=mycompany.fr; t=1512125392;\n\tbh=+cEL8GCaNKPGfAl6U7yCNuKL4UTN2UkhX+jOBsmT7iY=;\n\th=To:From:Subject:Date:From;\n\tb=QuPhO2+UuQM8K5Bc8AMNRXC/bC7pTaXq42ambTZz89BZcyzAFKiSZzLdAr99T+o/o\n\t thGNDzH26zgG4oxQK3szqxKzsJgsXijhJPftgDECFWIb5HOiCnb77hbYvBJ5dQ1ECz\n\t BlyasjpiMHrsALRjNAEettV58VMlDhqQaj8J7rSc=\nX-Virus-Scanned: Debian amavisd-new at debian8amd64prodOE\nX-Spam-Flag: NO\nX-Spam-Score: -2.998\nX-Spam-Level:\nX-Spam-Status: No, score=-2.998 tagged_above=-999 required=5.5\n\ttests=[ALL_TRUSTED=-1, BAYES_00=-1.9, DKIM_SIGNED=0.1,\n\tDKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001,\n\tURIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no\nReceived: from mail.mycompany.fr ([127.0.0.1])\n\tby localhost (mail.mycompany.fr [127.0.0.1]) (amavisd-new, port 10024)\n\twith LMTP id c9qer7n5h2LZ for <ticket@mycompany.fr>;\n\tFri,  1 Dec 2017 11:49:51 +0100 (CET)\nReceived: from [192.168.103.194] (unknown [109.190.128.27])\n\t(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))\n\t(No client certificate requested)\n\tby mail.mycompany.fr (Postfix::smtpd) with ESMTPSA id 994FA4D1\n\tfor <ticket@mycompany.fr>; Fri,  1 Dec 2017 11:49:51 +0100 (CET)\nDKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=mycompany.fr;\n\ts=mycompany.fr; t=1512125391;\n\tbh=+cEL8GCaNKPGfAl6U7yCNuKL4UTN2UkhX+jOBsmT7iY=;\n\th=To:From:Subject:Date:From;\n\tb=TMesqjQtZ7ut1zYR91GtKcyz6GxYHjO7zoV6tKtAlGlT1K7EQlCC+LX0NVKGBWkgp\n\t mP4FjKVN+W1lTvgeHxFd1zcHDISeC0WGqHv5S5D94PEdaj8bpbluS0IeROPxgo8fhH\n\t IRR0nOqnWXRi15ck3qm7WR61kg/0I02o4MpQSryQ=\nDKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=mycompany.fr;\n\ts=mycompany.fr; t=1512125391;\n\tbh=+cEL8GCaNKPGfAl6U7yCNuKL4UTN2UkhX+jOBsmT7iY=;\n\th=To:From:Subject:Date:From;\n\tb=TMesqjQtZ7ut1zYR91GtKcyz6GxYHjO7zoV6tKtAlGlT1K7EQlCC+LX0NVKGBWkgp\n\t mP4FjKVN+W1lTvgeHxFd1zcHDISeC0WGqHv5S5D94PEdaj8bpbluS0IeROPxgo8fhH\n\t IRR0nOqnWXRi15ck3qm7WR61kg/0I02o4MpQSryQ=\nTo: ticket@mycompany.fr\nFrom: Julien TEHERY <julinux@mycompany.fr>\nSubject: TEST 16\nMessage-ID: <e51fd0a1-b014-ef78-2ff2-4ef38a130388@mycompany.fr>\nDate: Fri, 1 Dec 2017 11:49:51 +0100\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n Thunderbird/52.4.0\nMIME-Version: 1.0\nContent-Type: multipart/alternative;\n boundary=\"------------4858C4CCDE564230BCC36310\"\nContent-Language: fr\n\nThis is a multi-part message in MIME format.\n--------------4858C4CCDE564230BCC36310\nContent-Type: text/plain; charset=utf-8; format=flowed\nContent-Transfer-Encoding: 8bit\n\nTEST 16\n\n-- \nCordialement,\n\nJulien Téhéry - Ingénieur Systèmes et Réseaux\nSociété mycompany - www.mycompany.ff
r <http://www.mycompany.fr>\nTél Support : 05 49 62 26 07\n\n\n--------------4858C4CCDE564230BCC36310\nContent-Type: text/html; charset=utf-8\nContent-Transfer-Encodii
ng: 8bit\n\n<html>\n  <head>\n\n    <meta http-equiv=\"content-type\" content=\"text/html; charset=utf-8\">\n  </head>\n  <body text=\"#000000\" bgcolor=\"#FFFFFF\">\n    <p>TEST 16<br>\n    </p>\n    <div class=\"moz-signature\">-- <br>\n      Cordialement,<br>\n      <br>\n      Julien Téhéry - Ingénieur Systèmes et Réseaux<br>>
\n      Société mycompany - <a href=\"http://www.mycompany.fr\"\n        title=\"mycompany\" target=\"_blank\">www.mycompany.fr</a><br>\n      Tél Suppoo
rt : 05 49 62 26 07<br>\n      <br>\n    </div>\n  </body>\n</html>\n\n--------------4858C4CCDE564230BCC36310--\n", "allow_override"=>"project,tracker,priority,status", "unknown_user"=>"accept", "default_group"=>nil, "no_account_notice"=>nil, "no_notification"=>nil, "no_permission_check"=>"1", "project_from_subaddress"=>nil, "issue"=>{"project"=>"clients-mycompany", "tracker"=>"Support"}}
  Rendered queries/_filters.html.erb (12.5ms)
  Rendered queries/_columns.html.erb (1.8ms)
  Rendered issues/_list.html.erb (35.5ms)
  Rendered issues/_sidebar.html.erb (21.2ms)
  Rendered issues/index.html.erb within layouts/base (81.4ms)
Completed 200 OK in 566ms (Views: 132.4ms | ActiveRecord: 12.4ms)

So I tried to figure out what was different in the incoming emails, looking at the source of them.
Didn't find anything particular excepted that "good emails" had "Content-Transfer-Encoding: quoted-printable" in their source code.

I don't know if this can be related or not to my problem.
But the fact is we did'nt changed anything regarding mail_handler or mailserver, and we lost the ability to get emails from everyone.
I must precise that we allready used latest version of mail_handler (see links above) before upgrading.

Any help would be appreciated !

History

#1 Updated by Julien TEHERY 10 days ago

Here are some precisions about the incoming emails:

Looking at the source code of the emails, here is what i noticed.

  1. BAD INCOMING MAIL TURNED INTO EMPTY ISSUE
    Content-Type: text/plain; charset=utf-8; format=flowed
    Content-Transfer-Encoding: 8bit
  1. GOOD MAIL TURNED INTO FULLFILLED ISSUE
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

I really don't have any idea why 8bit encoding would cause such a behavior, but the fact is that all
bad emails are formated this way.

#2 Updated by Julien TEHERY 10 days ago

Ok, i can confirm this is the cause.

I'm using thunderbird to send emails, and it's configured to transfer mails in 8bits by default.
It has always been the case and worked until the upgrade in 3.2.8.
Now i had to setup thunedrbird to use MIME for outgoing email. With that email arrive with "Content-Transfer-Encoding: quoted-printable" and are well processed by the mail_handler.

I think it's clearly a bug, because it has always worked through the past. I won't tell my customers to change their MTA configuration to send me emails.

If it can help someone...

#3 Updated by Jessie Lee 8 days ago

encountering same parsing bug with extended utf-8 characters. same discovery as comment 1.

When set to unknown_user=create we get

MailHandler: ignoring email from unknown user []
Completed 422 Unprocessable Entity in 4ms (ActiveRecord: 0.0ms)

and when set to unknown_user=accept we get a blank no subject no description email.

example of source of email that causes problem below:

From: REDACTED
Subject: emdash test
To: REDACTED
Message-ID: <eba059c5-7557-9744-61e2-8c52c46cea86@palantetech.coop>
Date: Tue, 5 Dec 2017 13:33:28 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------F142713BE64E3D9019D87845" 
Content-Language: en-US

This is a multi-part message in MIME format.
--------------F142713BE64E3D9019D87845
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

— — — —

--------------F142713BE64E3D9019D87845
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64

PGh0bWw+DQogIDxoZWFkPg0KICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCiAgPC9oZWFkPg0KICA8Ym9k
eSB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIj4NCiAgICA8cD48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUIDwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUIDwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUIDwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYig2OCwgNjgsIDY4KTsgZm9udC1mYW1pbHk6ICZxdW90O09wZW4gU2Fu
cyZxdW90Oywgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMy41cHg7IGZvbnQtc3R5bGU6IG5v
cm1hbDsgZm9udC12YXJpYW50LWxpZ2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fw
czogbm9ybWFsOyBmb250LXdlaWdodDogNDAwOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBv
cnBoYW5zOiAyOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10
cmFuc2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBwcmUtd3JhcDsgd2lkb3dzOiAyOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyBiYWNrZ3Jv
dW5kLWNvbG9yOiByZ2JhKDY4LCA2OCwgNjgsIDAuMDgpOyB0ZXh0LWRlY29yYXRpb24tc3R5
bGU6IGluaXRpYWw7IHRleHQtZGVjb3JhdGlvbi1jb2xvcjogaW5pdGlhbDsgZGlzcGxheTog
aW5saW5lICFpbXBvcnRhbnQ7IGZsb2F0OiBub25lOyI+4oCUPC9zcGFuPjwvcD4NCiAgICA8
cHJlIGNsYXNzPSJtb3otc2lnbmF0dXJlIiBjb2xzPSI3MiI+LS0gDQpKZXNzaWUgUm9zZSBM
ZWUNClBhbGFudGUgVGVjaG5vbG9neSBDb29wZXJhdGl2ZQ0KPGEgY2xhc3M9Im1vei10eHQt
bGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL3BhbGFudGV0ZWNoLmNvb3AiPmh0dHA6Ly9w
YWxhbnRldGVjaC5jb29wPC9hPg0KKDg4OCkgNDgxLTM2NTUgeC4gMTA2PC9wcmU+DQogIDwv
Ym9keT4NCjwvaHRtbD4NCg==
--------------F142713BE64E3D9019D87845--

current redmine version:

Environment:
  Redmine version                3.4.3.stable
  Ruby version                   2.2.5-p319 (2016-04-26) [x86_64-linux]
  Rails version                  4.2.8
  Environment                    production
  Database adapter               Mysql2
SCM:
  Bazaar                         2.7.0
  Git                            2.1.4
  Filesystem                     
  GitRemote                      2.1.4
Redmine plugins:
  clipboard_image_paste          1.12
  redmine_git_remote             0.0.1
  redmine_local_avatars          1.0.3
  redmine_mattermost             0.6
  redmine_stealth                0.6.0
  redmine_wiki_extensions        0.8.1

MTA has same version of ruby

#4 Updated by Jessie Lee 7 days ago

ran irb rdm-mailhandler.rb

handler = RedmineMailHandler.new
#<RedmineMailHandler:0x000000010b9c48 @issue_attributes={"project"=>"catch", "tracker"=>"support"}, @verbose=true, @url="REDACTED", @key="REDACTED">
exit(handler.submit(STDIN.read))
Posting to https://REDACTED/mail_handler...
Response received: 422
Request was denied by your Redmine server. Possible reasons: email is sent from an invalid email address or is missing some information.
                        

#5 Updated by Michael Schaefer 3 days ago

In can confirm the issue in 3.4.3.stable. The issue didn't occur in 3.4.1. After updating to 3.4.3 I'm experiencing the same issue as mentioned by Jessie Lee.
I ran a few quick test and for the exact same E-Mail I get an

MailHandler: ignoring email from unknown user []

if

Content-Transfer-Encoding: 8bit

while

Content-Transfer-Encoding: 7bit

or

Content-Transfer-Encoding: quoted-printable

works as expected an opens a ticket for the correct user. However I seems to depend on the content itself. Not all message with the 8bit enconding get bounced.

Environment:
  Redmine version                3.4.3.stable
  Ruby version                   2.3.1-p112 (2016-04-26) [x86_64-freebsd10.3]
  Rails version                  4.2.8
  Environment                    production
  Database adapter               Mysql2
SCM:
  Git                            2.14.1
  Filesystem                     
Redmine plugins:
  redmine_checklists             3.1.10
  redmine_default_custom_query   1.3.0
  redmine_ldap_sync              2.0.9.devel

#6 Updated by Michael Schaefer 3 days ago

I have to correct myself. It seems that also for 7bit encoding I can trigger the bug. If for instance sending a mail which contains a German Umlaut äüö...

Also available in: Atom PDF