Defect #17096

Issue emails cannot be threaded by some mailers due to inconsistent Message-ID and References field

Added by Toshi MARUYAMA about 5 years ago. Updated 6 months ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Go MAEDA% Done:

0%

Category:Email notifications
Target version:4.1.0
Resolution:Fixed Affected version:2.4.4

Description

Redmine 1.4:

Issue adding:

Message-Id: <redmine.issue-1.20140605133813@example.net>

Issue editing:

Message-Id: <redmine.journal-1.20140605134129@example.net>
References: <redmine.issue-1.20140605133813@example.net>

Redmine 2.4:

Issue adding:

Message-ID: <redmine.issue-7.20140605044637.b10ff7cb511bccec@example.net>
References: <redmine.issue-7.20140605044637@example.net>

Issue editing:

Message-ID: <redmine.journal-8.20140605044912.bdbae389b638e71d@example.net>
References: <redmine.issue-7.20140605044637@example.net>

Redmine 2.4 issue adding "Message-ID" and editing "References" are not equal.

So, some mailers cannot thread issue mails.

mail-17096-01.diff Magnifier (4.64 KB) Toshi MARUYAMA, 2014-06-05 15:18

mail-17096-20140608.diff Magnifier (4.71 KB) Toshi MARUYAMA, 2014-06-08 11:47

17096-fix-emails-threading.patch Magnifier (7.58 KB) Go MAEDA, 2019-02-17 14:56

17096-fix-emails-threading-v2.patch Magnifier (7.48 KB) Go MAEDA, 2019-03-03 10:01


Related issues

Related to Redmine - Feature #5037: Role-based issue custom field visibility Closed 2010-03-10
Related to Redmine - Feature #26791: Send individual notification mails per mail recipient Closed
Related to Redmine - Feature #13111: New setting to include the status changes in issue mail n... Closed 2008-02-04

Associated revisions

Revision 17927
Added by Go MAEDA 6 months ago

Issue emails cannot be threaded by some mailers due to inconsistent Message-ID and References field (#17096).

Patch by Go MAEDA.

History

#1 Updated by Toshi MARUYAMA about 5 years ago

  • Description updated (diff)

#2 Updated by Toshi MARUYAMA about 5 years ago

  • Subject changed from Issue mails cannot be sorting to Issue mails cannot be threaded
  • Description updated (diff)

#3 Updated by Toshi MARUYAMA about 5 years ago

Behavior changed by #5037 r12012.

test/unit/mailer_test.rb

@@ -231,19 +231,21 @@
   end

   def test_issue_add_message_id
-    issue = Issue.find(1)
-    Mailer.issue_add(issue).deliver
+    issue = Issue.find(2)
+    Mailer.deliver_issue_add(issue)
     mail = last_email
-    assert_equal Mailer.message_id_for(issue), mail.message_id
-    assert_nil mail.references
+    assert_match /^redmine\.issue-2\.20060719190421\.[a-f0-9]+@example\.net/, mail.message_id
+    assert_include "redmine.issue-2.20060719190421@example.net", mail.references
   end

   def test_issue_edit_message_id
-    journal = Journal.find(1)
-    Mailer.issue_edit(journal).deliver
+    journal = Journal.find(3)
+    journal.issue = Issue.find(2)
+
+    Mailer.deliver_issue_edit(journal)
     mail = last_email
-    assert_equal Mailer.message_id_for(journal), mail.message_id
-    assert_include Mailer.message_id_for(journal.issue), mail.references
+    assert_match /^redmine\.journal-3\.\d+\.[a-f0-9]+@example\.net/, mail.message_id
+    assert_include "redmine.issue-2.20060719190421@example.net", mail.references
     assert_select_email do
       # link to the update
       assert_select "a[href=?]",

#4 Updated by Toshi MARUYAMA about 5 years ago

  • Target version set to 2.6.0

#5 Updated by Toshi MARUYAMA about 5 years ago

  • Target version changed from 2.6.0 to 2.5.2

#6 Updated by Toshi MARUYAMA about 5 years ago

This is patch to fix.

#7 Updated by Toshi MARUYAMA about 5 years ago

This is new patch.
In note 6 patch, mail receiver can guess user id.

#8 Updated by Jean-Philippe Lang about 5 years ago

  • Subject changed from Issue mails cannot be threaded to Issue mails cannot be threaded by some mailers

As far as I can see, this patch only works if the same users are notified during all the issue thread. But this is not the case.

#9 Updated by Toshi MARUYAMA about 5 years ago

Jean-Philippe Lang wrote:

As far as I can see, this patch only works if the same users are notified during all the issue thread. But this is not the case.

Yes, it is same behavior before #5037.

Note-7 patch message id:

Issue adding:

Message-ID: <redmine.issue-72.20140627064350.1ebc022e@example.net>

Issue editing:

Message-ID: <redmine.journal-142.20140627064557.1ebc022e@example.net>
References: <redmine.issue-72.20140627064350.1ebc022e@example.net>

#10 Updated by Jean-Philippe Lang about 5 years ago

No it's not the same behaviour as before #5037. Before #5037, we always had a single email notification, the first message-id and all references were fixed, eg. <redmine.issue-72.20140627064350@example.net>.

With the patch in note-7 applied, the references header changes if a new custom field is added or if you change the permissions on custom fields. And that breaks all the thread. Exemple:

Issue creation:

Message-ID: <redmine.issue-16507.20140705102246.3ecc5e01@redmine.org>

First update (OK):

Message-ID: <redmine.journal-55676.20140705102328.3ecc5e01@redmine.org>
References: <redmine.issue-16507.20140705102246.3ecc5e01@redmine.org>

After adding a new custom field, you get:

Message-ID: <redmine.journal-55677.20140705102736.49bb14f7@redmine.org>
References: <redmine.issue-16507.20140705102246.49bb14f7@redmine.org>

#11 Updated by Toshi MARUYAMA about 5 years ago

Jean-Philippe Lang wrote:

With the patch in note-7 applied, the references header changes if a new custom field is added or if you change the permissions on custom fields. And that breaks all the thread.

Yes. But, it keeps while custom field condition.
In other case before #5037, if user not get issue creating mail, then by watching user get issue editing mail, mail thread is broken on some mailer.

This issue is mail thead broken on some mail in spĂ­te of getting issue creating mail and not changing custom field condition.

#12 Updated by Jean-Philippe Lang about 5 years ago

  • Target version changed from 2.5.2 to Candidate for next major release

#14 Updated by Toshi MARUYAMA about 4 years ago

  • Related to Feature #5037: Role-based issue custom field visibility added

#15 Updated by T E almost 3 years ago

Hi,

are there any news on this? I am having problems with my notification mails, too and would love to see that fixed.

#16 Updated by Lukas Zapletal over 2 years ago

are there any news on this? I am having problems with my notification mails, too and would love to see that fixed.

Yeah is gmail related to this? Having troubles too.

#17 Updated by Go MAEDA almost 2 years ago

I think that adding consistent Message-Id and References filed will be easier if per user notification sending is implemented by #26791. We don't have to use random_hex in Mailer.token_for method. Instead, we can use EmailAddress#id or some hash value generated from an email address.

The following is an example of the idea.

Issue adding:

Message-Id: <redmine.issue-#{object.id}.#{timestamp}.#{email_address.id}@example.net>

Issue editing:

Message-Id: <redmine.journal-#{object.id}.#{timestamp}.#{email_address.id}@example.net>
References: <redmine.issue-#{object.id}.#{timestamp}.#{email_address.id}@example.net>

#18 Updated by Go MAEDA almost 2 years ago

  • Related to Feature #26791: Send individual notification mails per mail recipient added

#19 Updated by Go MAEDA 6 months ago

Go MAEDA wrote:

I think that adding consistent Message-Id and References filed will be easier if per user notification sending is implemented by #26791. We don't have to use random_hex in Mailer.token_for method. Instead, we can use EmailAddress#id or some hash value generated from an email address.

I wrote a patch.

In Redmine 3.4, Redmine may generate multiple notification messages and each message is sent to multiple recipients grouped by custom fields visibility. For example, if there are 6 recipients and they are grouped to 3 groups by custom fields visibility, Redmine generates 3 messages and then pass those to a mail server. Of course, each message has 2 recipients.

Since multiple messages are generated at once when updating one issue, Redmine adds a random key to the message-id to ensure the message-id is unique. Because the random key is never stored, Redmine cannot reproduce the message-id generated in the past and cannot set exact id to the References field in the notification message when the issue is updated. This is why emails are not properly threaded.

In the latest Redmine 4.0, one message is always sent to one recipient. Since one message is never addressed to multiple recipients, we can use recipient user information like user_id. If we use user_id instead of a random key to make a unique message-id, we can set the correct and exact value to References field in the message when the issue is updated because message ids generated in the past is always reproducible. By doing that, emails should be threaded properly.

The attached patch uses user_id to generate unique message-ids.

Issue added:

Message-Id: <redmine.issue-#{issue.id}.#{issue.created_on}.#{user_id}@example.net>

Issue updated:

Message-Id: <redmine.journal-#{jounal.id}.#{journal.created_on}.#{user_id}@example.net>
References: <redmine.issue-#{issue.id}.#{issue.created_on}.#{user_id}@example.net>

#20 Updated by Marius BALTEANU 6 months ago

  • Related to Feature #13111: New setting to include the status changes in issue mail notifications subject added

#21 Updated by Go MAEDA 6 months ago

Updated the patch to apply cleanly to the current trunk (r17913).

#22 Updated by Go MAEDA 6 months ago

  • Target version changed from Candidate for next major release to 4.1.0

Setting the target version to 4.1.0.

#23 Updated by Go MAEDA 6 months ago

  • Subject changed from Issue mails cannot be threaded by some mailers to Notification emails are not threaded in some email clients due to referencing a nonexistent Message-ID

#24 Updated by Go MAEDA 6 months ago

  • Subject changed from Notification emails are not threaded in some email clients due to referencing a nonexistent Message-ID to Issue emails cannot be threaded by some mailers due to inconsistent Message-ID and References field

#25 Updated by Go MAEDA 6 months ago

  • Status changed from New to Closed
  • Assignee set to Go MAEDA
  • Resolution set to Fixed

Committed the patch. The new Message-ID formats are as follows:

Issue added:

Message-Id: <redmine.issue-#{issue.id}.#{issue.created_on}.#{user_id}@example.net>

Issue updated:

Message-Id: <redmine.journal-#{jounal.id}.#{journal.created_on}.#{user_id}@example.net>
References: <redmine.issue-#{issue.id}.#{issue.created_on}.#{user_id}@example.net>

Also available in: Atom PDF