Feature #30838

Option to parse HTML part of multipart (HTML) emails first

Added by Go MAEDA about 1 month ago. Updated 14 days ago.

Status:ReopenedStart date:
Priority:NormalDue date:
Assignee:Go MAEDA% Done:

0%

Category:Email receiving
Target version:4.1.0
Resolution:Fixed

Description

Currently, when mail_handler receives a multipart email that contains both text and HTML part, it tries to retrieve text parts first. HTML parts are processed only when the email does not have any text parts.

However, in 2019, unlike in 2007 which is the year when mail_handler was introduced, HTML mail is becoming increasingly mainstream. Although most emails still have both text and HTML parts, the text part of some email is useless. For example, a simple message "Please open with a mailer which supports HTML emails" or auto-generated text from the HTML part. In such cases, we can get better text by parsing the HTML part with MailHandler.html_body_to_text method than retrieving the content of the text part.

In order to cope with such circumstance, I propose to add an option "Preferred part in HTML emails" to set the parsing order of multipart emails. The setting has two options, "Text" and "HTML". The first option is the default and keeps the current behavior of mail_handler (that means mail_handler tries to get text part first). When Admin chooses the second option "HTML", mail_handler tries to get HTML part first and converts the HTML to plain text. Text part is retrieved only when the email does not have HTML part.

  • Setting name: "Preferred part in HTML emails"
  • Available options: "Text" (Default), "HTML"

I am sure that this feature is really beneficial for users who use the email receiving feature. Since Admin can determine the behavior and the default does not change the current behavior at all, this feature never causes compatibility problems.

add_settings_incoming_emails.png (106 KB) Yuichi HARADA, 2019-02-22 05:46

30838-preferred-part-multipart-email.patch Magnifier (5.52 KB) Yuichi HARADA, 2019-02-22 05:51

30838-preferred-part-multipart-email-without-gui.patch Magnifier - Use configuration.yml instead of adding a new setting to the admin page (3.43 KB) Go MAEDA, 2019-02-26 03:03

Associated revisions

Revision 17913
Added by Go MAEDA 20 days ago

Option to parse HTML part of multipart (HTML) emails first (#30838).

Patch by Yuichi HARADA.

Revision 17922
Added by Go MAEDA 19 days ago

Update locales (#13111, #30838).

History

#1 Updated by Go MAEDA about 1 month ago

  • Subject changed from Option to parse HTML part of emails first to Option to parse HTML part of multipart (HTML) emails first

#2 Updated by Yuichi HARADA 29 days ago

Go MAEDA wrote:

In order to cope with such circumstance, I propose to add an option "Preferred part in HTML emails" to set the parsing order of multipart emails. The setting has two options, "Text" and "HTML". The first option is the default and keeps the current behavior of mail_handler (that means mail_handler tries to get text part first). When Admin chooses the second option "HTML", mail_handler tries to get HTML part first and converts the HTML to plain text. Text part is retrieved only when the email does not have HTML part.

  • Setting name: "Preferred part in HTML emails"
  • Available options: "Text" (Default), "HTML"

The "Preferred part of multipart (HTML) emails" has been added to [Administration] > [Settings] > [Incoming emails].
I attached a patch.

#3 Updated by Yuichi HARADA 29 days ago

I forgot to attach a patch.

#4 Updated by Marius BALTEANU 29 days ago

Do you see any side effect if we switch to HTML part first without adding a new setting?

#5 Updated by Go MAEDA 28 days ago

  • Assignee set to Marius BALTEANU

Marius BALTEANU wrote:

Do you see any side effect if we switch to HTML part first without adding a new setting?

Maybe it is OK to read an HTML part first by default. It works without a big problem, I think. But I feel anxiety that some users will encounter unexpected behavior of the HTML email parser or some users don't like that converted text has Textile/Markdown markups such as bold, italic, underlines, and headings.

Therefore, I think we had better add an option to control which part is extracted. It does not have to be in the admin GUI. Adding a setting in the configuration.yml file is also OK.

The following is a summary of my post. Marius, what do you think?

  • Reading an HTML part first is OK
  • We had better have an option to control the preferred part in the admin GUI or configuration.yml.

#6 Updated by Marius BALTEANU 28 days ago

Go MAEDA wrote:

The following is a summary of my post. Marius, what do you think?

  • Reading an HTML part first is OK
  • We had better have an option to control the preferred part in the admin GUI or configuration.yml.

I agree, it is safer.

#7 Updated by Go MAEDA 25 days ago

I modified the patch not to add a setting on the admin page. Instead, it reads the setting from configuration.yml. We can avoid increasing the complexity of the admin page.

I think that this setting should not be changed often or easily, so the configuration.yml may be a good place to put the setting.

We may change the default value to 'html' in the future version of Redmine but I think it is safer to stay in 'plain' in Redmine 4.1 because this unreleased feature is not widely tested and used yet in the production environments. Maybe it is better to mark this feature as "Experimental" like Markdown formatter introduced in Redmine 2.5 (#15520).

#8 Updated by Go MAEDA 23 days ago

  • Target version changed from Candidate for next major release to 4.1.0

Setting the target version to 4.1.0.

#9 Updated by Holger Just 23 days ago

If this setting is supposed to be configureable, I do think that it belongs on the Settings page rather than the configuration.yml file.

When considering the distinction between the "server administrator" who is responsible for the software itself and the Redmine administrator who is responsible for the projects and communication, I don't see this setting in the realm of the server administrator. Instead, i think it should be up to the Redmine administrator to decide based on the emails they receive whether to prefer one or the other option.

As for the feature itself, I'm in favour of seeing it added to Redmine (although I hate that it is apparently necessary in the first place due to broken email senders who abuse perfectly fine technology without any gain)

#10 Updated by Go MAEDA 22 days ago

Holger Just wrote:

If this setting is supposed to be configureable, I do think that it belongs on the Settings page rather than the configuration.yml file.

Thank you for the advice. It makes sense. I attached 30838-preferred-part-multipart-email-without-gui.patch to avoid increasing the complexity of the setting page, but now I think the setting should be on the page.

#11 Updated by Go MAEDA 20 days ago

  • Status changed from New to Closed
  • Assignee set to Go MAEDA
  • Resolution set to Fixed

Committed the patch. Thank you for improving Redmine.

#12 Updated by Go MAEDA 20 days ago

  • Category changed from Email notifications to Email receiving

#13 Updated by Go MAEDA 15 days ago

  • Status changed from Closed to Reopened

The caption of the options for the setting is hardcoded in source:trunk/app/views/settings/_mail_handler.html.erb@17913#L21 and cannot be localized.

#14 Updated by Marius BALTEANU 14 days ago

Go MAEDA wrote:

We may change the default value to 'html' in the future version of Redmine but I think it is safer to stay in 'plain' in Redmine 4.1 because this unreleased feature is not widely tested and used yet in the production environments. Maybe it is better to mark this feature as "Experimental" like Markdown formatter introduced in Redmine 2.5 (#15520).

I completely agree, let mark this feature as "Experimental".

Also available in: Atom PDF