Defect #16122

redmine:email:receive should not abort with buggy mail

Added by Vedran Rodic almost 6 years ago. Updated almost 5 years ago.

Status:NewStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Email receiving
Target version:-
Resolution: Affected version:

Description

Hi

I'm testing issue creation via email with redmine-2.4.3. It works fine when I have plain Ascii mails, but fails with Croatian characters. Gmail chooses to use latin2 for them, and that works fine in all other apps, but not with redmine 2.4.3 when receiving email. I've attached the .eml file that gmail sends to our mail server.

I've also tried with patch in http://www.redmine.org/projects/redmine/repository/revisions/12474, but issue is still present.

rake -f ~/redmine-2.4.3/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=xxxxxxx username=xxxxx password=xxxx project=operativa tracker=Bug allow_override=tracker,priority

rake aborted!
"\xB9" from ASCII-8BIT to UTF-8
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/fields/unstructured_field.rb:149:in `encode!'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/fields/unstructured_field.rb:149:in `fold'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/fields/unstructured_field.rb:103:in `wrapped_value'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/fields/unstructured_field.rb:69:in `do_encode'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/fields/unstructured_field.rb:51:in `encoded'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/field.rb:167:in `method_missing'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/header.rb:206:in `block in encoded'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/header.rb:205:in `each'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/header.rb:205:in `encoded'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/mail-2.5.4/lib/mail/message.rb:1801:in `encoded'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/actionmailer-3.2.16/lib/action_mailer/base.rb:434:in `set_payload_for_mail'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/actionmailer-3.2.16/lib/action_mailer/base.rb:403:in `block in receive'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/activesupport-3.2.16/lib/active_support/notifications.rb:123:in `block in instrument'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/activesupport-3.2.16/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/activesupport-3.2.16/lib/active_support/notifications.rb:123:in `instrument'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/gems/actionmailer-3.2.16/lib/action_mailer/base.rb:401:in `receive'
/home/redmine/redmine-2.4.3/app/models/mail_handler.rb:46:in `receive'
/home/redmine/redmine-2.4.3/lib/redmine/imap.rb:35:in `block in check'
/home/redmine/redmine-2.4.3/lib/redmine/imap.rb:32:in `each'
/home/redmine/redmine-2.4.3/lib/redmine/imap.rb:32:in `check'
/home/redmine/redmine-2.4.3/lib/tasks/email.rake:125:in `block (3 levels) in <top (required)>'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/bin/ruby_executable_hooks:15:in `eval'
/home/redmine/.rvm/gems/ruby-1.9.3-p484/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => redmine:email:receive_imap
(See full trace by running task with --trace)

Please tell if you need more data or testing.

Issue looks very similar as the one here: http://stackoverflow.com/questions/19450199/redmine-receive-mail-field


Related issues

Related to Redmine - Patch #29614: redmine:email:read and rdm-mailhandler.rb should use safe... Closed
Duplicated by Redmine - Defect #16261: tasks by email Closed
Duplicated by Redmine - Defect #18649: mail_handler.rb - does not encode non-utf8 characters Closed

Associated revisions

Revision 12903
Added by Toshi MARUYAMA almost 6 years ago

add test of gmail ISO-8859-2 mail receiving (#16122)

Revision 12904
Added by Toshi MARUYAMA almost 6 years ago

fix newline of fixture (#16122)

Revision 12905
Added by Toshi MARUYAMA almost 6 years ago

svn propset svn:eol-style native fixture (#16122)

Revision 12995
Added by Jean-Philippe Lang over 5 years ago

Don't abort processing of emails when an email triggers a parsing exception (#16122).

History

#1 Updated by Toshi MARUYAMA almost 6 years ago

  • File deleted (gmail-iso8859-2.eml)

#2 Updated by Toshi MARUYAMA almost 6 years ago

  • Category set to Email receiving

#3 Updated by Toshi MARUYAMA almost 6 years ago

I removed attachment because it includes private data.

#4 Updated by Toshi MARUYAMA almost 6 years ago

  • Status changed from New to Needs feedback
  • Resolution set to Duplicate

I added test and test passes.

#5 Updated by Vedran Rodic almost 6 years ago

When trying with my original file it doesn't pass, but that might be a test infrastructure problem.

Your file passes the test, so I guess the test isn't complete. I'll try to investigate by printing out what we get from our email system but I'm a ruby newbie and would like to have some advice on how to proceed.

Btw, this is the output when running with my original .eml file, how to get a full trace of the failure instead of just this:
~/redmine-2.4.3]# ruby test/unit/mail_handler_test.rb -n test_gmail_iso8859_2
Run options: -n test_gmail_iso8859_2

  1. Running tests:

F

Finished tests in 3.561271s, 0.2808 tests/s, 0.2808 assertions/s.

1) Failure:
test_gmail_iso8859_2(MailHandlerTest) [test/unit/mail_handler_test.rb:535]:
Expected false to be a kind of Issue, not FalseClass.

1 tests, 1 assertions, 1 failures, 0 errors, 0 skips

#6 Updated by Vedran Rodic almost 6 years ago

Okay, my original report is not complete.

I added puts(msg) in imap.rb to see the complete message we get get from our system and it contained this (containing part of my signature):

X-Ham-Report: Spam detection software, running on the system "xxxxxxx", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
root\@localhost for details.

Content preview:  -- Vedran Rodi� Software developer Tau on-line d.o.o. Hondlova
2/11, HR-10000 Zagreb Phone+385 (1) xxxxxxx [...]
X-Spam-Flag: NO

Now, this � is not in any encoding mentioned anywhere.

This is added by our SpamAssassin setup, which might be buggy.

I have a feeling that redmine shouldn't just abort processing this message in these cases, but it's your choice on how to handle this.

#7 Updated by Toshi MARUYAMA almost 6 years ago

  • Subject changed from error with redmine:email:receive when receiving ISO-8859-2 mails mailed from gmail to redmine:email:receive should not abort with buggy mail
  • Status changed from Needs feedback to New
  • Resolution deleted (Duplicate)

#8 Updated by Toshi MARUYAMA over 5 years ago

#9 Updated by Andre Oliveira about 5 years ago

I solved that problem in my company by forcing the encoding...

In redmine_folder/app/models/mail_handler.rb

add:
email.force_encoding('UTF-8')

after:
def self.receive(email, options={})

and remove:
email.force_encoding('ASCII-8BIT') if email.respond_to?(:force_encoding)

Hope it helps

#10 Updated by Toshi MARUYAMA about 5 years ago

Luis Fontes wrote:

I solved that problem in my company by forcing the encoding...

In redmine_folder/app/models/mail_handler.rb

add:
email.force_encoding('UTF-8')

Could you try this instead?

email.force_encoding('ASCII-8BIT')

Because Rails 4 porting has this code.
source:trunk/app/models/mail_handler.rb@13623#L45

after
def self.receive(email, options={})

#11 Updated by Toshi MARUYAMA almost 5 years ago

  • Duplicated by Defect #18649: mail_handler.rb - does not encode non-utf8 characters added

#12 Updated by Toshi MARUYAMA almost 5 years ago

FTR:
In #18649, there is request that Redmine should not ignore but replace invalid characters.

#13 Updated by Andre Oliveira almost 5 years ago

Toshi MARUYAMA wrote:

Could you try this instead?

email.force_encoding('ASCII-8BIT')

Because Rails 4 porting has this code.
source:trunk/app/models/mail_handler.rb@13623#L45

That did not work in my case,I forgot to tell to remove that line.

When I did this changes emails without subject were causing errors too, because an Issue needs to have the subject, so if anyone has this problem too just need to add

issue.subject = '(no subject)' if issue.subject.blank?

after
issue.subject = cleaned_up_subject

(this is in the receive_issue method)

#14 Updated by Go MAEDA about 1 year ago

  • Related to Patch #29614: redmine:email:read and rdm-mailhandler.rb should use safe_receive instead of receive added

Also available in: Atom PDF