Defect #11191: receive_imap failed with localizaed characters in attachments or subject - Redmine

Actions

Copy link

#1

Updated by Arnis Juraga almost 12 years ago

File attachedtextfile.txt attachedtextfile.txt added

Actions

Copy link

#2

Updated by Jean-Philippe Lang almost 12 years ago

Please, attach the full raw email that contains this attachment and give your exact Ruby and Redmine versions.

Actions

Copy link

#3

Updated by Arnis Juraga almost 12 years ago

~# ruby -v
ruby 1.8.7 (2010-08-16 patchlevel 302) [x86_64-linux]

Redmine version: == 2012-06-05 v2.0.2

Today I had another incoming email, which broke email subject. I noticed, that on some e-mail clients issue subject was readable, but with different encoding.
In message source Subject was set to: =?UTF-8?Q?[*_ien=C4=81ko=C5=A1ie_e-pasti]_#287_-_Apm=E2c=EEbu?=
=?UTF-8?Q?_materi=E2ls_darbam_ar_jauno_webserveri?=

Anyway - another task with incoming attachemtn
Rake task result (notice warning: Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.):

** Invoke redmine:email:receive_imap (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:email:receive_imap
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
rake aborted!
"šrutu apraksti \r"...
/var/lib/gems/1.8/gems/mail-2.4.4/lib/mail/message.rb:2046:in `conv'
/var/lib/gems/1.8/gems/mail-2.4.4/lib/mail/message.rb:2046:in `decode_body_as_te                                                                                                 xt'
/var/lib/gems/1.8/gems/mail-2.4.4/lib/mail/message.rb:1786:in `decoded'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:245:in `add_attachments'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:243:in `each'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:243:in `add_attachments'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:175:in `receive_issue'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:152:in `dispatch_to_defaul                                                                                                 t'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:137:in `dispatch'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:113:in `receive'
/var/lib/gems/1.8/gems/actionmailer-3.2.5/lib/action_mailer/base.rb:404:in `rece                                                                                                 ive'
/var/lib/gems/1.8/gems/activesupport-3.2.5/lib/active_support/notifications.rb:1                                                                                                 23:in `instrument'
/var/lib/gems/1.8/gems/activesupport-3.2.5/lib/active_support/notifications/inst                                                                                                 rumenter.rb:20:in `instrument'
/var/lib/gems/1.8/gems/activesupport-3.2.5/lib/active_support/notifications.rb:1                                                                                                 23:in `instrument'
/var/lib/gems/1.8/gems/actionmailer-3.2.5/lib/action_mailer/base.rb:401:in `rece                                                                                                 ive'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:44:in `receive'
/opt/redmine/redmine-2.0.2/lib/redmine/imap.rb:35:in `check'
/opt/redmine/redmine-2.0.2/lib/redmine/imap.rb:32:in `each'
/opt/redmine/redmine-2.0.2/lib/redmine/imap.rb:32:in `check'
/opt/redmine/redmine-2.0.2/lib/tasks/email.rake:133
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `execute'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `invoke_with_call_ch                                                                                                 ain'
/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_ch                                                                                                 ain'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exc                                                                                                 eption_handling'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `run'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exc                                                                                                 eption_handling'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/var/lib/gems/1.8/gems/rake-0.9.2.2/bin/rake:33
/var/lib/gems/1.8/bin/rake:19:in `load'
/var/lib/gems/1.8/bin/rake:19
Tasks: TOP => redmine:email:receive_imap

Message source

Delivered-To: support@***OURDOMAIN***
Received: by 10.68.21.38 with SMTP id s6csp293641pbe;
        Tue, 19 Jun 2012 07:47:22 -0700 (PDT)
Received: by 10.204.145.89 with SMTP id c25mr8626763bkv.5.1340117241694;
        Tue, 19 Jun 2012 07:47:21 -0700 (PDT)
Return-Path: <arnis@***OURDOMAIN*** >
Received: from mail-bk0-f52.google.com (mail-bk0-f52.google.com [IPADDRESS])
        by mx.google.com with ESMTPS id hy15si15537481bkc.22.2012.06.19.07.47.20
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 19 Jun 2012 07:47:21 -0700 (PDT)
Received-SPF: neutral (google.com: IPADDRESS is neither permitted nor denied by best guess record for domain of arnis@***OURDOMAIN***) client-ip=***OURDOMAIN***;
Authentication-Results: mx.google.com; spf=neutral (google.com: ***OURDOMAIN*** is neither permitted nor denied by best guess record for domain of arnis@***OURDOMAIN***) smtp.mail=arnis@***OURDOMAIN***
Received: by bkcjc3 with SMTP id jc3so6049822bkc.11
        for <support@***OURDOMAIN***>; Tue, 19 Jun 2012 07:47:20 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:x-gm-message-state;
        bh=PgwfJNVBaPo4u5uWl09nBLc/uIMfI3YZuOPe3LzMKiw=;
        b=IJ2OYVqU8no5TDGTqnKusBNqDT5gdL5kOse57Cu5HTM8l3hxsHiQWYd4IXafVbQ+WJ
         a2kgnuUh3BgqIHeEmT7bTLfXfy7CnyTTLgILjFi3iCJi9QXSVSKrjiaulEM5gwFX47xs
         P0c7vHI4m+PDkgPtbgj9DMRjqRiRigZlS8LOVduEt3NJAfrCceecDqnEiJ1V405HBSlp
         CUzl34p5LzS6GULkHtzR0dI9A4v0eHVSXkHfqrjFoFYetroz20sfuXt2K2h2c59Idk34
         9gpf5+J+0g9xmTW5ik2a1u75L/rLVLSl9ONjxsHWMMWA973Og3Us90dx93BKRdLQxVfb
         7WOQ==
Received: by 10.204.128.201 with SMTP id l9mr8468091bks.68.1340117240493;
        Tue, 19 Jun 2012 07:47:20 -0700 (PDT)
Return-Path: <arnis@***OURDOMAIN***>
Received: from [192.168.1.101] ([***OURDOMAIN***])
        by mx.google.com with ESMTPS id u8sm23640286bks.0.2012.06.19.07.47.18
        (version=SSLv3 cipher=OTHER);
        Tue, 19 Jun 2012 07:47:19 -0700 (PDT)
Message-ID: <4FE090F1.8020307@***OURDOMAIN***>
Date: Tue, 19 Jun 2012 17:47:13 +0300
From: Arnis Juraga <arnis@***OURDOMAIN***>
User-Agent: Postbox 3.0.3 (Windows/20120304)
MIME-Version: 1.0
To: Support Desk <support@***OURDOMAIN***>
Subject: =?UTF-8?B?TWFyxaFydXR1IGFwcmFrc3Rp?=
Content-Type: multipart/mixed;
 boundary="------------090206070609080307050305" 
X-Gm-Message-State: ALoCoQlJdbjyt4PqpQW8lU4Ygmz18ItlnPoDs9v5/TRKDIBSWkJ7DaVb7QlWZ4J/d0r16/y9qbg8

This is a multi-part message in MIME format.
--------------090206070609080307050305
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

test vs "maršrutu apraksti".

Arnis

--------------090206070609080307050305
Content-Type: text/plain;
 name="attachedtextfile.txt" 
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="attachedtextfile.txt" 

MikgbWFy8HJ1dHUgYXByYWtzdGkgDQoNCg0KDQoNCiA=
--------------090206070609080307050305--

Actions

Copy link

#4

Updated by Arnis Juraga almost 12 years ago

Still, the same problems!
I can see, there are some problems with encoding, charsets?
When e-mail has been read, the message Subject, for example, is incorrectly rewritten. I can see in Message source:

ORIGINAL MESSAGE AS ON MAIL SERVER:
Subject: =?windows-1257?Q?sinhroniz=E2cija?=

Email, sent by REDMINE:
Subject: =?UTF-8?Q?[*_ien=C4=81ko=C5=A1ie_e-pasti]_#266_-_sinhroniz=E2cija?=

Can it be fixed in config files?

Actions

Copy link

#5

Updated by Arnis Juraga almost 12 years ago

Also, after incoming email has been read in, Redmine saves "truncated" message suject, for example:
Incoming message with subject: "sinhronizācija"
is created as issue with subject: "sinhroniz"

Actions

Copy link

#6

Updated by Andre Rodier almost 12 years ago

I confirm that I have the same issue.

My email contain my name (André) in the signature, and the imap importation fail:

Gem.source_index called from /usr/share/redmine/vendor/rails/railties/lib/rails/gem_dependency.rb:21.
** Invoke redmine:email:receive_imap (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:email:receive_imap
rake aborted!  
incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
/usr/share/redmine/app/models/mail_handler.rb:421:in `gsub'
/usr/share/redmine/app/models/mail_handler.rb:421:in `cleanup_body'
/usr/share/redmine/app/models/mail_handler.rb:359:in `cleaned_up_text_body'
/usr/share/redmine/app/models/mail_handler.rb:192:in `receive_issue_reply'
/usr/share/redmine/app/models/mail_handler.rb:205:in `receive_journal_reply'
/usr/share/redmine/app/models/mail_handler.rb:122:in `dispatch'
/usr/share/redmine/app/models/mail_handler.rb:107:in `receive'
/usr/lib/ruby/vendor_ruby/action_mailer/base.rb:452:in `receive'
/usr/share/redmine/app/models/mail_handler.rb:42:in `receive'
/usr/share/redmine/lib/redmine/imap.rb:35:in `block in check'
/usr/share/redmine/lib/redmine/imap.rb:32:in `each'
/usr/share/redmine/lib/redmine/imap.rb:32:in `check'
/usr/share/redmine/lib/tasks/email.rake:133:in `block (3 levels) in <top (required)>'
/usr/lib/ruby/vendor_ruby/rake/task.rb:205:in `call'
/usr/lib/ruby/vendor_ruby/rake/task.rb:205:in `block in execute'
/usr/lib/ruby/vendor_ruby/rake/task.rb:200:in `each'
/usr/lib/ruby/vendor_ruby/rake/task.rb:200:in `execute'
/usr/lib/ruby/vendor_ruby/rake/task.rb:158:in `block in invoke_with_call_chain'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/lib/ruby/vendor_ruby/rake/task.rb:151:in `invoke_with_call_chain'
/usr/lib/ruby/vendor_ruby/rake/task.rb:144:in `invoke'
/usr/lib/ruby/vendor_ruby/rake/application.rb:116:in `invoke_task'
/usr/lib/ruby/vendor_ruby/rake/application.rb:94:in `block (2 levels) in top_level'
/usr/lib/ruby/vendor_ruby/rake/application.rb:94:in `each'
/usr/lib/ruby/vendor_ruby/rake/application.rb:94:in `block in top_level'
/usr/lib/ruby/vendor_ruby/rake/application.rb:133:in `standard_exception_handling'
/usr/lib/ruby/vendor_ruby/rake/application.rb:88:in `top_level'
/usr/lib/ruby/vendor_ruby/rake/application.rb:66:in `block in run'
/usr/lib/ruby/vendor_ruby/rake/application.rb:133:in `standard_exception_handling'
/usr/lib/ruby/vendor_ruby/rake/application.rb:63:in `run'
/usr/bin/rake:27:in `<main>'
Tasks: TOP => redmine:email:receive_imap

However, if I remove my name, it's working...

Actions

Copy link

#7

Updated by Bjarni Thorisson over 11 years ago

I have the same problem.

To reproduce, log into a gmail account. Go to settings, on the first tab (General) make sure the "Outgoing message encoding" is set to "Use default text encoding for outgoing messages"

Then send an email containing non ASCII letters, for example ú æ í þ ð.

It will not turn into a ticket, the error I get is:
------------------------------------------
rake aborted!
PG::Error: ERROR: invalid byte sequence for encoding "UTF8": 0xe16c70
: INSERT INTO "issues" ("assigned_to_id", "author_id", "category_id", "created_on", "description", "done_ratio", "due_date", "estimated_hours", "fixed_version_id", "is_private", "lft", "lock_version", "parent_id", "priority_id", "project_id", "rgt", "root_id", "start_date", "status_id", "subject", "tracker_id", "updated_on") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22) RETURNING "id"
------------------------------------------

If you then try setting the "Outgoing message encoding" to utf8 it works again.

This obviously means I can only accept email from people with their email client set to utf8, if they use special characters.

Actions

Copy link

#8

Updated by Bjarni Thorisson over 11 years ago

I fixed it, this only blew up if the subject contained special characters.

in mail_handler.rb I changed the receive_issue function:

  def receive_issue
    project = target_project
    # check permission
    unless @@handler_options[:no_permission_check]
      raise UnauthorizedAction unless user.allowed_to?(:add_issues, project)
    end

    issue = Issue.new(:author => user, :project => project)
    issue.safe_attributes = issue_attributes_from_keywords(issue)
    issue.safe_attributes = {'custom_field_values' => custom_field_values_from_keywords(issue)}

    # CHANGED THE FOLLOWING FOUR LINES:
    body_part = email.text_part || email.html_part || email
    subj = email.subject
    subj = Redmine::CodesetUtil.to_utf8(subj, body_part.charset)
    issue.subject = subj.chomp[0,255]

    if issue.subject.blank?
      issue.subject = '(no subject)'
    end
    issue.description = cleaned_up_text_body

    # add To and Cc as watchers before saving so the watchers can reply to Redmine
    add_watchers(issue)
    issue.save!
    add_attachments(issue)
    logger.info "MailHandler: issue ##{issue.id} created by #{user}" if logger && logger.info
    issue
  end

As I am not a ruby programmer I'm sure this could be done in a more elegant way. The main thing is to make sure that the subject is also converted to utf8, you can check its' current encoding by checking the encoding of the body.

Actions

Copy link

#9

Updated by Bjarni Thorisson over 11 years ago

BTW: I'm guessing something similar has to be done for txt attachments

Actions

Copy link

#10

Updated by Steffen Zieger about 11 years ago

Any news on this? I'm running into this issue currently.

Redmine version 2.3.0

Actions

Copy link

#11

Updated by Steffen Zieger about 11 years ago

File redmine_mail_receive.diff redmine_mail_receive.diff added

The attached patch fixes this issue for me and is tested against Redmine 2.3.0

Actions

Copy link

#12

Updated by Toshi MARUYAMA about 11 years ago

Steffen Zieger wrote:

The attached patch fixes this issue for me and is tested against Redmine 2.3.0

Could you add tests?
See:

Actions

Copy link

#13

Updated by Steffen Zieger almost 11 years ago

File redmine_attachment_charset_issue.diff redmine_attachment_charset_issue.diff added

Toshi MARUYAMA wrote:

Could you add tests?

As time is problematic right now, I can't promise it, but I'll do my best.

Attached is a patch, which fixes this issue in attachments, too.
Maybe there is a better way, but at least it's working for me.

Actions

Copy link

#14

Updated by Toshi MARUYAMA almost 11 years ago

Category set to Email receiving

Actions

Copy link

#15

Updated by Bernd Köster over 10 years ago

Are there any reasons that these patches are not set for the 2.3.3 version? Can I help out?

Actions

Copy link

#16

Updated by Toshi MARUYAMA over 10 years ago

Bernd Köster wrote:

Are there any reasons that these patches are not set for the 2.3.3 version?

Because this issue patches have no tests.

Actions

Copy link

#17

Updated by Arnis Juraga over 10 years ago

And still - problem persists in the latest Redmine version as well.

Can I do tests, to verify fix?

Actions

Copy link

#18

Updated by Aníbal Pacheco over 9 years ago

I got the same warning by email from cron. :( using redmine 2.6.0

Actions

Copy link

#19

Updated by Arnis Juraga over 9 years ago

Look at mine Redmine/Rails/Gems version, may be upgrade can help for You?

I cannot reproduce it today anymore, with the same old file I have attached myself,

Imap read script:

#!/bin/bash
export GEM_HOME=/var/lib/gems/1.9.1/gems/
export PATH=/var/lib/gems/1.9.1/bin:$PATH
RAILS_ENV=production
dirpath="/opt/redmine/redmine-ri-2.6.0" 
cd $dirpath

/usr/local/bin/rake -f $dirpath/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=imap.gmail.com ssl=SSL \
port=993 username=support@DOMAIN password=PASSWORD move_on_success=izlasitie \
move_on_failure=failed project=incoming unknown_user=create no_permission_check=1 no_account_notice=1 --trace

Email was received and added to Redmine correctly now!!!

And received ouput:

** Invoke redmine:email:receive_imap (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:email:receive_imap
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.

As You can see, IMAP read scripts uses gems 1.9.1,
but the Redmine version itself is as follows:

Environment:
  Redmine version                2.6.0.stable
  Ruby version                   1.9.3-p194 (2012-04-20) [i486-linux]
  Rails version                  3.2.19
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.6.17
  Mercurial                      2.2.2
  Git                            1.7.10.4
  Filesystem                     
Redmine plugins:
  add_link                       0.0.1
  progressive_projects_list      1.0.0
  redmine_agile                  1.3.5
  redmine_changeauthor           1.0.0
  redmine_checklists             3.0.1
  redmine_custom_css             0.1.4
  redmine_github_hook            2.1.0
  redmine_helpdesk               0.0.8
  redmine_image_clipboard_paste  1.0.0
  redmine_lightbox2              0.0.3
  redmine_planning               0.7.6
  redmine_slack                  0.1

Actions

Copy link

#20

Updated by Go MAEDA about 5 years ago

Status changed from New to Closed
Resolution set to Cant reproduce

I cannot reproduce the problem with the raw message attached in #11191#note-3 in Redmine 4.0.1.devel.17851. An issue was successfully created and the Russian subject was not broken.

Maybe something was fixed in some version of Mail gem.

Actions

Copy link

#21

Updated by Daniel Drexlmaier about 2 years ago

ERROR -- : MailHandler: an unexpected error occurred when receiving email: Mysql2::Error: Incorrect string value: '\xF0\x9F\x94\xA5 S...' for column `xxx`.`issues`.`description` at row 1: INSERT INTO `issues` (`tracker_id`, `project_id`, `subject`, `description`, `status_id`, `assigned_to_id`, `priority_id`, `author_id`, `created_on`, `updated_on`, `start_date`, `lock_version`) VALUES

REASON: utf8mb -> Unicode Symbols

Actions

Copy link

#22

Updated by Daniel Drexlmaier about 2 years ago

Solution: Convert table to utf8mb and set in file database.yml "encoding: utf8mb4"

attachedtextfile.txt (32 Bytes) attachedtextfile.txt	Email attachment	Arnis Juraga, 2012-06-18 15:12
redmine_mail_receive.diff (487 Bytes) redmine_mail_receive.diff		Steffen Zieger, 2013-04-16 15:38
redmine_attachment_charset_issue.diff (1.69 KB) redmine_attachment_charset_issue.diff		Steffen Zieger, 2013-06-03 16:36

Project

General

Profile

Redmine

Custom queries

Defect #11191

receive_imap failed with localizaed characters in attachments or subject

Updated by Arnis Juraga almost 12 years ago

Updated by Jean-Philippe Lang almost 12 years ago

Updated by Arnis Juraga almost 12 years ago

Updated by Arnis Juraga almost 12 years ago

Updated by Arnis Juraga almost 12 years ago

Updated by Andre Rodier almost 12 years ago

Updated by Bjarni Thorisson over 11 years ago

Updated by Bjarni Thorisson over 11 years ago

Updated by Bjarni Thorisson over 11 years ago

Updated by Steffen Zieger about 11 years ago

Updated by Steffen Zieger about 11 years ago

Updated by Toshi MARUYAMA about 11 years ago

Updated by Steffen Zieger almost 11 years ago

Updated by Toshi MARUYAMA almost 11 years ago

Updated by Bernd Köster over 10 years ago

Updated by Toshi MARUYAMA over 10 years ago

Updated by Arnis Juraga over 10 years ago

Updated by Aníbal Pacheco over 9 years ago

Updated by Arnis Juraga over 9 years ago

Updated by Go MAEDA about 5 years ago

Updated by Daniel Drexlmaier about 2 years ago

Updated by Daniel Drexlmaier about 2 years ago