Redmine

Please, attach the full raw email that contains this attachment and give your exact Ruby and Redmine versions.

~# ruby -v
ruby 1.8.7 (2010-08-16 patchlevel 302) [x86_64-linux]

Redmine version: == 2012-06-05 v2.0.2

Today I had another incoming email, which broke email subject. I noticed, that on some e-mail clients issue subject was readable, but with different encoding.
In message source Subject was set to: =?UTF-8?Q?[*_ien=C4=81ko=C5=A1ie_e-pasti]_#287_-_Apm=E2c=EEbu?=
=?UTF-8?Q?_materi=E2ls_darbam_ar_jauno_webserveri?=

Anyway - another task with incoming attachemtn
Rake task result (notice warning: Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.):

** Invoke redmine:email:receive_imap (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:email:receive_imap
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
rake aborted!
"šrutu apraksti \r"...
/var/lib/gems/1.8/gems/mail-2.4.4/lib/mail/message.rb:2046:in `conv'
/var/lib/gems/1.8/gems/mail-2.4.4/lib/mail/message.rb:2046:in `decode_body_as_te                                                                                                 xt'
/var/lib/gems/1.8/gems/mail-2.4.4/lib/mail/message.rb:1786:in `decoded'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:245:in `add_attachments'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:243:in `each'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:243:in `add_attachments'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:175:in `receive_issue'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:152:in `dispatch_to_defaul                                                                                                 t'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:137:in `dispatch'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:113:in `receive'
/var/lib/gems/1.8/gems/actionmailer-3.2.5/lib/action_mailer/base.rb:404:in `rece                                                                                                 ive'
/var/lib/gems/1.8/gems/activesupport-3.2.5/lib/active_support/notifications.rb:1                                                                                                 23:in `instrument'
/var/lib/gems/1.8/gems/activesupport-3.2.5/lib/active_support/notifications/inst                                                                                                 rumenter.rb:20:in `instrument'
/var/lib/gems/1.8/gems/activesupport-3.2.5/lib/active_support/notifications.rb:1                                                                                                 23:in `instrument'
/var/lib/gems/1.8/gems/actionmailer-3.2.5/lib/action_mailer/base.rb:401:in `rece                                                                                                 ive'
/opt/redmine/redmine-2.0.2/app/models/mail_handler.rb:44:in `receive'
/opt/redmine/redmine-2.0.2/lib/redmine/imap.rb:35:in `check'
/opt/redmine/redmine-2.0.2/lib/redmine/imap.rb:32:in `each'
/opt/redmine/redmine-2.0.2/lib/redmine/imap.rb:32:in `check'
/opt/redmine/redmine-2.0.2/lib/tasks/email.rake:133
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `execute'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `invoke_with_call_ch                                                                                                 ain'
/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_ch                                                                                                 ain'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `top_level'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exc                                                                                                 eption_handling'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `run'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exc                                                                                                 eption_handling'
/var/lib/gems/1.8/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/var/lib/gems/1.8/gems/rake-0.9.2.2/bin/rake:33
/var/lib/gems/1.8/bin/rake:19:in `load'
/var/lib/gems/1.8/bin/rake:19
Tasks: TOP => redmine:email:receive_imap

Message source

Delivered-To: support@***OURDOMAIN***
Received: by 10.68.21.38 with SMTP id s6csp293641pbe;
        Tue, 19 Jun 2012 07:47:22 -0700 (PDT)
Received: by 10.204.145.89 with SMTP id c25mr8626763bkv.5.1340117241694;
        Tue, 19 Jun 2012 07:47:21 -0700 (PDT)
Return-Path: <arnis@***OURDOMAIN*** >
Received: from mail-bk0-f52.google.com (mail-bk0-f52.google.com [IPADDRESS])
        by mx.google.com with ESMTPS id hy15si15537481bkc.22.2012.06.19.07.47.20
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 19 Jun 2012 07:47:21 -0700 (PDT)
Received-SPF: neutral (google.com: IPADDRESS is neither permitted nor denied by best guess record for domain of arnis@***OURDOMAIN***) client-ip=***OURDOMAIN***;
Authentication-Results: mx.google.com; spf=neutral (google.com: ***OURDOMAIN*** is neither permitted nor denied by best guess record for domain of arnis@***OURDOMAIN***) smtp.mail=arnis@***OURDOMAIN***
Received: by bkcjc3 with SMTP id jc3so6049822bkc.11
        for <support@***OURDOMAIN***>; Tue, 19 Jun 2012 07:47:20 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20120113;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:x-gm-message-state;
        bh=PgwfJNVBaPo4u5uWl09nBLc/uIMfI3YZuOPe3LzMKiw=;
        b=IJ2OYVqU8no5TDGTqnKusBNqDT5gdL5kOse57Cu5HTM8l3hxsHiQWYd4IXafVbQ+WJ
         a2kgnuUh3BgqIHeEmT7bTLfXfy7CnyTTLgILjFi3iCJi9QXSVSKrjiaulEM5gwFX47xs
         P0c7vHI4m+PDkgPtbgj9DMRjqRiRigZlS8LOVduEt3NJAfrCceecDqnEiJ1V405HBSlp
         CUzl34p5LzS6GULkHtzR0dI9A4v0eHVSXkHfqrjFoFYetroz20sfuXt2K2h2c59Idk34
         9gpf5+J+0g9xmTW5ik2a1u75L/rLVLSl9ONjxsHWMMWA973Og3Us90dx93BKRdLQxVfb
         7WOQ==
Received: by 10.204.128.201 with SMTP id l9mr8468091bks.68.1340117240493;
        Tue, 19 Jun 2012 07:47:20 -0700 (PDT)
Return-Path: <arnis@***OURDOMAIN***>
Received: from [192.168.1.101] ([***OURDOMAIN***])
        by mx.google.com with ESMTPS id u8sm23640286bks.0.2012.06.19.07.47.18
        (version=SSLv3 cipher=OTHER);
        Tue, 19 Jun 2012 07:47:19 -0700 (PDT)
Message-ID: <4FE090F1.8020307@***OURDOMAIN***>
Date: Tue, 19 Jun 2012 17:47:13 +0300
From: Arnis Juraga <arnis@***OURDOMAIN***>
User-Agent: Postbox 3.0.3 (Windows/20120304)
MIME-Version: 1.0
To: Support Desk <support@***OURDOMAIN***>
Subject: =?UTF-8?B?TWFyxaFydXR1IGFwcmFrc3Rp?=
Content-Type: multipart/mixed;
 boundary="------------090206070609080307050305" 
X-Gm-Message-State: ALoCoQlJdbjyt4PqpQW8lU4Ygmz18ItlnPoDs9v5/TRKDIBSWkJ7DaVb7QlWZ4J/d0r16/y9qbg8

This is a multi-part message in MIME format.
--------------090206070609080307050305
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

test vs "maršrutu apraksti".

Arnis

--------------090206070609080307050305
Content-Type: text/plain;
 name="attachedtextfile.txt" 
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="attachedtextfile.txt" 

MikgbWFy8HJ1dHUgYXByYWtzdGkgDQoNCg0KDQoNCiA=
--------------090206070609080307050305--

Still, the same problems!
I can see, there are some problems with encoding, charsets?
When e-mail has been read, the message Subject, for example, is incorrectly rewritten. I can see in Message source:

ORIGINAL MESSAGE AS ON MAIL SERVER:
Subject: =?windows-1257?Q?sinhroniz=E2cija?=

Email, sent by REDMINE:
Subject: =?UTF-8?Q?[*_ien=C4=81ko=C5=A1ie_e-pasti]_#266_-_sinhroniz=E2cija?=

Can it be fixed in config files?

Also, after incoming email has been read in, Redmine saves "truncated" message suject, for example:
Incoming message with subject: "sinhronizācija"
is created as issue with subject: "sinhroniz"

I confirm that I have the same issue.

My email contain my name (André) in the signature, and the imap importation fail:

Gem.source_index called from /usr/share/redmine/vendor/rails/railties/lib/rails/gem_dependency.rb:21.
** Invoke redmine:email:receive_imap (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:email:receive_imap
rake aborted!  
incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
/usr/share/redmine/app/models/mail_handler.rb:421:in `gsub'
/usr/share/redmine/app/models/mail_handler.rb:421:in `cleanup_body'
/usr/share/redmine/app/models/mail_handler.rb:359:in `cleaned_up_text_body'
/usr/share/redmine/app/models/mail_handler.rb:192:in `receive_issue_reply'
/usr/share/redmine/app/models/mail_handler.rb:205:in `receive_journal_reply'
/usr/share/redmine/app/models/mail_handler.rb:122:in `dispatch'
/usr/share/redmine/app/models/mail_handler.rb:107:in `receive'
/usr/lib/ruby/vendor_ruby/action_mailer/base.rb:452:in `receive'
/usr/share/redmine/app/models/mail_handler.rb:42:in `receive'
/usr/share/redmine/lib/redmine/imap.rb:35:in `block in check'
/usr/share/redmine/lib/redmine/imap.rb:32:in `each'
/usr/share/redmine/lib/redmine/imap.rb:32:in `check'
/usr/share/redmine/lib/tasks/email.rake:133:in `block (3 levels) in <top (required)>'
/usr/lib/ruby/vendor_ruby/rake/task.rb:205:in `call'
/usr/lib/ruby/vendor_ruby/rake/task.rb:205:in `block in execute'
/usr/lib/ruby/vendor_ruby/rake/task.rb:200:in `each'
/usr/lib/ruby/vendor_ruby/rake/task.rb:200:in `execute'
/usr/lib/ruby/vendor_ruby/rake/task.rb:158:in `block in invoke_with_call_chain'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/lib/ruby/vendor_ruby/rake/task.rb:151:in `invoke_with_call_chain'
/usr/lib/ruby/vendor_ruby/rake/task.rb:144:in `invoke'
/usr/lib/ruby/vendor_ruby/rake/application.rb:116:in `invoke_task'
/usr/lib/ruby/vendor_ruby/rake/application.rb:94:in `block (2 levels) in top_level'
/usr/lib/ruby/vendor_ruby/rake/application.rb:94:in `each'
/usr/lib/ruby/vendor_ruby/rake/application.rb:94:in `block in top_level'
/usr/lib/ruby/vendor_ruby/rake/application.rb:133:in `standard_exception_handling'
/usr/lib/ruby/vendor_ruby/rake/application.rb:88:in `top_level'
/usr/lib/ruby/vendor_ruby/rake/application.rb:66:in `block in run'
/usr/lib/ruby/vendor_ruby/rake/application.rb:133:in `standard_exception_handling'
/usr/lib/ruby/vendor_ruby/rake/application.rb:63:in `run'
/usr/bin/rake:27:in `<main>'
Tasks: TOP => redmine:email:receive_imap

However, if I remove my name, it's working...

I have the same problem.

To reproduce, log into a gmail account. Go to settings, on the first tab (General) make sure the "Outgoing message encoding" is set to "Use default text encoding for outgoing messages"

Then send an email containing non ASCII letters, for example ú æ í þ ð.

It will not turn into a ticket, the error I get is:
------------------------------------------
rake aborted!
PG::Error: ERROR: invalid byte sequence for encoding "UTF8": 0xe16c70
: INSERT INTO "issues" ("assigned_to_id", "author_id", "category_id", "created_on", "description", "done_ratio", "due_date", "estimated_hours", "fixed_version_id", "is_private", "lft", "lock_version", "parent_id", "priority_id", "project_id", "rgt", "root_id", "start_date", "status_id", "subject", "tracker_id", "updated_on") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22) RETURNING "id"
------------------------------------------

If you then try setting the "Outgoing message encoding" to utf8 it works again.

This obviously means I can only accept email from people with their email client set to utf8, if they use special characters.

I fixed it, this only blew up if the subject contained special characters.

in mail_handler.rb I changed the receive_issue function:

  def receive_issue
    project = target_project
    # check permission
    unless @@handler_options[:no_permission_check]
      raise UnauthorizedAction unless user.allowed_to?(:add_issues, project)
    end

    issue = Issue.new(:author => user, :project => project)
    issue.safe_attributes = issue_attributes_from_keywords(issue)
    issue.safe_attributes = {'custom_field_values' => custom_field_values_from_keywords(issue)}

    # CHANGED THE FOLLOWING FOUR LINES:
    body_part = email.text_part || email.html_part || email
    subj = email.subject
    subj = Redmine::CodesetUtil.to_utf8(subj, body_part.charset)
    issue.subject = subj.chomp[0,255]

    if issue.subject.blank?
      issue.subject = '(no subject)'
    end
    issue.description = cleaned_up_text_body

    # add To and Cc as watchers before saving so the watchers can reply to Redmine
    add_watchers(issue)
    issue.save!
    add_attachments(issue)
    logger.info "MailHandler: issue ##{issue.id} created by #{user}" if logger && logger.info
    issue
  end

As I am not a ruby programmer I'm sure this could be done in a more elegant way. The main thing is to make sure that the subject is also converted to utf8, you can check its' current encoding by checking the encoding of the body.

BTW: I'm guessing something similar has to be done for txt attachments

Any news on this? I'm running into this issue currently.

Redmine version 2.3.0

Steffen Zieger wrote:

The attached patch fixes this issue for me and is tested against Redmine 2.3.0

• source:tags/2.3.0/test/unit/mail_handler_test.rb#L495
• source:tags/2.3.0/test/fixtures/mail_handler/subject_as_iso-8859-1.eml

Are there any reasons that these patches are not set for the 2.3.3 version? Can I help out?

Bernd Köster wrote:

Are there any reasons that these patches are not set for the 2.3.3 version?

Because this issue patches have no tests.

And still - problem persists in the latest Redmine version as well.

Can I do tests, to verify fix?

I got the same warning by email from cron. :( using redmine 2.6.0

Look at mine Redmine/Rails/Gems version, may be upgrade can help for You?

I cannot reproduce it today anymore, with the same old file I have attached myself,

Imap read script:

#!/bin/bash
export GEM_HOME=/var/lib/gems/1.9.1/gems/
export PATH=/var/lib/gems/1.9.1/bin:$PATH
RAILS_ENV=production
dirpath="/opt/redmine/redmine-ri-2.6.0" 
cd $dirpath

/usr/local/bin/rake -f $dirpath/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=imap.gmail.com ssl=SSL \
port=993 username=support@DOMAIN password=PASSWORD move_on_success=izlasitie \
move_on_failure=failed project=incoming unknown_user=create no_permission_check=1 no_account_notice=1 --trace

Email was received and added to Redmine correctly now!!!

And received ouput:

** Invoke redmine:email:receive_imap (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:email:receive_imap
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.

As You can see, IMAP read scripts uses gems 1.9.1,
but the Redmine version itself is as follows:

Environment:
  Redmine version                2.6.0.stable
  Ruby version                   1.9.3-p194 (2012-04-20) [i486-linux]
  Rails version                  3.2.19
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.6.17
  Mercurial                      2.2.2
  Git                            1.7.10.4
  Filesystem                     
Redmine plugins:
  add_link                       0.0.1
  progressive_projects_list      1.0.0
  redmine_agile                  1.3.5
  redmine_changeauthor           1.0.0
  redmine_checklists             3.0.1
  redmine_custom_css             0.1.4
  redmine_github_hook            2.1.0
  redmine_helpdesk               0.0.8
  redmine_image_clipboard_paste  1.0.0
  redmine_lightbox2              0.0.3
  redmine_planning               0.7.6
  redmine_slack                  0.1

ERROR -- : MailHandler: an unexpected error occurred when receiving email: Mysql2::Error: Incorrect string value: '\xF0\x9F\x94\xA5 S...' for column `xxx`.`issues`.`description` at row 1: INSERT INTO `issues` (`tracker_id`, `project_id`, `subject`, `description`, `status_id`, `assigned_to_id`, `priority_id`, `author_id`, `created_on`, `updated_on`, `start_date`, `lock_version`) VALUES

REASON: utf8mb -> Unicode Symbols

Solution: Convert table to utf8mb and set in file database.yml "encoding: utf8mb4"