Defect #21742

Received text attachments doesn't hold the original encoding on Ruby >= 2.1

Added by Pavel Rosický over 2 years ago. Updated over 2 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Email receiving
Target version:3.3.0
Resolution: Affected version:

Description

Hello,
if I send an email with .txt file encoded in ISO 8859-2, it's received from the mailbox and saved to /files, but the encoding is different (some characters are lost). I think this is originaly caused by the utf-8 header added by email clients. In my opinion attachments shouln't be reencoded. Am I missed somethig?

Attached patch fixes the issue.

Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: base64

Environment:
  Redmine version                3.2.0.devel.15058
  Ruby version                   2.1.5-p273 (2014-11-13) [i386-mingw32]
  Rails version                  4.2.5
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.8.13
  Git                            1.9.5
  Filesystem
Redmine plugins:
  no plugin installed

mail_handler.rb.patch Magnifier (639 Bytes) Pavel Rosický, 2016-01-17 18:15

mail_handler_test.rb.patch Magnifier (1.15 KB) Pavel Rosický, 2016-01-18 11:55

ticket_with_text_attachment_iso-8859-2.eml.patch Magnifier (1.2 KB) Pavel Rosický, 2016-01-18 11:56

original.txt Magnifier (19 Bytes) Pavel Rosický, 2016-02-08 10:46

encoded.txt Magnifier (12 Bytes) Pavel Rosický, 2016-02-08 10:46

Associated revisions

Revision 15183
Added by Toshi MARUYAMA over 2 years ago

add test which ensure received mail text attachment keeps original encoding (#21742)

This test fails on Ruby >= 2.1 and will be fixed by next revision.

Contributed by Pavel Rosický.

Revision 15184
Added by Toshi MARUYAMA over 2 years ago

fix received mail text attachment does not keep original encoding on Ruby >= 2.1 (#21742)

Contributed by Pavel Rosický.

Revision 15185
Added by Toshi MARUYAMA over 2 years ago

svn propset svn:eol-style native fixture (#21742)

History

#2 Updated by Pavel Rosický over 2 years ago

Done

#4 Updated by Toshi MARUYAMA over 2 years ago

I cannot reproduce on trunk r15148 with ruby-1.9.3-p551.
Test passes without mail_handler.rb.patch .
You can see latin2 encoded file in files/ directory with following change.

diff --git a/app/models/attachment.rb b/app/models/attachment.rb
--- a/app/models/attachment.rb
+++ b/app/models/attachment.rb
@@ -53,7 +53,7 @@ class Attachment < ActiveRecord::Base
   @@thumbnails_storage_path = File.join(Rails.root, "tmp", "thumbnails")

   before_create :files_to_final_location
-  after_rollback :delete_from_disk, :on => :create
+  # after_rollback :delete_from_disk, :on => :create
   after_commit :delete_from_disk, :on => :destroy

   # Returns an unsaved copy of the attachment

$ ruby test/unit/mail_handler_test.rb -n test_mail_with_attachment_latin2
Run options: -n test_mail_with_attachment_latin2 --seed 14783

# Running:

.

Finished in 2.025752s, 0.4936 runs/s, 3.4555 assertions/s.

1 runs, 7 assertions, 0 failures, 0 errors, 0 skips
$ cat files/2016/02/160208173558_yourfile.txt | iconv -f latin2 -t utf8
příliš žluoučký kůn

#5 Updated by Pavel Rosický over 2 years ago

retested against revision r15148 without mail_handler.rb.patch

MailHandlerTest#test_mail_with_attachment_latin2 [C:/redmine svn/test/unit/mail_handler_test.rb:590]:
Expected: 19
Actual: 12

About your application's environment
Rails version 4.2.5.1
Ruby version 2.1.5-p273 (x64-mingw32)
RubyGems version 2.2.3
Rack version 1.6.4
Middleware Rack::Sendfile, ActionDispatch::Static, Rack::Lock, #<
ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x000000094378e0>, Rack::
Runtime, Rack::MethodOverride, ActionDispatch::RequestId, Rails::Rack::Logger, A
ctionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::
RemoteIp, ActionDispatch::Reloader, ActionDispatch::Callbacks, ActiveRecord::Con
nectionAdapters::ConnectionManagement, ActiveRecord::QueryCache, ActionDispatch:
:Cookies, ActionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDis
patch::ParamsParser, ActionDispatch::XmlParamsParser, Rack::Head, Rack::Conditio
nalGet, Rack::ETag, Rack::ContentLength, RequestStore::Middleware, OpenIdAuthent
ication
Application root C:/redmine svn
Environment development
Database adapter mysql2
Database schema version 20151031095005

Environment:
Redmine version 3.2.0.devel
Ruby version 2.1.5-p273 (2014-11-13) [x64-mingw32]
Rails version 4.2.5.1
Environment production
Database adapter Mysql2
SCM:
Git 1.9.5
Filesystem
Redmine plugins:
no plugin installed

no other tests are failing (except IssueNestedSetConcurrencyTest which supposed to be a known problem on mysql) and with mail_handler.rb.patch applied this one passes as well.

IssueNestedSetConcurrencyTest#test_concurrency [C:/redmine svn/test/unit/issue_nested_set_concurrency_test.rb:45]:
Expected "Mysql2::Error: Deadlock found when trying to get lock; try restarting
transaction: SELECT `issues`.`id` FROM `issues` WHERE (root_id IN (SELECT root_i
d FROM issues WHERE id IN (800,794))) ORDER BY `issues`.`id` ASC FOR UPDATE" to
be nil.

I attached both attachments, the original and the one saved to files (encoded.txt). Maybe it depends on the environment, I'll try to find out.

#6 Updated by Pavel Rosický over 2 years ago

@Toshi MARUYAMA - could you retest it on a more recent Ruby version?

passed - ruby 1.9.3p551 (2014-11-13) [i386-mingw32]
failed - ruby 2.1.5p273 (2014-11-13 revision 48405) [i386-mingw32]
failed - ruby 2.2.3p173 (2015-08-18 revision 51636) [x64-mingw32]
failed - jruby 9.0.5.0-SNAPSHOT (2.2.3) 2016-01-20 775b79d Java HotSpot(TM) 64-Bit Server

#7 Updated by Toshi MARUYAMA over 2 years ago

  • Target version set to 3.3.0

#8 Updated by Toshi MARUYAMA over 2 years ago

  • Tracker changed from Patch to Defect
  • Subject changed from Received text attachments doesn't hold the original encoding to Received text attachments doesn't hold the original encoding on Ruby >= 2.1
  • Status changed from Needs feedback to New

#9 Updated by Toshi MARUYAMA over 2 years ago

  • Status changed from New to Closed

Committed in trunk, thanks.
I have reported to upstream.
https://github.com/mikel/mail/issues/972

Also available in: Atom PDF