Project

General

Profile

Actions

Defect #24910

open

Issue processing UTF7 mails

Defect #24910: Issue processing UTF7 mails

Added by Falk Eckert almost 10 years ago. Updated 4 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Email receiving
Target version:
-
Resolution:
Affected version:

Description

Hello,

we have an issue with processing UTF7 mails:

exception: code converter not found (UTF-7 to UTF-8)
["/srv/redmine/public_html/lib/redmine/codeset_util.rb:25:in `encode'", "/srv/redmine/public_html/lib/redmine/codeset_util.rb:25:in `to_utf8'"

Are there any ideas to solve this issue?

Thx
Falk


Files

converter_not_found.patch (1.45 KB) converter_not_found.patch Karel Pičman, 2020-09-14 13:27
converter_not_found2.patch (2.78 KB) converter_not_found2.patch Karel Pičman, 2020-09-17 14:37

Updated by Go MAEDA almost 10 years ago Actions #1

  • Category set to Email receiving

Updated by Toshi MARUYAMA almost 10 years ago Actions #2

$ cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)

$ irb
1.9.3-p551 :001 > str = "test".force_encoding("UTF-7")
 => "\x74\x65\x73\x74" 
1.9.3-p551 :002 > str = str.encode("UTF-8", :invalid => :replace,:undef => :replace, :replace => '?')
Encoding::ConverterNotFoundError: code converter not found (UTF-7 to UTF-8)
    from (irb):2:in `encode'
    from (irb):2
    from /home/xxxx/.rvm/rubies/ruby-1.9.3-p551/bin/irb:12:in `<main>'

Updated by Toshi MARUYAMA almost 10 years ago Actions #3

  • Priority changed from Normal to Low

It seems Ruby does not have UTF-7 converter.

"Code converter not found (UTF-8 to EUC-TW)": https://bugs.ruby-lang.org/issues/6995

Updated by Fred Sanger over 9 years ago Actions #4

This encyclopedia article for UTF7 seems to be a good place to start

Updated by Pavel Rosický over 9 years ago Actions #5

@Toshi: utf-7 is relevant only for email processing. Ruby still supports it, but it's a little bit tricky. This will work:

require "net/imap" 
Net::IMAP.decode_utf7(mail_body)

Updated by Karel Pičman about 6 years ago Actions #6

The same problem with Windows-1258. Attached patch should cover all not supported code pages.

Updated by Pavel Rosický about 6 years ago Actions #7

@karel
could you add a more realistic test case with an email as a fixture?
https://github.com/redmine/redmine/blob/3ac569c4d8c8b04a9cc33e3c2a05f8553d99827e/test/unit/mail_handler_test.rb#L542

it should be already fixed by
https://github.com/mikel/mail/pull/1112

Updated by Karel Pičman about 6 years ago Actions #8

A test with a real Vietnamese email added into the patch.

Updated by Bart Dopheide 4 months ago Actions #9

The Microsoft Outlook client forwards plain text e-mails (like cron messages) in UTF-7, triggering this bug. The following diff works for me. Disclaimer: I'm completely new to Ruby. I declare the following work public domain.

--- lib/redmine/codeset_util.rb.orig    2026-03-17 08:40:05.000000000 +0100
+++ lib/redmine/codeset_util.rb    2026-07-17 14:25:52.476070776 +0200
@@ -17,6 +17,8 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

+require 'base64'
+
 module Redmine
   module CodesetUtil
     def self.replace_invalid_utf8(str)
@@ -31,6 +33,19 @@
       str
     end

+    ## https://www.redmine.org/issues/24910
+    def self.decode_utf7(str)
+      str.gsub(/\+([A-Za-z0-9\/]+)-/) do
+        b64 = Regexp.last_match(1)
+
+        utf16be = Base64.decode64(
+          b64 + "=" * ((4 - b64.length % 4) % 4)
+        )
+
+        utf16be.force_encoding("UTF-16BE").encode("UTF-8")
+      end.gsub(/\+-/, '+')
+    end
+
     def self.to_utf8(str, encoding)
       return if str.nil?

@@ -40,10 +55,20 @@
         return str
       end
       enc = encoding.blank? ? "UTF-8" : encoding
-      if enc.casecmp("UTF-8") != 0
-        str.force_encoding(enc)
-        str = str.encode("UTF-8", :invalid => :replace,
-              :undef => :replace, :replace => '?')
+      if enc.casecmp("UTF-7").zero?
+        Rails.logger.info ">>> Decoding UTF-7 content." 
+        str = decode_utf7(str)
+      elsif enc.casecmp("UTF-8") != 0
+        begin
+          str.force_encoding(enc)
+          str = str.encode("UTF-8", :invalid => :replace,
+                :undef => :replace, :replace => '?')
+        rescue Encoding::ConverterNotFoundError => e
+          Rails.logger.warn ">>> ConverterNotFoundError (#{enc} to UTF-8)" 
+          Rails.logger.warn ">>> #{e.class}: #{e.message}" 
+          Rails.logger.warn e.backtrace.join("\n")
+          str = replace_invalid_utf8(str)
+        end
       else
         str = replace_invalid_utf8(str)
       end
Actions

Also available in: PDF Atom