Project

General

Profile

Actions

Defect #24910

open

Issue processing UTF7 mails

Added by Falk Eckert over 9 years ago. Updated 9 days ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Email receiving
Target version:
-
Resolution:
Affected version:

Description

Hello,

we have an issue with processing UTF7 mails:

exception: code converter not found (UTF-7 to UTF-8)
["/srv/redmine/public_html/lib/redmine/codeset_util.rb:25:in `encode'", "/srv/redmine/public_html/lib/redmine/codeset_util.rb:25:in `to_utf8'"

Are there any ideas to solve this issue?

Thx
Falk


Files

converter_not_found.patch (1.45 KB) converter_not_found.patch Karel Pičman, 2020-09-14 13:27
converter_not_found2.patch (2.78 KB) converter_not_found2.patch Karel Pičman, 2020-09-17 14:37
Actions #1

Updated by Go MAEDA over 9 years ago

  • Category set to Email receiving
Actions #2

Updated by Toshi MARUYAMA over 9 years ago

$ cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)

$ irb
1.9.3-p551 :001 > str = "test".force_encoding("UTF-7")
 => "\x74\x65\x73\x74" 
1.9.3-p551 :002 > str = str.encode("UTF-8", :invalid => :replace,:undef => :replace, :replace => '?')
Encoding::ConverterNotFoundError: code converter not found (UTF-7 to UTF-8)
    from (irb):2:in `encode'
    from (irb):2
    from /home/xxxx/.rvm/rubies/ruby-1.9.3-p551/bin/irb:12:in `<main>'
Actions #3

Updated by Toshi MARUYAMA over 9 years ago

  • Priority changed from Normal to Low

It seems Ruby does not have UTF-7 converter.

"Code converter not found (UTF-8 to EUC-TW)": https://bugs.ruby-lang.org/issues/6995

Actions #4

Updated by Fred Sanger over 9 years ago

This encyclopedia article for UTF7 seems to be a good place to start

Actions #5

Updated by Pavel Rosický over 9 years ago

toshio harita: utf-7 is relevant only for email processing. Ruby still supports it, but it's a little bit tricky. This will work:

require "net/imap" 
Net::IMAP.decode_utf7(mail_body)
Actions #6

Updated by Karel Pičman almost 6 years ago

The same problem with Windows-1258. Attached patch should cover all not supported code pages.

Actions #8

Updated by Karel Pičman almost 6 years ago

A test with a real Vietnamese email added into the patch.

Actions #9

Updated by Bart Dopheide 9 days ago

The Microsoft Outlook client forwards plain text e-mails (like cron messages) in UTF-7, triggering this bug. The following diff works for me. Disclaimer: I'm completely new to Ruby. I declare the following work public domain.

--- lib/redmine/codeset_util.rb.orig    2026-03-17 08:40:05.000000000 +0100
+++ lib/redmine/codeset_util.rb    2026-07-17 14:25:52.476070776 +0200
@@ -17,6 +17,8 @@
 # along with this program; if not, write to the Free Software
 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

+require 'base64'
+
 module Redmine
   module CodesetUtil
     def self.replace_invalid_utf8(str)
@@ -31,6 +33,19 @@
       str
     end

+    ## https://www.redmine.org/issues/24910
+    def self.decode_utf7(str)
+      str.gsub(/\+([A-Za-z0-9\/]+)-/) do
+        b64 = Regexp.last_match(1)
+
+        utf16be = Base64.decode64(
+          b64 + "=" * ((4 - b64.length % 4) % 4)
+        )
+
+        utf16be.force_encoding("UTF-16BE").encode("UTF-8")
+      end.gsub(/\+-/, '+')
+    end
+
     def self.to_utf8(str, encoding)
       return if str.nil?

@@ -40,10 +55,20 @@
         return str
       end
       enc = encoding.blank? ? "UTF-8" : encoding
-      if enc.casecmp("UTF-8") != 0
-        str.force_encoding(enc)
-        str = str.encode("UTF-8", :invalid => :replace,
-              :undef => :replace, :replace => '?')
+      if enc.casecmp("UTF-7").zero?
+        Rails.logger.info ">>> Decoding UTF-7 content." 
+        str = decode_utf7(str)
+      elsif enc.casecmp("UTF-8") != 0
+        begin
+          str.force_encoding(enc)
+          str = str.encode("UTF-8", :invalid => :replace,
+                :undef => :replace, :replace => '?')
+        rescue Encoding::ConverterNotFoundError => e
+          Rails.logger.warn ">>> ConverterNotFoundError (#{enc} to UTF-8)" 
+          Rails.logger.warn ">>> #{e.class}: #{e.message}" 
+          Rails.logger.warn e.backtrace.join("\n")
+          str = replace_invalid_utf8(str)
+        end
       else
         str = replace_invalid_utf8(str)
       end
Actions

Also available in: Atom PDF