Defect #24910
openIssue processing UTF7 mails
Description
Hello,
we have an issue with processing UTF7 mails:
exception: code converter not found (UTF-7 to UTF-8)
["/srv/redmine/public_html/lib/redmine/codeset_util.rb:25:in `encode'", "/srv/redmine/public_html/lib/redmine/codeset_util.rb:25:in `to_utf8'"
Are there any ideas to solve this issue?
Thx
Falk
Files
Updated by Toshi MARUYAMA over 9 years ago
$ cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
$ irb
1.9.3-p551 :001 > str = "test".force_encoding("UTF-7")
=> "\x74\x65\x73\x74"
1.9.3-p551 :002 > str = str.encode("UTF-8", :invalid => :replace,:undef => :replace, :replace => '?')
Encoding::ConverterNotFoundError: code converter not found (UTF-7 to UTF-8)
from (irb):2:in `encode'
from (irb):2
from /home/xxxx/.rvm/rubies/ruby-1.9.3-p551/bin/irb:12:in `<main>'
Updated by Toshi MARUYAMA over 9 years ago
- Priority changed from Normal to Low
It seems Ruby does not have UTF-7 converter.
"Code converter not found (UTF-8 to EUC-TW)": https://bugs.ruby-lang.org/issues/6995
Updated by Fred Sanger over 9 years ago
This encyclopedia article for UTF7 seems to be a good place to start
Updated by Pavel Rosický over 9 years ago
toshio harita: utf-7 is relevant only for email processing. Ruby still supports it, but it's a little bit tricky. This will work:
require "net/imap" Net::IMAP.decode_utf7(mail_body)
Updated by Karel Pičman almost 6 years ago
The same problem with Windows-1258. Attached patch should cover all not supported code pages.
Updated by Pavel Rosický almost 6 years ago
Иван Карельский
could you add a more realistic test case with an email as a fixture?
https://github.com/redmine/redmine/blob/3ac569c4d8c8b04a9cc33e3c2a05f8553d99827e/test/unit/mail_handler_test.rb#L542
it should be already fixed by
https://github.com/mikel/mail/pull/1112
Updated by Karel Pičman almost 6 years ago
A test with a real Vietnamese email added into the patch.
Updated by Bart Dopheide 9 days ago
The Microsoft Outlook client forwards plain text e-mails (like cron messages) in UTF-7, triggering this bug. The following diff works for me. Disclaimer: I'm completely new to Ruby. I declare the following work public domain.
--- lib/redmine/codeset_util.rb.orig 2026-03-17 08:40:05.000000000 +0100
+++ lib/redmine/codeset_util.rb 2026-07-17 14:25:52.476070776 +0200
@@ -17,6 +17,8 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+require 'base64'
+
module Redmine
module CodesetUtil
def self.replace_invalid_utf8(str)
@@ -31,6 +33,19 @@
str
end
+ ## https://www.redmine.org/issues/24910
+ def self.decode_utf7(str)
+ str.gsub(/\+([A-Za-z0-9\/]+)-/) do
+ b64 = Regexp.last_match(1)
+
+ utf16be = Base64.decode64(
+ b64 + "=" * ((4 - b64.length % 4) % 4)
+ )
+
+ utf16be.force_encoding("UTF-16BE").encode("UTF-8")
+ end.gsub(/\+-/, '+')
+ end
+
def self.to_utf8(str, encoding)
return if str.nil?
@@ -40,10 +55,20 @@
return str
end
enc = encoding.blank? ? "UTF-8" : encoding
- if enc.casecmp("UTF-8") != 0
- str.force_encoding(enc)
- str = str.encode("UTF-8", :invalid => :replace,
- :undef => :replace, :replace => '?')
+ if enc.casecmp("UTF-7").zero?
+ Rails.logger.info ">>> Decoding UTF-7 content."
+ str = decode_utf7(str)
+ elsif enc.casecmp("UTF-8") != 0
+ begin
+ str.force_encoding(enc)
+ str = str.encode("UTF-8", :invalid => :replace,
+ :undef => :replace, :replace => '?')
+ rescue Encoding::ConverterNotFoundError => e
+ Rails.logger.warn ">>> ConverterNotFoundError (#{enc} to UTF-8)"
+ Rails.logger.warn ">>> #{e.class}: #{e.message}"
+ Rails.logger.warn e.backtrace.join("\n")
+ str = replace_invalid_utf8(str)
+ end
else
str = replace_invalid_utf8(str)
end