MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8)

Added by Deoren Moor 5 months ago

Background

I've seen these errors off/on for a while now and the workaround is always to manually process the emails, either copy/paste into a new "ticket" or strip out all non-ASCII characters and reprocess. As we're increasing use of Redmine within our organization we're running into this a lot more often, especially with groups that process foreign language material.

A small sample of the messages I've seen:

  • MailHandler: an unexpected error occurred when receiving email: code converter not found (Windows-1258 to UTF-8)
  • MailHandler: an unexpected error occurred when receiving email: code converter not found (UTF-7 to UTF-8)

Thank you in advance for looking over this and for any suggestions you may have to help resolve it.

Config files

grep -ri utf /opt/redmine/config/ | grep -v locales

/opt/redmine/config/database.yml.example:  encoding: utf8
/opt/redmine/config/database.yml.example:  encoding: utf8
/opt/redmine/config/database.yml.example:  encoding: utf8
/opt/redmine/config/application.rb:    config.encoding = "utf-8" 
/opt/redmine/config/settings.yml:# encodings used to convert repository files content to UTF-8
/opt/redmine/config/settings.yml:# encoding used to convert commit logs to UTF-8
/opt/redmine/config/settings.yml:  default: 'UTF-8'
/opt/redmine/config/database.yml:  encoding: utf8
/opt/redmine/config/database.yml:  encoding: utf8

bin/about

bundle list

Scraping script

In the script we're using to scrape emails (basically a wrapper around the rake task) we're setting multiple environment variables in an attempt to workaround the issue. Here is a trimmed down version of that script:

Replies (4)

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Matthew Paul 5 months ago

Deoren - that info you posted on my forum thread was excellent -

Some related issues: #24992, #21398

#21398#note-10 in particular goes into a good level of detail.

and really it gives the 'solution' which is to run a conversion on your DB to handle utf8mb4. But, it all takes a long time and some technical expertise, neither of which I have. Even your script would seem to require some knowledge of running a wrapper script etc.

For many (like me) - what is needed is a simple plugin that accepts any inbound email, updates what it can in the description/notes and simply converts to '?' (or similar) anything that causes an error. That's not a great loss because we're only talking about emojii and the like in any case. My workflow plugin + script does that, but it's not a single plugin, and it may miss stuff. So I don't know if any such plugin would be available, basically to never 'error out' but always to log what it can as best it can. But if you find something please let me know I'd be interested :-)

I know you're looking for a more complete solution, so thanks again and sorry to bust into this thread...

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Toshi MARUYAMA 5 months ago

Matthew Paul wrote:

Deoren - that info you posted on my forum thread was excellent -

Some related issues: #24992, #21398

#21398#note-10 in particular goes into a good level of detail.

and really it gives the 'solution' which is to run a conversion on your DB to handle utf8mb4.

No. It is because Ruby does not have code converter of Windows-1258 and UTF-7.

$ cat ja.yml | iconv -f utf8 -t EUC-JP | ruby -ne '$_.force_encoding("EUC-JP") ;puts $_.encode("UTF-8")' > /dev/null
$ cat vi.yml | iconv -f utf8 -t Windows-1258 | ruby -ne '$_.force_encoding("Windows-1258") ;puts $_.encode("UTF-8")' > /dev/null
-e:1:in `encode': code converter not found (Windows-1258 to UTF-8) (Encoding::ConverterNotFoundError)
    from -e:1:in `<main>'
$ cat vi.yml | iconv -f utf8 -t utf-7 | ruby -ne '$_.force_encoding("UTF-7") ;puts $_.encode("UTF-8")' > /dev/null
-e:1:in `encode': code converter not found (UTF-7 to UTF-8) (Encoding::ConverterNotFoundError)
    from -e:1:in `<main>'

RE: MailHandler: an unexpected error occurred when receiving email: code converter not found (xxx to UTF-8) - Added by Matthew Paul 5 months ago

So, according to that ticket, it says it can be handled?

require "net/imap"
Net::IMAP.decode_utf7(mail_body)

Would this go into mailer.rb? Or somewhere else?

(1-4/4)