[Solved] - Emoticons/Emojii/4-byte Non-UTF-8 characters
I logged this as a bug but it was rejected. But, it really is a bug, so I though I'd log the solution here in case anyone else has the same problem and wants that.
So, the problem is caused when a user sends an email, or pastes into the notes or description, text that includes many of the new emoticons that are now available. I thought it was a problem with Japanese characters but in fact, it's just that they use emoticons/emojii a lot more, so you can get it for US/English also.
So the problem is caused when you are using MySQL (i.e. a lot of people) and when you create the tables in the STANDARD UTF-8 character set, which, again, is a lot. In my case it's the bitnami stack which has pre-created those tables for me, but it isn't just bitnami stacks, it's a lot of other people will have this problem too.
So, the real solution is not to use UTF8 but instead UTF8MB4 which is the full 4 byte implementation of utf8. So you'd think that utf8 would actually be already fully able to handle that, but in fact in mysql it isn't. So, you have 2 choices -
1) Convert the various affected tables (issues, journals mostly) to UTF8MB4 - which I tried but I'm pretty useless at mysql so I couldn't get the syntax right. But that info is in here elsewhere as an issue I think.
2) You can simply strip/convert any 4 byte emoticon/emojii characters in redmine before you send that to the db. So, that's what this does.
So the solution, again, is kind of a hack because I'm not good enough to write a plugin or do it properly. But you can -
1) Install plugin custom_workflow. Which is awesome in any case.
2) Install this code as a flow for all projects. It will strip out any emoticons and replace with ? - again, if I were any good at ruby/redmine/mysql/anything I would try and do this better, but this works so... :-)
Ok, here it is (and es, that regex is really that big, lots of characters) -
Ok, that's t, no answers, responses required, and not sure if I put this in the correct place, but like I say wanted to log it anyway.
RE: Invalid Characters/Redmine crashes or rejects emails for non-UTF-8 characters - Solution - Added by Matthew Paul 11 months ago
PS - it is also a bug in this redmine, but whereas on my stack it actually crashes, here it simply cuts all text off after the offending 4 byte emoticon... which you obviously don't want to happen...
That's great info! Thanks very much!