Redmine

The Problem is, that for example the following diff-lines

- часа" 
+ часов" 

are parsed in Redmine as UTF-8 like this:

\xD1\x87\xD0\xB0\xD1\x81\xD0<span>\xB0</span>&quot;
\xD1\x87\xD0\xB0\xD1\x81\xD0<span>\xBE\xD0\xB2</span>&quot;

This is wrong, because the leading byte \xD0 is part of the cyrillic 2-Byte character "а" in the <span>-tag, but it's actually outside of the <span>-tag. Therefore charaters will be misinterpreted and will be displayed with "?".

Correct UTF-8 would be:

\xD1\x87\xD0\xB0\xD1\x81<span>\xD0\xB0</span>&quot;
\xD1\x87\xD0\xB0\xD1\x81<span>\xD0\xBE\xD0\xB2</span>&quot;

So we have for the first line "...<span>\xD0\xB0</span>..." instead of "...\xD0<span>\xB0</span>...". The attached patch searchs for the last leading byte, if the unmatching byte is a continuation byte (and not a leading byte or a single character byte).

A continuation byte has the binary format 10xxxxxx, so we can determine it with myContinuationByte.ord.between?(128, 191)

This problem occurs always, when the first determined difference between two bytes are continuation bytes. An other example in japanese you find in #13350.

A much better way to fix this problem is to set an UTF-8 encoding. :-)

Committed in, thanks.