Diff outputs become ??? in some non ASCII words.
|Assignee:||Toshi MARUYAMA||% Done:|
fix that diff outputs become ??? in some non ASCII words (#12641)
Contributed by Filou Centrinov.
#1 Updated by Filou Centrinov over 5 years ago
- File unified_diff.rb.diff added
The Problem is, that for example the following diff-lines
- часа" + часов"
are parsed in Redmine as UTF-8 like this:
This is wrong, because the leading byte
\xD0 is part of the cyrillic 2-Byte character "
а" in the <span>-tag, but it's actually outside of the <span>-tag. Therefore charaters will be misinterpreted and will be displayed with "?".
Correct UTF-8 would be:
So we have for the first line "
...<span>\xD0\xB0</span>..." instead of "
...\xD0<span>\xB0</span>...". The attached patch searchs for the last leading byte, if the unmatching byte is a continuation byte (and not a leading byte or a single character byte).
A continuation byte has the binary format 10xxxxxx, so we can determine it with
This problem occurs always, when the first determined difference between two bytes are continuation bytes. An other example in japanese you find in #13350.