UTF-8 encoding not showing correctly when looking highlighted file contents
|Target version:||Candidate for next minor release|
Ruby version 1.9.3 (x86_64-linux) RubyGems version 1.8.11 Rack version 1.4 Rails version 3.2.5 Database adapter mysql2 Database schema version 20120422150750 Git version 184.108.40.206
When I request file to see it contents (repository/revisions/HASH/entry) instead UTF-8 text I get '???'.
I'm using Git SCM and my files are valid UTF-8 (without BOM). I have this problem with Chineses, Russian, Thai and other scripts than latin.
However seeing diff's and attached utf-8 files are okay.
#4 Updated by Toshi MARUYAMA 12 months ago
Redmine uses "git show".
Git 220.127.116.11, "git show --help" says
The contents of the blob objects are uninterpreted sequences of bytes. There is no encoding translation at the core level.
#5 Updated by Troex Nevelin 12 months ago
I understand that git stores files in binary form, but calling from console:
git show --no-color HEAD:.../lang/ru/component.php
returns UTF-8 valid text, as I understand Redmine tries to guess encoding and sanitise content making sure no invalid characters pass to view.
For example source:trunk/config/locales/ja.yml this displays up correctly (but it uses SVN).
I think there is encoding guess problem in source:tags/2.0.1/lib/redmine/codeset_util.rb#L84 calling
.to_utf8_by_setting_internal(str) sets ASCII-8BIT encoding on line 94?
#6 Updated by Toshi MARUYAMA 12 months ago
- File gh-new-d7e2a66d.png added
I cannot reproduce.
Could you attach this "git show" output file?
#7 Updated by Troex Nevelin 12 months ago
- File git-show-component.php.txt added
git show --no-color HEAD:.../lang/ru/component.php > git-show-component.php.txt
I'm running Redmine on Debian 6, with ruby 1.9.3p125 (2012-02-16) [x86_64-linux] package compiled from debian ruby repository, using unicorn rack server.
I'm almost sure this is local related problem. Can you guide me how to debug this problem? I'm familier with ruby and ror. I have tried to output raw content in
app/views/common/_file.html.erb but it gives me
ActionView::Template::Error (incompatible character encodings: UTF-8 and ASCII-8BIT) error
#9 Updated by Troex Nevelin 12 months ago
- File issue-attached-files.png added
- File test-file-with-php-ext.png added
- File test-file-with-txt-ext.png added
I've made one more test on my setup, I've attached the same file to an issue but with different extensions .txt and .php and when trying to see attached file I get an issue with viewing syntax highlighted file. So this is not only Git related problem.
But no issue here in this ticket.
# grep coderay Gemfile.lock coderay (1.0.6) coderay (~> 1.0.6)
#16 Updated by Etienne Massip about 1 month ago
- Subject changed from UTF-8 encoding not showing correctly when looking file contents to UTF-8 encoding not showing correctly when looking highlighted file contents
- Category set to Text formatting
- Status changed from New to Confirmed
- Target version set to Candidate for next minor release
Upgrade dep to 1.0.9 or 1.1.