Project

General

Profile

Actions
Copy link

Defect #1663

closed

Redmine+Git+PostgreSQL 8.3 Cannot work with multi-bytes language

Added by Justin Waters almost 16 years ago. Updated about 14 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Jean-Philippe Lang
Category:
SCM
Target version:
0.8
Start date:
2008-07-18
Due date:
% Done:

0%

Estimated time:
Resolution:
Fixed
Affected version:

Description

I'm running in to the same issues as the following two bugs, but using Git as the repository:

#834
#917

RuntimeError: ERROR    C22021    Minvalid byte sequence for encoding "UTF8": 0xa0    HThis error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".    Fwchar.c    L1545    Rreport_invalid_encoding: INSERT INTO changesets ("commit_date", "comments", "committed_on", "revision", "scmid", "repository_id", "committer") VALUES('2005-04-16 15:24:43.000000', E'[PATCH] intel8x0: AC\'97 audio patch for Intel ESB2

This patch adds the Intel ESB2 DID\'s to the intel8x0.c file for AC\'97 audio
support.

Signed-off-by: �Jason Gaston <Jason.d.gaston@intel.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>', '2005-04-16 15:24:43.000000', E'c4c8ea948aa21527d502e87227b2f1d951bc506d', E'c4c8ea948aa21527d502e87227b2f1d951bc506d', 9, E'Jason Gaston <jason.d.gaston@intel.com>')

/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract_adapter.rb:150:in `log'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:407:in `execute'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:156:in `insert_sql'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:44:in `insert_without_query_dirty'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/query_cache.rb:19:in `insert'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/postgresql_adapter.rb:390:in `insert'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/base.rb:2272:in `create_without_callbacks'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:226:in `create_without_timestamps'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/timestamp.rb:29:in `create'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/base.rb:2238:in `create_or_update_without_callbacks'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/callbacks.rb:213:in `create_or_update'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/base.rb:1972:in `save_without_validation'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/validations.rb:934:in `save_without_transactions'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:108:in `save'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:80:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:100:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:108:in `save'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:120:in `rollback_active_record_state!'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:108:in `save'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/base.rb:572:in `create'
app/models/repository/git.rb:51:in `fetch_changesets'
app/models/repository/git.rb:50:in `reverse_each'
app/models/repository/git.rb:50:in `fetch_changesets'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:80:in `transaction'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/transactions.rb:100:in `transaction'
app/models/repository/git.rb:49:in `fetch_changesets'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/associations/association_proxy.rb:125:in `send'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/associations/association_proxy.rb:125:in `method_missing'
app/controllers/repositories_controller.rb:53:in `show'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in `send'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:1158:in `perform_action_without_filters'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:697:in `call_filters'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:689:in `perform_action_without_benchmark'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/usr/local/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/rescue.rb:199:in `perform_action_without_caching'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:678:in `perform_action'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/connection_adapters/abstract/query_cache.rb:33:in `cache'
/usr/local/lib/ruby/gems/1.8/gems/activerecord-2.0.2/lib/active_record/query_cache.rb:8:in `cache'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/caching.rb:677:in `perform_action'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `send'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:524:in `process_without_filters'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/filters.rb:685:in `process_without_session_management_support'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/session_management.rb:123:in `process'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/base.rb:388:in `process'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:171:in `handle_request'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:115:in `dispatch'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:126:in `dispatch_cgi'
/usr/local/lib/ruby/gems/1.8/gems/actionpack-2.0.2/lib/action_controller/dispatcher.rb:9:in `dispatch'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/rails.rb:76:in `process'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/rails.rb:74:in `synchronize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/rails.rb:74:in `process'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:159:in `process_client'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:158:in `each'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:158:in `process_client'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `new'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:285:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in `initialize'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in `new'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel.rb:268:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/configurator.rb:282:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/configurator.rb:281:in `each'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/configurator.rb:281:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/lib/mongrel/command.rb:212:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/local/bin/mongrel_rails:19:in `load'
/usr/local/bin/mongrel_rails:19

Environment

  • Server OS: Debian Etch Stable AMD64
  • Redmine: 0.7.3
  • Ruby: 1.8.6
  • Rails: 2.0.2
  • PostgreSQL: 8.3.3
  • Git: 1.4.4.4
Actions
Copy link
 #1

Updated by Jean-Philippe Lang over 15 years ago

I think you can configure git so that it outputs utf8 encoded commit logs.

Actions
Copy link
 #2

Updated by Jean-Philippe Lang over 15 years ago

r1767 adds support for commit messages reencoding.
Choose the appropriate source enconding in Application settings -> Repositories. Messages will be reencoded to UTF-8 before insertion in the database.

Actions
Copy link
 #3

Updated by Jean-Philippe Lang over 15 years ago

  • Status changed from New to Closed
  • Target version set to 0.8
  • Resolution set to Fixed
Actions
Copy link
 #4

Updated by Jérémy Lal about 15 years ago

  • Status changed from Closed to Reopened
  • Assignee set to Jean-Philippe Lang
Something quite similar is happening now, some guy used latin-1 to type his log message in git (it should have been UTF-8).
  • UTF-8 is checked in settings/repositories.
  • locale and database : unicode
Rendering /usr/share/redmine/public/500.html (500 Internal Server Error)

Processing RepositoriesController#show (for 90.20.10.177 at 2009-01-22 07:54:43) [GET]
  Session ID: bdea66d703997d016cd8d48f05b88262
  Parameters: {"action"=>"show", "id"=>"test", "controller"=>"repositories"}

ActiveRecord::StatementInvalid (PGError: ERREUR:  séquence d'octets invalide pour l'encodage « UTF8 » : 0x88
HINT:  Cette erreur peut aussi survenir si la séquence d'octets ne correspond pas
au jeu de caractères attendu par le serveur, le jeu étant contrôlé par
« client_encoding ».
: INSERT INTO "changesets" ("committed_on", "comments", "commit_date", "scmid", "user_id", "revision", "repository_id", "committer") VALUES('2009-01-19 18:39:28.000000', E'HERE ~^H ENCODING ERROR', '2009-01-19 18:39:28.000000', E'09ddea6b5f0822f8181bbc616beed35b04176e2f', 3, E'09ddea6b5f0822f8181bbc616beed35b04176e2f', 1, E'Some Name <somename@some.com>') RETURNING "id"):
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract_adapter.rb:147:in `log'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/postgresql_adapter.rb:484:in `execute'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/postgresql_adapter.rb:929:in `select_raw'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/postgresql_adapter.rb:916:in `select'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:59:in `select_all'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:80:in `cache_sql'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/query_cache.rb:59:in `select_all'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:13:in `select_one'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:19:in `select_value'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/postgresql_adapter.rb:433:in `insert'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/base.rb:2517:in `create_without_callbacks'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/callbacks.rb:220:in `create_without_timestamps'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/timestamp.rb:29:in `create'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/base.rb:2483:in `create_or_update_without_callbacks'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/callbacks.rb:207:in `create_or_update'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/base.rb:2217:in `save_without_validation!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/validations.rb:921:in `save_without_dirty!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/dirty.rb:83:in `save_without_transactions!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:110:in `save!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:79:in `transaction'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:98:in `transaction'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:110:in `save!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:118:in `rollback_active_record_state!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:110:in `save!'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/validations.rb:892:in `create!'
    /app/models/repository/git.rb:50:in `fetch_changesets'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:79:in `transaction'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/transactions.rb:98:in `transaction'
    /app/models/repository/git.rb:49:in `fetch_changesets'
    /lib/redmine/scm/adapters/git_adapter.rb:165:in `revisions'
    /lib/redmine/scm/adapters/git_adapter.rb:151:in `each_line'
    /lib/redmine/scm/adapters/git_adapter.rb:151:in `revisions'
    /lib/redmine/scm/adapters/abstract_adapter.rb:181:in `call'
    /lib/redmine/scm/adapters/abstract_adapter.rb:181:in `shellout'
    /lib/redmine/scm/adapters/abstract_adapter.rb:179:in `popen'
    /lib/redmine/scm/adapters/abstract_adapter.rb:179:in `shellout'
    /lib/redmine/scm/adapters/abstract_adapter.rb:165:in `shellout'
    /lib/redmine/scm/adapters/git_adapter.rb:145:in `revisions'
    /app/models/repository/git.rb:47:in `fetch_changesets'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/associations/association_proxy.rb:173:in `send'
    /var/lib/gems/1.8/gems/activerecord-2.1.2/lib/active_record/associations/association_proxy.rb:173:in `method_missing'
    /app/controllers/repositories_controller.rb:69:in `show'

Environment
  • Server OS : Debian etch
  • Redmine : 0.8.0
  • Ruby : 1.8.7
  • Rails : 2.1.2
  • Postgresql : 8.3.5
  • Git : 1.5.6.5

Actions
Copy link
 #5

Updated by Jérémy Lal about 15 years ago

indeed git-log returns the comment as :
some stuff <88> and stuff
it should be "à", but it seems that the client's editor screwed that character encoding.
is there a way to correct this ? i'm not going to do a rebase to correct the commit messages...

Actions
Copy link
 #6

Updated by Jérémy Lal about 15 years ago

after investigation, instead of doing nothing to the string, may be this would be better :
in app/models/changeset.rb, line 148 :

return Iconv.conv('UTF-8//IGNORE', encoding, str)

that would strip illegal characters, so it's failure proof.

Actions
Copy link
 #7

Updated by Jérémy Lal about 15 years ago

i just made that modification, restarted web server, and i still get the error.
It seems
changeset.rb#to_utf8
is not called. I'm probably missing something.
Sorry for the hammering.

Actions
Copy link
 #8

Updated by Jean-Philippe Lang about 15 years ago

  • Status changed from Reopened to Closed

It's not called if 'Commit messages encoding' is set to UTF-8 (no reencoding required).
It should be set to something like ISO-8859-1 since your git logs are not UTF-8.

Open a new ticket if needed.

Actions
Copy link
 #9

Updated by Oren Laadan about 14 years ago

Hi,

I also hit this one before, and now again.
You asked to open a new ticket - so here it is:
Issue #4773

Thanks,

Oren.

Actions
Copy link

Also available in: Atom PDF