Project

General

Profile

RedmineMigrate: invalid byte sequence in UTF-8 after migrate from Trac 0.12

Added by Stephan Krinetzki over 9 years ago

Hello,

i've got some issues with importing UTF-8 MySQL table to redmine. My trac database is mysql, my redmine database too.
After getting the first "invalid byte sequence in UTF-8" Error, i just changed this code:


   # Basic wiki syntax conversion
      def self.convert_wiki_text(text)
        # Titles
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/^(\=+)\s(.+)\s(\=+)/) {|s| "\nh#{$1.length}. #{$2}\n"}
        # External Links
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/\[(http[^\s]+)\s+([^\]]+)\]/) {|s| "\"#{$2}\":#{$1}"}
        # Ticket links:
        #      [ticket:234 Text],[ticket:234 This is a test]
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/\[ticket\:([^\ ]+)\ (.+?)\]/, '"\2":/issues/show/\1')
        #      ticket:1234
        #      #1 is working cause Redmine uses the same syntax.
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/ticket\:([^\ ]+)/, '#\1')
        # Milestone links:
        #      [milestone:"0.1.0 Mercury" Milestone 0.1.0 (Mercury)]
        #      The text "Milestone 0.1.0 (Mercury)" is not converted,
        #      cause Redmine's wiki does not support this.
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/\[milestone\:\"([^\"]+)\"\ (.+?)\]/, 'version:"\1"')
        #      [milestone:"0.1.0 Mercury"]
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/\[milestone\:\"([^\"]+)\"\]/, 'version:"\1"')
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/milestone\:\"([^\"]+)\"/, 'version:"\1"')
        #      milestone:0.1.0
        text = text
                .encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
                .gsub(/\[milestone\:([^\ ]+)\]/, 'version:\1')

This fixed the first UTF-8 Error, the next one drives me crazy:

$/var/data/redmine# rake redmine:migrate_from_trac RAILS_ENV="production" --trace
** Invoke redmine:migrate_from_trac (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:migrate_from_trac

WARNING: a new project will be added to Redmine during this process.
Are you sure you want to continue ? [y/N] y

Trac directory []: /home/stephan/it_trac
/home/stephan/it_trac/attachments doesn't exist!
Trac directory []: /home/stephan/it_trac/trac
Trac database adapter (sqlite3, mysql2, postgresql) [sqlite3]: mysql
Trac database host [localhost]: 
Trac database port [3306]: 
Trac database name []: it_trac
Trac database schema [public]: 
Trac database username []: root
Trac database password []: ****
Trac database encoding [UTF-8]: 
Target project identifier []: it_trac

Migrating components................
Migrating milestones
Migrating custom fields.
Migrating tickets.........................rake aborted!
invalid byte sequence in UTF-8
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/core_ext/object/blank.rb:105:in `=~'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/core_ext/object/blank.rb:105:in `!~'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/core_ext/object/blank.rb:105:in `blank?'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/errors.rb:255:in `block in add_on_blank'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/errors.rb:253:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/errors.rb:253:in `add_on_blank'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/validations/presence.rb:8:in `validate'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:310:in `_callback_before_1499'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:506:in `_run__1411299390745521346__validate__4113007513488049294__callbacks'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:385:in `_run_validate_callbacks'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/validations.rb:228:in `run_validations!'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/validations/callbacks.rb:53:in `block in run_validations!'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:403:in `_run__1411299390745521346__validation__4113007513488049294__callbacks'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:385:in `_run_validation_callbacks'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/validations/callbacks.rb:53:in `run_validations!'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activemodel-3.2.19/lib/active_model/validations.rb:195:in `valid?'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/validations.rb:69:in `valid?'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/validations.rb:77:in `perform_validations'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/validations.rb:50:in `save'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:208:in `transaction'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:259:in `block in save'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/transactions.rb:258:in `save'
/var/data/redmine/lib/tasks/migrate_from_trac.rake:542:in `block (2 levels) in migrate'
/var/data/redmine/lib/tasks/migrate_from_trac.rake:81:in `fake'
/var/data/redmine/lib/tasks/migrate_from_trac.rake:542:in `block in migrate'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/relation/batches.rb:21:in `block (2 levels) in find_each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/relation/batches.rb:21:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/relation/batches.rb:21:in `block in find_each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/relation/batches.rb:72:in `find_in_batches'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/relation/batches.rb:20:in `find_each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/activerecord-3.2.19/lib/active_record/querying.rb:8:in `find_each'
/var/data/redmine/lib/tasks/migrate_from_trac.rake:528:in `migrate'
/var/data/redmine/lib/tasks/migrate_from_trac.rake:835:in `block (2 levels) in <top (required)>'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:236:in `call'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:236:in `block in execute'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:231:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:231:in `execute'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:175:in `block in invoke_with_call_chain'
/usr/local/rvm/rubies/ruby-2.0.0-p576/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:168:in `invoke_with_call_chain'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/task.rb:161:in `invoke'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:149:in `invoke_task'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:106:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:106:in `block in top_level'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:115:in `run_with_threads'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:100:in `top_level'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:78:in `block in run'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:165:in `standard_exception_handling'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/lib/rake/application.rb:75:in `run'
/usr/local/rvm/gems/ruby-2.0.0-p576/gems/rake-10.1.1/bin/rake:33:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.0.0-p576/bin/rake:23:in `load'
/usr/local/rvm/gems/ruby-2.0.0-p576/bin/rake:23:in `<main>'
/usr/local/rvm/gems/ruby-2.0.0-p576/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.0.0-p576/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => redmine:migrate_from_trac

Any ideas, how to fix this Error?

EDIT: script/about

sh: 1: darcs: not found
sh: 1: hg: not found
sh: 1: cvs: not found
sh: 1: bzr: not found
Environment:
  Redmine version                2.5.2.stable.13413
  Ruby version                   2.0.0-p576 (2014-09-19) [x86_64-linux]
  Rails version                  3.2.19
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.6.17
  Git                            1.7.10.4
  Filesystem                     
Redmine plugins:
  no plugin installed