Trac Importer Patch Coordination

Added by Bryce Nordgren almost 5 years ago

There are a number of patches on the issue tracker concerning Trac import, many of which represent substantial improvements and/or contributions. See the following screenshot:

Screenshot of importer issues

I merged two of these contributions into patch #5110. I stumbled across these because I was searching for issues which addressed shortcomings I needed to have addressed. As soon as I proudly made my own patch, I thought I'd look at all the open patches for all topics just to see what else was out there. To my dismay, I found that some of the wiki formatting improvements were already handled in a patch which wasn't on my radar. They solved many of the same problems independently.

This thread is a place to coordinate the integration of these disparate patches into a single Trac importer improvement which attempts to address the entire set of open issues while resolving redundancies. At the end of the effort, hopefully all the issues can be closed simultaneously.

The relevant set of issues are: #5110, #5065, #5035, #2748, #2504, #2114, #2054, #1008

importers.png - Screenshot of importer issues (49 KB)

Replies (14)

Ticket dependency tree - Added by Bryce Nordgren over 4 years ago

The following pattern is followed:

  • Ticket
    • Incorporates this ticket
    • Incorporates this ticket too...

Issues havnig to do with encoding/multibyte characters - Added by Bryce Nordgren over 4 years ago

The following issues both deal with multibyte character set problems:

The author of #1008 seems to feel that:

Why the task doesn't work?
Because Ruby's String#[pos, length] counts bytes (Ruby1.8), but database counts characters (varchar(30)).

The patch attached to #1008 alters the String#[pos,length] method to count characters instead of bytes, but is purported to be slow. A workaround suggested in #1008 is to change the database schema such that:
issue_categories name varchar(30) -> varchar(255)

The author of #5065 has chosen to encode(text), where "text" consists of all wiki page names, wiki text, wiki author names, and wiki page comments. In addition all attachment descriptions and author names are encode()-ed.

To confuse things, the Ruby Cookbook by Oreilly suggests that to handle international strings you need only put this at the beginning of your program:

$KCODE='u'
require 'jcode'

Specifically, the above code fragment is supposed to override most of the methods on the String class to make it capable of handling multibyte characters.

I'm very new to Ruby, but it seems that the obvious questions are:
  • Would adding the code fragment suggested by the Oreilly cookbook solve both problems? It certainly seems like it would solve #1008 for sure.
  • If not, did the author of #5065 encode() everything that needs to be encode()-ed, or are there more strings that need to be handled?

Thoughts on this?

Headers with no blank lines in between - Added by Bryce Nordgren over 4 years ago

#5035 has vastly improved the "header recognition" code, to the point where it no longer closely resembles what is on trunk.

#3910 reports a problem where two headers one immediately following another (with no blank line in between) are incorrectly converted into garbage. This problem is reported to be current as of 29 days ago (beginning of Feb, 2010).

I would suggest considering this a test case for #5035. If it addresses the problem reported in #3910, we can probably close #3910 as fixed. If not, #5035 should probably attempt to address this issue by incorporating the proposed solution.

Note that #3910 reports two "problems", and the first one essentially amounts to a request to support page hierarchies on import. This half of #3910 is more clearly articulated in #1842; so I would suggest closing #3910 as soon as the header import issue is fixed.

Tables and Special Links - Added by Bryce Nordgren over 4 years ago

Ticket #2114 consolidated many issues into a single unit, some of which have been closed already. This leads me to suspect that the solution proposed in #2114 was not adopted. However, it is worth examining the remaining three issues to ensure that #5035 addresses them.

I would suggest that we check to see that #5035 addresses #2068 and #2089, and make it address them if not. That way, when we close #5035, we can close #2114, #2054, #2068, and #2089.

Ticket keywords - Added by Bryce Nordgren over 4 years ago

The last issue listed above, #2504, deals with the import of Trac's ticket keywords. Might as well throw it on the pile and git'r'done.

RE: Trac Importer Patch Coordination - Added by Mike Stupalov over 4 years ago

I made several additions to the patch in #5035.

And another patch #5154, which decides #3392 and maybe #1008.

RE: Trac Importer Patch Coordination - Added by Mike Stupalov over 4 years ago

I update migrate_from_trac.rake

Changes:
  • Changed progress meter to a simple progress bar.
  • Moved requires for task ':migrate_from_trac_svn' to own module.

See latest version here: http://github.com/landy2005/Redmine-migrate-from-Trac

RE: Trac Importer Patch Coordination - Added by Anthony Callegaro over 3 years ago

Hey everybody,

I had to migrate our trac installation to redmine recently and because the vanilla script failed to import a lot of data I gave the scripts on this page a go.

From my testing the most advanced script is the one from Mike Stupalov in github it seems to regroup all patches previously posted on this thread.

After some testing I realised that even that script wasn't importing all trac's data so I customised it. My modifications add the following enhancements :
  • Handle many more ticket property changes :
    • change of ownership (re-assignement)
    • change of components (category)
    • change of priority
    • change in the summary (one line description)
    • change of issue type (bug, feature...)
    • change of completion ratio
  • Populate the completion ratio. This is done through a mapping hash that maps it to trac's resolution. By default :
    RATIO_MAPPING = {'' => 0,
                     'fixed' => 100,
                     'invalid' => 0,
                     'wontfix' => 0,
                     'duplicate' => 100,
                     'worksforme' => 0} 
    
  • Handle CCs field by adding watchers accordingly
  • Add start date (first's comment date) and the due date (when done_ratio is 100%) to populate the calendar and gantts charts
  • Add timelog entries to handle migration from timeandestimation plugin (hours spent, total hours), estimated hours already exists in redmine. Timelog comments links to the correct note in the ticket.
  • Add a new task tracker with a correct workflow and map it to trac's tasks.
  • Enable all redmine modules by default (yes it is pure lazyness :o) )
It also fixes some bugs in the current script :
  • Add a ticket's author to the list of project reporter
  • Put correct updated time on ticket
  • Plenty of small issues
And some ugly hack that might be removed :
  • Add an ugly hack to set the 'billable' custom field type to a checkbox. A proper fix would require to parse the trac.ini which would be a bigger implementation.

The script was tested on Redmine 1.0.1, 1.0.5, 1.1.2 and 1.2 (but not as extensively) on data coming from trac 0.10, 0.11 and 0.12.

I cleaned up the specific cases handling for our data and rebased the patch on Redmine's trunk (1.2). I didn't know from which revision Mike base his patch but it seems it was this one : source:trunk/lib/tasks/migrate_from_trac.rake@3517 so I merged Mike script's with my modifications and r4449 and r4599.

There were so many patches that I can't be sure how how many bugs got fixed by it. I guess it would be easier to ask people to test on the new version to see if their issue is fixed.

I also wrote a full how-to to migrate all data from Redmine to Trac.

Let me know what you think
Hope this help and that this can be included in a future version to avoid more patch diversions.

LeTic

RE: Trac Importer Patch Coordination - Added by Anthony Callegaro over 3 years ago

The patch and the script is the how-to but I should have also included them here,

They are also attached to this posts.

Sorry
LeTic

RE: Trac Importer Patch Coordination - Added by Gilles Cornu over 3 years ago

Thanx a lot!

I successfully integrated a sub-project previously managed by an external partner (running with Trac) into our Redmine Platform (recently upgraded to 1.2.0). I had to improve SVN conversion to be able to change the commit author names and also add the possibility to migrate the 'found in version' field, which is a standard feature in Trac. Everything ran perfectly...
Without all your precedent contributions, I'm sure I would never reach such '1to1' migration quality. Below, more details about the version of the script I used... There are still things to clean up and improve, but it worked for me and I guess I never have to import again from Trac... so I let the hand to the next one interested...

Source Code

Latest changes compared to Callegaro's version:

migrate_from_trac:

  • Added a new custom field Found in Version (based on feature #2096, Redmine ≥ 1.2.0 required)
  • Trackers find by Name instead by position (we could fetch other objects this way as well, like priorities or statuses)
    • In my environment the Redmine Tracker 'Support' is used like the Trac 'Task'
  • Sample for overriding method 'find_or_create_user' if Trac logins do not match with Redmine logins
  • Redmine Version Date is Trac Milestone Completed Date or Trac Milsetone Due Date if milestone not completed yet
  • In order to give more examples/ideas to Trac Importers, I let in comments some very ugly and very useful hacks :
    • skip unwanted custom fields
    • handle custom fields in 'list' format, see field 'browser' example
  • Affect 'Resolution' and 'TracID' custom fields only to the Trackers types that need it
  • Use a new Ticket ID anyway (no Ticket ID recycling)
  • Hack to be able run migrate_from_trac task with Ruby ≤ 1.8.6 (Problem introduced with nice progress-bars where .length() vs .count() methods backward compatibility has been broken by ruby 1.8.7)
  • Integrated fixes from Thomas Recloux, see http://www.redmine.org/issues/2748#note-1

migrate_from_trac_svn:
  • Original task renamed as 'migrate_svn_commit_properties', because now 2 migration operations are available:
    1. rename the author for each SVN commit (new feature)
    2. replace Trac Ticket IDs with Redmine Issue IDs in all SVN commit messages (original feature)

Remaining TODOs

Stuff that I won't implement since my migration tasks are done ;-)

  • Username Mapping should be correctly programmed/configured and shared between both tasks (migrate_svn_properties.migrate_authors + migrate_from_trac)
  • Refactor ModuleSvnMigrate to provide two independent tasks (instead one task with prompt):
    1. migrate_svn_commit_authors
    2. migrate_svn_commit_messages

migrate_from_trac.rake.Callegaro-Cornu.diff Magnifier - Changes proposed after patch from Anthony Callegaro (25 KB)

RE: Trac Importer Patch Coordination - Added by Gilles Cornu over 3 years ago

Gilles Cornu wrote:

  • I attach the diff between my script and Anthony's one, which I took as development base: attachment:migrate_from_trac.rake.Callegaro-Cornu.diff

sorry, previous .diff file was not in unified format. This one should be better handled by Redmine: migrate_from_trac.rake.Callegaro-Cornu.patch

migrate_from_trac.rake.Callegaro-Cornu.patch Magnifier - Changes proposed after patch from Anthony Callegaro (19 KB)

RE: Trac Importer Patch Coordination - Added by Gilles Cornu over 3 years ago

Gilles Cornu wrote:

  • I attach the diff between my script and Anthony's one, which I took as development base: attachment:migrate_from_trac.rake.Callegaro-Cornu.diff

sorry, previous .diff file was not in unified format. This one should be better handled by Redmine: migrate_from_trac.rake.Callegaro-Cornu.patch

Ouch, too late man... my diff arguments were inverted. Third and Last Try: migrate_from_trac.rake.Callegaro-Cornu.patch

migrate_from_trac.rake.Callegaro-Cornu.patch Magnifier - Changes proposed after patch from Anthony Callegaro (diff-errors in the 2 precedent posts should be fixed) (19 KB)

RE: Trac Importer Patch Coordination - Added by Verdura navarra almost 3 years ago

Ticket data can be imported from Bugzilla using the bugzilla2trac.py script, available in the contrib/ directory of the Trac distribution.

$ bugzilla2trac.py
bugzilla2trac - Imports a bug database from Bugzilla into Trac.

Usage: bugzilla2trac.py [options]

Available Options:
--db <MySQL dbname> - Bugzilla's database
--tracenv /path/to/trac/env - full path to Trac db environment
-h | --host <MySQL hostname> - Bugzilla's DNS host name
-u | --user <MySQL username> - effective Bugzilla's database user
-p | --passwd <MySQL password> - Bugzilla's user password
-c | --clean - remove current Trac tickets before importing
--help | help - this help info

Additional configuration options can be defined directly in the script.

Currently, the following data is imported from Bugzilla:

  • bugs
  • bug activity (field changes)
  • bug attachments
  • user names and passwords (put into a htpasswd file)
    verdura a domicilio
    You can only go as far as you push

RE: Trac Importer Patch Coordination - Added by Misty Pillow almost 3 years ago

This new version is one of the genius works I've read but if this be done in shortcut steps to make it lighter to grasp then that would be great. Thanks.[[www.echeapcarinsurancequotes.com]]

(1-14/14)