Patch #5035

Migrate from trac - Errors in wiki formating

Added by Mike Stupalov almost 8 years ago. Updated about 7 years ago.

Status:NewStart date:2010-03-10
Priority:NormalDue date:
Assignee:-% Done:

60%

Category:Importers
Target version:-

Description

I have written a patch for a script of migration from Trac.

The list of the brought changes:
  1. More complete regular expressions in many replacements of formatting wiki pages.
  2. The first and last name of the user are correctly recognised
  3. convert [[BR]] - before all
  4. Titles correctly converted and only h1. to h6. (Also work for #3910)
  5. External Links correctly converted (include mailto: links)
  6. Ticket links correctly converted
  7. Internal Links correctly converted
  8. Inline code are converted
  9. Tables are converted
  10. Lists correctly converted (include numbered)
  11. Inline images are converted
  12. Changed default database adapter to sqlite3

The patch works for audit 3517 and above
The additional information is in remarks. The patch works for r3517 and above.

File: migrate_from_trac_v1.patch

migrate_from_trac_v1.patch Magnifier (6.33 KB) Mike Stupalov, 2010-03-10 14:40

migrate_from_trac.rake Magnifier (40 KB) Mike Stupalov, 2010-03-19 14:15

migrate_from_trac_v3.patch Magnifier (23.7 KB) Mike Stupalov, 2010-03-19 14:15

migrate_from_trac.rake Magnifier - migrate_from_trac (v4) (41.6 KB) Mike Stupalov, 2010-03-22 15:08

migrate_from_trac_v4.patch Magnifier (25.9 KB) Mike Stupalov, 2010-03-22 15:08

migrate_from_trac_v5.patch Magnifier (27.5 KB) Mike Stupalov, 2010-03-23 15:44

migrate_from_trac.rake Magnifier (45.8 KB) Bryce Nordgren, 2010-04-28 17:56


Related issues

Related to Redmine - Patch #5110: Trac importer improvement patch Closed 2010-03-17

History

#1 Updated by Bryce Nordgren almost 8 years ago

This appears to be more comprehensive in terms of wiki formatting fixes than #5110. #5110 adds ticket number mapping on import and Subversion log message changing.

#2 Updated by Mike Stupalov almost 8 years ago

Very good. I combined these patches.

For migrate svn, use:

rake redmine:migrate_from_trac_svn RAILS_ENV="production"

out:

WARNING: all commit messages with references to trac pages will be modified
Are you sure you want to continue ? [y/N] y

Subversion repository url []: http://svn.server.dom/project
Subversion repository username []: user
Subversion repository password []: secret
Redmine project identifier []: project-id

Also made a few corrections and additions to convert textile format (including previous):

  • Convert macro - [TOC]
  • Convert source links.
  • In links recognized quotes and double quotes. (Because Trac supports both types of quotes).

remaining problems of converting the format:

  • before and after the tables have to add blank lines. (Trac supports a table without the initial and final blank line, and Redmine - no)
  • text within the code should be left as is. (now converted to all, including the code.)

Patch: migrate_from_trac_v3.patch
File: migrate_from_trac.rake

#3 Updated by Bryce Nordgren almost 8 years ago

Hi Mike,

I closed #5110 as resolved because you incorporated it here. I did some more issue triage over on the forums. There were open issues related to multibyte characters, header imports, "special links", and importing ticket keywords. I also included a "dependency graph" of sorts, indicating which issues tried to address problems raised in which other issues.

I'm new to Ruby, so I constrained myself to highlighting the main issues. I'd be grateful if you took a look. If everything on that page is addressed by your patch, I think a whole slew of issues could be closed when #5035 is.

Check out: http://www.redmine.org/boards/1/topics/12273

Thanks for your work!

#4 Updated by Mike Stupalov over 7 years ago

Updated version.
Changes:

  • completely fixed bugs #3910
  • small fixes conversion links.
  • added conversion attachments links (ex. [attachment:file.txt]). Closes #2146
  • importing Trac keywords (include corrected patch from #2504)

Patch: migrate_from_trac_v4.patch
File: migrate_from_trac.rake

#5 Updated by Mike Stupalov over 7 years ago

I founded and fix some new issues:
  1. Not converted an attachments in non-latin filename encoding.
    • Trac encodes the filenames in hexadecimal number using uppercase letters.
    • For the wiki pages also need to encode the subdirectory.
  2. "Issue categories" not converted "Assigned to" user.
  3. If specify the "Target project identifier" with a capital letter, then the error occurred.

Patch: migrate_from_trac_v5.patch

While no changes are made to the main repository, I made a fork of the file migrate_from_trac.rake here: http://github.com/landy2005/Redmine-migrate-from-Trac

P.S. All the changes I check on 3 big projects that we want to migrate from Trac.

#6 Updated by Mike Stupalov over 7 years ago

I update migrate_from_trac.rake

Changes:
  • Changed progress meter to a simple progress bar.
  • Moved requires for task ':migrate_from_trac_svn' to own module.

See latest version here: http://github.com/landy2005/Redmine-migrate-from-Trac

#7 Updated by Bryce Nordgren over 7 years ago

Hi Mike,

Still watching your progress and appreciate all your work. I'm actually delaying the migration of our repositories until your improvements are finalized. :)

Thanks again!

#8 Updated by Bryce Nordgren over 7 years ago

Hi Mike. I'm starting to use this (the version on github) to migrate our collection of Trac instances over to a new server. Great job! :)

I did have to add "PageTemplates" to the list of Trac pages to discard. I also commented out the part that creates a wiki page for every milestone.

I'm bumping this to 90% done just because I think the only thing left to do is have the admins integrate it with the main code base. I can verify that it works with my redmine, which was built from trunk on April 26, 2010.

#9 Updated by Bryce Nordgren over 7 years ago

I know there are too many files named the same thing. This migrate_from_trac.rake is the version from github that I used. It has "PageTemplates" added, but it will still create a wiki page for every milestone.

#10 Updated by Mike Stupalov over 7 years ago

OK, I also add "PageTemplates" to the list of Trac pages to discard. :)
But I do not understand what you propose to do with milestone? In your code, no changes associated with it.

In addition, I discovered another problem. The script does not convert the CC list to watchers. I'm working on correcting this.

#11 Updated by Bryce Nordgren over 7 years ago

Well, in the version I uploaded, I didn't change the milestone part. In the version I used to migrate my trac instances, I commented out the 6 lines that create a wiki page for every milestone. Basically I did this because I have no use for a wiki page with the milestone text on it. The "version" page (with the same text) serves my purposes.

However, I didn't consider that to be an "error", just my personal preference, so I didn't change the code for that part.

Sorry for the confusion.

#12 Updated by Bryce Nordgren over 7 years ago

One thing I did find is that when I converted trac instances where we never bothered to make a personal page (e.g., first name, last name, email), the script now assigns all tickets to Redmine Admin. As a matter of fact, I'm finding that converting trac instances with properly defined users causes the conversion script to create many unwanted accounts. This is mostly because, at the same time I'm switching from Trac to Redmine, I'm also switching from authenticating off of htpasswd accounts to the corporate Active Directory. I find myself locking the accounts which were created during the conversion process, but which can never be used again due to the conversion.

#13 Updated by Michalis Miatidis over 7 years ago

Everything OK except:

1) Requires merge with Patch #5764 to work with TRAC 0.12

2) At my local installation, migrated everything apart from users with Firstname of single world and Lastname as null.

#14 Updated by patrick Sherman about 7 years ago

Hi,

This rake works perfectly with tickets but not with the wiki.
It stop conversion at the first accent and links with another page are lost.
What should I need to do?

Regards

#15 Updated by Alexander Sapozhnikov about 7 years ago

redmine:migrate_from_trac_svn doesn't work. First time I got message:

Could not find issue custom field 'TracID'

I add custom field TracID and got new error:

wrong number of arguments (7 for 4)

When spawned with --trace both on Redmine 0.9.6 and on 1.0.2 too:

path/to/redmine/lib/tasks/migrate_from_trac.rake:910:in `initialize'
path/to/redmine/lib/tasks/migrate_from_trac.rake:910:in `new'
path/to/redmine/lib/tasks/migrate_from_trac.rake:910:in `scm'
path/to/redmine/lib/tasks/migrate_from_trac.rake:873:in `migrate'
path/to/redmine/lib/tasks/migrate_from_trac.rake:935

#16 Updated by Alexander Sapozhnikov about 7 years ago

Line 910 is:

@scm ||= SvnExtendedAdapter.new @@svn_url, @@svn_url, @@svn_username, @@svn_password, 0, "", nil
I remove hide 3 last arguments, but see same error.

#17 Updated by Alexander Sapozhnikov about 7 years ago

redmine:migrate_from_trac works, but acts as described in #6632 — it ignore fields lft, rgt and root_id in table issues.

#18 Updated by Mike Stupalov about 7 years ago

  • Assignee changed from Jean-Philippe Lang to Felix Schäfer
  • % Done changed from 90 to 60

All many thanks for the found errors.
Unfortunately the main developers redmine don't give attention to a script of migration from trac.

I will try to look at the arisen errors. But in connection with study at me hasn't enough time, therefore fast correction I do not promise :)I will try to look at the arisen errors. But in connection with study at me hasn't enough time, therefore fast correction I do not promise :)

Please send more version informations trac, what database is used, a debug output of a script and certainly a command output:

RAILS_ENV="production" script/about

#19 Updated by Felix Schäfer about 7 years ago

  • Assignee deleted (Felix Schäfer)

Please don't assign issues to anyone who hasn't said to do so, thanks.

#20 Updated by Michał Wróbel about 7 years ago

I made some patches which improve TracWiki conversion.

Also available in: Atom PDF