Feature #3647

Data Import/Export System

Added by Mr Embedded over 5 years ago. Updated about 1 month ago.

Status:NewStart date:2009-07-21
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Importers
Target version:-
Resolution:

Description

Hi All,

I think it will be very useful to have some kind of data import/export system for Redmine with the following features:

1. Entire projects could exported or imported to different instances of Redmine. I know this is dependent on user information being included and restored with the project in some capacity.

2. Selectively export/import administrative settings including user information. This one probably needs to be smart enough to allow a controlled data merge to take place during an import.

3. This could serve also as the basis of a backup system for various areas of Redmine

I got the idea as I was deleting a test project. If you can delete a project in its entirety, you should be able to archive all of that information before deletion in one simple step as the delete vs copy logic is pretty much the same. And if you can do that and then delete your project, you should be able to take that exact same archive and restore it.

I haven't investigated this at the code level but wanted to post this because it seems a good idea at this moment and a search of data export did not provide any interesting hits


Related issues

Related to Feature #17259: Import_Button Closed
Related to Feature #7884: Import issues from Excel or CSV file New 2011-03-16
Duplicated by Feature #9993: Export and import all information (Database) Closed

History

#1 Updated by Kirill Ponomarev over 5 years ago

+1
Good idea.
Also it can be a simple way to make backup.

#2 Updated by Hans Fangohr over 5 years ago

I agree this would be great, and just what we need for a number of cases.

This relates to http://www.redmine.org/boards/2/topics/5754#message-5761, where just this functionality is required.

#3 Updated by Mr Embedded over 5 years ago

I would love to help out if I can. I am not a coder. My skill set is more geared to project management and testing/debugging. Let me know if I can assist.

#4 Updated by Nanda Palaniswamy almost 5 years ago

+1

Recently I received a request to Import only a specific project from our test server to PROD. (MySQL Server)

I did it manually exported records from table by table & imported them in PROD. Took few hours for about 100 Issues.

It will be nice to have this feature to export a single or multiple project data from one server & import to another RedMine server.

#5 Updated by Andrea Denzler almost 5 years ago

+1

Yes, it would be very nice to export/import projects and administration/user settings. Good for backup, moving between servers :)

#6 Updated by Piotr Bazan over 4 years ago

+1

#7 Updated by Balazs Nadasdi over 4 years ago

+1

We have 3 redmine server... And now i try to merge it into one server...
Server 1: 14 project [100+ issues]
Server 2: 3 project [100+ issues]
Server 3: 8 project [100+ issues]

Pls make a migration tool...

/* mysql_dump is not a good way: 'issueId = (1..100+) is in all server */

thx

#8 Updated by Peter Schuster over 4 years ago

+1

#9 Updated by Eraldo Girardi over 4 years ago

+1

#10 Updated by minkbear minkbear over 4 years ago

+1

#11 Updated by Marcus Bitzl over 4 years ago

+1

#12 Updated by Pieter Smith over 4 years ago

+1

#13 Updated by Nikolay Kotlyarov over 4 years ago

+1

#14 Updated by Daniel Nautré over 4 years ago

+1

#15 Updated by Denis Gorodnichev over 4 years ago

+1

#17 Updated by Andreas Schuh over 4 years ago

+1

#18 Updated by saturntechweb saturntechweb over 4 years ago

Raito Yitsushi wrote:

+1

We have 3 redmine server... And now i try to merge it into one server...
Server 1: 14 project [100+ issues]
Server 2: 3 project [100+ issues]
Server 3: 8 project [100+ issues]

Pls make a migration tool...

/* mysql_dump is not a good way: 'issueId = (1..100+) is in all server */

thx

dfdf

#19 Updated by Colan Schwartz over 4 years ago

+1

#20 Updated by Roland Discein about 4 years ago

+1

#21 Updated by Thomas Zlika about 4 years ago

+1
The real problem relates to the issues IDs (and in particular the fact that these IDs can be used in SVN commit messages to reference the issue). If Redmine had project-specific issue IDs (for example: shortName-1234), it would be simple to import a project.
With global server IDs, I fear that the correct import of a project will not be possible (because that will change the IDs, so at least the references on SVN commits will be lost or wrong).
However, it's a real problem !

#22 Updated by Andrea Denzler about 4 years ago

Thomas Zlika wrote:

+1
The real problem relates to the issues IDs (and in particular the fact that these IDs can be used in SVN commit messages to reference the issue). If Redmine had project-specific issue IDs (for example: shortName-1234), it would be simple to import a project.
With global server IDs, I fear that the correct import of a project will not be possible (because that will change the IDs, so at least the references on SVN commits will be lost or wrong).
However, it's a real problem !

mmm....

1. If you have two different SVN repositories you must first of all merge them, maintaining all the history and creating a report with OldCommitReference->NewCommitReference. This is not a easy task, in any case it is related only to SVN. With such a report you have to update the RedMine servers renaming oldReference with newReference wherever it was used (wiki, notes, etc).

2. Once you merge two RedMine Servers you need a report with OldIssueID->NewIssueId. With that report you have to update all the Commit Comments in your SVN repository, replacing old issue id with new issue id as described in the Subversion FAQ (the Import process could do that automatically). You need to replace the issue IDs not only in SVN but also in all the notes, wiki of RedMine.

Sure, with a global issue ID (like with GIT commits) it would be easier to merge two servers, but the simple incremental Issue ID number is more human friendly....

#23 Updated by Anthony Cartmell almost 4 years ago

+1 This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.

#24 Updated by Jean-Baptiste Barth almost 4 years ago

It's nearly impossible to have a clean answer to this question. Something could be built on top of yaml_db gem (https://github.com/ludicast/yaml_db ; it generates a database-independent export in YAML). But the latest posts are right on issue IDs, and I can't imagine such a change in a near future sorry.

#25 Updated by Hans Fangohr almost 4 years ago

Dear Jean-Baptiste,

thank you for taking the time to comment.

Okay, it is technically hard (nearly impossible) to provide this feature, so we'll have to leave without it (unless somebody volunteers to pay [a lot] for the development work).

Suppose several projects grow on one Redmine instance, and one of them needs to be transferred to a new server. Could one clone the whole system, and then on the cloned system delete all projects apart from the desired one? This is not pretty, but at least a workable solution available now.
Would we expect this to work?

Thank you,

Hans

#26 Updated by Jean-Baptiste Barth almost 4 years ago

Hans: indeed ! It can be a solution if your projects/users are strongly separated from each other. It could even be partially scripted.

Anyway, if you experience load problems :
  1. you might be able to solve it outside of Redmine, with a caching reverse-proxy, multiple passenger/mongrel instances (or even a different hosting solution), etc.
  2. you can report any specific problem here, I think we would be happy to help Redmine be better with high traffic loads

#27 Updated by Ryan Cross over 3 years ago

A couple things:

I also saw some code for a tentative way to do this over here: http://www.redmine.org/boards/1/topics/8077

Also, the unique id thing is the main problem definitely. I think there are a couple ways to possibly handle this:

1) start down the path of UUID's for objects. This would also have the benefit of allowing people to choose between a system wide unique issue number or a project wide unique issue number. This could possibly be done with the help of a plugin, but would more likely be best implemented at the core of redmine. See http://en.wikipedia.org/wiki/Universally_unique_identifier for some background. Drupal is a project I know that has some implementations of this http://drupal.org/project/uuid but almost all programming languages have at least one library for using them (ruby has 3-4 i think)

2) use a plugin to manage the mapping of old id's to new id's and then supersede the use of the old id's when linking between version control, issues, wikis, etc.
For example you import issue #253 from site1.com and it becomes #12 on site2.com... when rendering the a link to issue #253 on site2.com, based on per-project settings, it knows to link to issue #12 instead. (The plugin would probably have to create a custom field on the relevant object to store the old id's)
This method could work relatively easily but would probably be a bit hacky. There might need to be a way to resolve/choose which reference you're talking about at times.

#28 Updated by Toshi MARUYAMA over 3 years ago

  • Category set to Importers

#29 Updated by mike G about 3 years ago

+1

#31 Updated by Snehal p almost 3 years ago

Could you please provide me updates about this feature?
I want to transfer redmine data from one machine to another machine.Is there any solution to perform this?

#32 Updated by Jonathan M almost 3 years ago

+1

Just built a new server for production and would like to import a project from my old test server. Over 100 issues

#33 Updated by Max Albrecht almost 3 years ago

+1

(it seems there is no easy way to switch from sqlite3 to mysql, so a data import/export feature could take care of that.)

#34 Updated by Marcelo Soares Souza almost 3 years ago

+1

I need merge several redmine servers (with different databases) in one

#35 Updated by Max Albrecht almost 3 years ago

Marcelo Soares Souza wrote:

I need merge several redmine servers (with different databases) in one

I haven't thought of that, but now that you mention it:
Being able to import/export single projects would mean way better data portability, thus also more peace of mind hosting-wise. You could start of a little redmine on your dev box with sqlite3 knowing that if demand gets out of hand you can switch to a 'managed' hosted (like plan.io).
It is already the case with wordpress(.org/.com) and their implementation works really well.

#36 Updated by D B almost 3 years ago

+1

#37 Updated by eeezyy - almost 3 years ago

+1

#38 Updated by Dieter Egert over 2 years ago

Can't it be done in this way:

All issue IDs will be extended internally by the according project ID. The projectID can be written in links, but is not needed in all cases.
The syntax for issue links (e.g. in wiki) will be extended by someting like #projectID:issueID
and alternatively with previous syntax #issueID referencing to local project (if #issueID occurs) else to other project (if #issueID occurs there). This means that the issueID will be specific only for 1 project in future (not for complete redmine).

So when a project is imported to a different redmine installation all issues will keep their numbers and will stay linked to the actually imported project. Just links to different projects need to be broken, if the according project is not imported in the same time.

Attachments need to be stored in a own folder per project.
Users will be created as new users, or (if name is same in old and new redmine) will be mapped accordingly.
All needed trackers will be transfered as well as workflows, news, wiki and forums.

Also if not all information is transferred it will be better than nothing.
This feature is really needed, if you use redmine for many projects (and therefore on some installations).

#39 Updated by Jason Palmer over 2 years ago

I don't like the idea of non-unique issue ID's.

We have two instances of Redmine currently, and are looking to merge these. Already staff use the issue ID to point people to the relevant issue, and with two instances this can be very confusing. If the issue ID was only relevant when mentioned in the context of the project there would be chaos! Different business units use different projects, with their work going into subprojects of that B.U. Project space. There is enough cross-functional/cross project team work going on that noone would know what issue 123 meant anymore (we are lazy enough to leave off the project identifier).

#40 Updated by Terence Mill over 2 years ago

Holger Kluge wrote:

Hi,

i also thought about this and stumbled about http://www.rubyrep.org. They have a >screencast for bidirectionally synchronizing two redmine instances at >http://www.rubyrep.org/screencast.html.

I did not test this myself, but it may be worth a try...

Why don't they sync the two tables 'schema_migrations' and plugin_shema_info'?

Does that mean i can have different plugins setups in both redmine instances?
If redmine standard tables are never touched by plugin migration, can this work?

#41 Updated by Jonathan East over 2 years ago

+1

#42 Updated by Maurizio Totti over 2 years ago

+1

#43 Updated by John NWW over 2 years ago

+1 It looks like these guys figured out a way. http://www.hostedredmine.com/news/500

#44 Updated by Brad Rushworth about 2 years ago

John NWW wrote:

+1 It looks like these guys figured out a way. http://www.hostedredmine.com/news/500

I manage HostedRedmine.com. I would be willing to release this code under GNU Affero General Public License. It is PHP code, but it is simple enough to use and modify.

This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.

This is the main use case. It allows you to extract out particular users from a Redmine installation and all their associated data (issues, wikis, etc). It must be imported into a fresh installation of Redmine, due to unique issue numbering etc.

#45 Updated by Fredrik Svensson about 2 years ago

Terence Mill wrote:

Holger Kluge wrote:

Hi,

i also thought about this and stumbled about http://www.rubyrep.org. They have a >screencast for bidirectionally synchronizing two redmine instances at >http://www.rubyrep.org/screencast.html.

I did not test this myself, but it may be worth a try...

Why don't they sync the two tables 'schema_migrations' and plugin_shema_info'?

Does that mean i can have different plugins setups in both redmine instances?
If redmine standard tables are never touched by plugin migration, can this work?

have no idea about the tables but the synchronization as i see could have been done with a normal mysql cluster. the fun thing would be to do asynchronous bi-directional sync but for that one need to change stepping of how sequence indexes are generated in redmine.

#46 Updated by Filou Centrinov almost 2 years ago

+1

#47 Updated by Sudhanshu Agrawal over 1 year ago

Brad Rushworth wrote:

John NWW wrote:

+1 It looks like these guys figured out a way. http://www.hostedredmine.com/news/500

I manage HostedRedmine.com. I would be willing to release this code under GNU Affero General Public License. It is PHP code, but it is simple enough to use and modify.

This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.

This is the main use case. It allows you to extract out particular users from a Redmine installation and all their associated data (issues, wikis, etc). It must be imported into a fresh installation of Redmine, due to unique issue numbering etc.

Hey Brad did you get a chance to release it under GNU? I am in a similar need. Please point me to the location if you published it.

#48 Updated by Brad Rushworth over 1 year ago

Sudhanshu Agrawal wrote:

Hey Brad did you get a chance to release it under GNU? I am in a similar need. Please point me to the location if you published it.

Yes, it is now attached here: http://www.hostedredmine.com/news/500

#49 Updated by Thomas Kohler over 1 year ago

+1
Hello,
there you can find an importer for "RequisitePro" baselines: https://github.com/gen2thomas/redmine_import_reqpro

"RequistePro" is using UID's. This ID's will be imported as custom field "RPUID".
Baselines are human readable xml files with special names and format (schemes). Relations between issues inside a project and also to other projects are supported currently.

The "DevelopmentV05"-Branch is working quite good. I have now imported more than 20000 issues with that plugin into my redmine database.

If anyone can write the same for export in such a baseline format 3 guys will be happy ;o}:
1.) The one who want to use his redmine datas in "RequistePro"
2.) That guy like you, who want to merge some redmine databases together
3.) Me, because I want to clean up my database for all the stuff I have added while testing the importer

Wiki and other associated data are not present in "RequisitePro". For that to export an separate file is possible which will be ignored for importing in "RequisitePro".

#50 Updated by Sam McLeod over 1 year ago

+1

#51 Updated by Lubos Racansky about 1 year ago

+1

#52 Updated by Viktor Berke 12 months ago

+1

In the meanwhile, those who need a guide, I've compiled a bunch of instructions about data migration and upgrade:

Installing, Migrating & Upgrading Redmine with LDAP on Ubuntu

I hope this may be of help to some of you. For me it took about a week to figure all this out lol.

#53 Updated by Kamil . 10 months ago

+1

#54 Updated by Kamil . 10 months ago

I've found an importer tool (https://github.com/leovitch/redmine_importer/wiki). Maybe it could be used with Brad's exporter (http://www.redmine.org/issues/3647#note-44) somehow?

#55 Updated by Andre Greeff 9 months ago

@Kamil, looks like that importer tool is about 2 years old.. If you don't have any luck getting it to work with the Redmine 2.x range, you might want to try this one.

#56 Updated by Toshi MARUYAMA 5 months ago

#57 Updated by Lu Pa 4 months ago

+1

#58 Updated by Toshi MARUYAMA 3 months ago

  • Related to Feature #7884: Import issues from Excel or CSV file added

#59 Updated by name zero about 1 month ago

Fredrik Svensson wrote:

have no idea about the tables but the synchronization as i see could have been done with a normal mysql cluster. the fun thing would be to do asynchronous bi-directional sync but for that one need to change stepping of how sequence indexes are generated in redmine.

The database system will take care of that. E.g. MySQL will issue even ID's on member1 and odd on member2. This works for >2 members, too.
See MySQL Multimaster replication and auto_increment_increment.

Also available in: Atom PDF