Feature #3647

Data Import/Export System

Added by Mr Embedded almost 3 years ago. Updated about 1 month ago.

Status:New Start date:2009-07-21
Priority:Normal Due date:
Assignee:- % Done:

0%

Category:Importers
Target version:-
Resolution:

Description

Hi All,

I think it will be very useful to have some kind of data import/export system for Redmine with the following features:

1. Entire projects could exported or imported to different instances of Redmine. I know this is dependent on user information being included and restored with the project in some capacity.

2. Selectively export/import administrative settings including user information. This one probably needs to be smart enough to allow a controlled data merge to take place during an import.

3. This could serve also as the basis of a backup system for various areas of Redmine

I got the idea as I was deleting a test project. If you can delete a project in its entirety, you should be able to archive all of that information before deletion in one simple step as the delete vs copy logic is pretty much the same. And if you can do that and then delete your project, you should be able to take that exact same archive and restore it.

I haven't investigated this at the code level but wanted to post this because it seems a good idea at this moment and a search of data export did not provide any interesting hits


Related issues

duplicated by Feature #9993: Export and import all information (Database) Closed

History

#1 Updated by Kirill Ponomarev almost 3 years ago

+1
Good idea.
Also it can be a simple way to make backup.

#2 Updated by Hans Fangohr almost 3 years ago

I agree this would be great, and just what we need for a number of cases.

This relates to http://www.redmine.org/boards/2/topics/5754#message-5761, where just this functionality is required.

#3 Updated by Mr Embedded almost 3 years ago

I would love to help out if I can. I am not a coder. My skill set is more geared to project management and testing/debugging. Let me know if I can assist.

#4 Updated by Nanda Palaniswamy over 2 years ago

+1

Recently I received a request to Import only a specific project from our test server to PROD. (MySQL Server)

I did it manually exported records from table by table & imported them in PROD. Took few hours for about 100 Issues.

It will be nice to have this feature to export a single or multiple project data from one server & import to another RedMine server.

#5 Updated by Andrea Denzler over 2 years ago

+1

Yes, it would be very nice to export/import projects and administration/user settings. Good for backup, moving between servers :)

#6 Updated by Piotr Bazan about 2 years ago

+1

#7 Updated by Balazs Nadasdi about 2 years ago

+1

We have 3 redmine server... And now i try to merge it into one server...
Server 1: 14 project [100+ issues]
Server 2: 3 project [100+ issues]
Server 3: 8 project [100+ issues]

Pls make a migration tool...

/* mysql_dump is not a good way: 'issueId = (1..100+) is in all server */

thx

#8 Updated by Peter Schuster about 2 years ago

+1

#9 Updated by Eraldo Girardi about 2 years ago

+1

#10 Updated by minkbear minkbear about 2 years ago

+1

#11 Updated by Marcus Bitzl about 2 years ago

+1

#12 Updated by Pieter Smith almost 2 years ago

+1

#14 Updated by Daniel Nautré almost 2 years ago

+1

#17 Updated by Andreas Schuh almost 2 years ago

+1

#18 Updated by saturntechweb saturntechweb almost 2 years ago

Raito Yitsushi wrote:

+1

We have 3 redmine server... And now i try to merge it into one server...
Server 1: 14 project [100+ issues]
Server 2: 3 project [100+ issues]
Server 3: 8 project [100+ issues]

Pls make a migration tool...

/* mysql_dump is not a good way: 'issueId = (1..100+) is in all server */

thx

dfdf

#19 Updated by Colan Schwartz almost 2 years ago

+1

#20 Updated by Roland Discein over 1 year ago

+1

#21 Updated by Thomas Zlika over 1 year ago

+1
The real problem relates to the issues IDs (and in particular the fact that these IDs can be used in SVN commit messages to reference the issue). If Redmine had project-specific issue IDs (for example: shortName-1234), it would be simple to import a project.
With global server IDs, I fear that the correct import of a project will not be possible (because that will change the IDs, so at least the references on SVN commits will be lost or wrong).
However, it's a real problem !

#22 Updated by Andrea Denzler over 1 year ago

Thomas Zlika wrote:

+1
The real problem relates to the issues IDs (and in particular the fact that these IDs can be used in SVN commit messages to reference the issue). If Redmine had project-specific issue IDs (for example: shortName-1234), it would be simple to import a project.
With global server IDs, I fear that the correct import of a project will not be possible (because that will change the IDs, so at least the references on SVN commits will be lost or wrong).
However, it's a real problem !

mmm....

1. If you have two different SVN repositories you must first of all merge them, maintaining all the history and creating a report with OldCommitReference->NewCommitReference. This is not a easy task, in any case it is related only to SVN. With such a report you have to update the RedMine servers renaming oldReference with newReference wherever it was used (wiki, notes, etc).

2. Once you merge two RedMine Servers you need a report with OldIssueID->NewIssueId. With that report you have to update all the Commit Comments in your SVN repository, replacing old issue id with new issue id as described in the Subversion FAQ (the Import process could do that automatically). You need to replace the issue IDs not only in SVN but also in all the notes, wiki of RedMine.

Sure, with a global issue ID (like with GIT commits) it would be easier to merge two servers, but the simple incremental Issue ID number is more human friendly....

#23 Updated by Anthony Cartmell over 1 year ago

+1 This would be useful, as I often projects that have started off in my own Redmine later require more access and need a Redmine of their own.

#24 Updated by Jean-Baptiste Barth over 1 year ago

It's nearly impossible to have a clean answer to this question. Something could be built on top of yaml_db gem (https://github.com/ludicast/yaml_db ; it generates a database-independent export in YAML). But the latest posts are right on issue IDs, and I can't imagine such a change in a near future sorry.

#25 Updated by Hans Fangohr over 1 year ago

Dear Jean-Baptiste,

thank you for taking the time to comment.

Okay, it is technically hard (nearly impossible) to provide this feature, so we'll have to leave without it (unless somebody volunteers to pay [a lot] for the development work).

Suppose several projects grow on one Redmine instance, and one of them needs to be transferred to a new server. Could one clone the whole system, and then on the cloned system delete all projects apart from the desired one? This is not pretty, but at least a workable solution available now.
Would we expect this to work?

Thank you,

Hans

#26 Updated by Jean-Baptiste Barth over 1 year ago

Hans: indeed ! It can be a solution if your projects/users are strongly separated from each other. It could even be partially scripted.

Anyway, if you experience load problems :
  1. you might be able to solve it outside of Redmine, with a caching reverse-proxy, multiple passenger/mongrel instances (or even a different hosting solution), etc.
  2. you can report any specific problem here, I think we would be happy to help Redmine be better with high traffic loads

#27 Updated by Ryan Cross about 1 year ago

A couple things:

I also saw some code for a tentative way to do this over here: http://www.redmine.org/boards/1/topics/8077

Also, the unique id thing is the main problem definitely. I think there are a couple ways to possibly handle this:

1) start down the path of UUID's for objects. This would also have the benefit of allowing people to choose between a system wide unique issue number or a project wide unique issue number. This could possibly be done with the help of a plugin, but would more likely be best implemented at the core of redmine. See http://en.wikipedia.org/wiki/Universally_unique_identifier for some background. Drupal is a project I know that has some implementations of this http://drupal.org/project/uuid but almost all programming languages have at least one library for using them (ruby has 3-4 i think)

2) use a plugin to manage the mapping of old id's to new id's and then supersede the use of the old id's when linking between version control, issues, wikis, etc.
For example you import issue #253 from site1.com and it becomes #12 on site2.com... when rendering the a link to issue #253 on site2.com, based on per-project settings, it knows to link to issue #12 instead. (The plugin would probably have to create a custom field on the relevant object to store the old id's)
This method could work relatively easily but would probably be a bit hacky. There might need to be a way to resolve/choose which reference you're talking about at times.

#28 Updated by Toshi MARUYAMA about 1 year ago

  • Category set to Importers

#29 Updated by mike G 8 months ago

+1

#30 Updated by Alcides Silveira Costa 7 months ago

+1

#31 Updated by Snehal G 6 months ago

Could you please provide me updates about this feature?
I want to transfer redmine data from one machine to another machine.Is there any solution to perform this?

#32 Updated by Jonathan M 5 months ago

+1

Just built a new server for production and would like to import a project from my old test server. Over 100 issues

#33 Updated by Max Albrecht 4 months ago

+1

(it seems there is no easy way to switch from sqlite3 to mysql, so a data import/export feature could take care of that.)

#34 Updated by Marcelo Soares Souza 4 months ago

+1

I need merge several redmine servers (with different databases) in one

#35 Updated by Max Albrecht 4 months ago

Marcelo Soares Souza wrote:

I need merge several redmine servers (with different databases) in one

I haven't thought of that, but now that you mention it:
Being able to import/export single projects would mean way better data portability, thus also more peace of mind hosting-wise. You could start of a little redmine on your dev box with sqlite3 knowing that if demand gets out of hand you can switch to a 'managed' hosted (like plan.io).
It is already the case with wordpress(.org/.com) and their implementation works really well.

#36 Updated by D B 4 months ago

+1

#37 Updated by eeezyy - 3 months ago

+1

#38 Updated by Dieter Egert 2 months ago

Can't it be done in this way:

All issue IDs will be extended internally by the according project ID. The projectID can be written in links, but is not needed in all cases.
The syntax for issue links (e.g. in wiki) will be extended by someting like #projectID:issueID
and alternatively with previous syntax #issueID referencing to local project (if #issueID occurs) else to other project (if #issueID occurs there). This means that the issueID will be specific only for 1 project in future (not for complete redmine).

So when a project is imported to a different redmine installation all issues will keep their numbers and will stay linked to the actually imported project. Just links to different projects need to be broken, if the according project is not imported in the same time.

Attachments need to be stored in a own folder per project.
Users will be created as new users, or (if name is same in old and new redmine) will be mapped accordingly.
All needed trackers will be transfered as well as workflows, news, wiki and forums.

Also if not all information is transferred it will be better than nothing.
This feature is really needed, if you use redmine for many projects (and therefore on some installations).

#39 Updated by Jason Palmer about 1 month ago

I don't like the idea of non-unique issue ID's.

We have two instances of Redmine currently, and are looking to merge these. Already staff use the issue ID to point people to the relevant issue, and with two instances this can be very confusing. If the issue ID was only relevant when mentioned in the context of the project there would be chaos! Different business units use different projects, with their work going into subprojects of that B.U. Project space. There is enough cross-functional/cross project team work going on that noone would know what issue 123 meant anymore (we are lazy enough to leave off the project identifier).

#40 Updated by Terence Mill about 1 month ago

Holger Kluge wrote:

Hi,

i also thought about this and stumbled about http://www.rubyrep.org. They have a >screencast for bidirectionally synchronizing two redmine instances at >http://www.rubyrep.org/screencast.html.

I did not test this myself, but it may be worth a try...

Why don't they sync the two tables 'schema_migrations' and plugin_shema_info'?

Does that mean i can have different plugins setups in both redmine instances?
If redmine standard tables are never touched by plugin migration, can this work?

Also available in: Atom PDF