No import function = giving up on Redmine

Added by James R over 7 years ago

Hello,

I am not a developer and have been looking at the possibility of using redmine to manage non IT projects.

I need to import about 100 existing projects with associated custom fields and issues. I have looked for importer plugins but none of them seem to function with v2.3. I have experimented with the API but there seems to be bugs when working with custom fields. I have scoured the forum and stack exchange looking for solutions and have posted a few questions looking for advice.
http://www.redmine.org/boards/2/topics/37645, http://www.redmine.org/boards/2/topics/37509, http://www.redmine.org/boards/2/topics/37595.

Now it seems that maybe python might be the best way of importing things: http://stackoverflow.com/questions/16200812/post-multiple-issues-to-redmine-via-api but this is yet another thing to learn.

It has been fun learning about curl, RESTful APIs and python but honestly at this point I just want to import my data and get on with my day job.

Surely the easy import/export of data to redmine must be a major issue for others people too and would be a major feature to prioritize.

Am I missing something?

Replies (30)

RE: No import function = giving up on Redmine - Added by Jean-Baptiste Barth over 7 years ago

You're not missing anything. There are already import tools from Mantis and Trac, two other popular management projects, but nothing official for a CSV file. Is your data somehow "private" ? I'd love to see something in core that would do exactly that, so if you can send me test data I'd be able to tell you if the task is difficult, and if not I could propose a script for that (we'd open an issue at this point, the forum is not the place where we should treat that kind of things). From what you said you should have two CSV files, one for projects, the other one for issues, am I right ? If your data is not too private, send me a version by email at " ".

RE: No import function = giving up on Redmine - Added by James R over 7 years ago

My data is messy but I am looking for a generic solution that would hopefully be useful for others too. I propose initially extracting this data to three csv files. In the examples below I do not include all the custom fields I will probably need in the end.

Project.csv

id,name,identifier,description,customfield1
1,BFA0241,bfa0241,a first test,AAA001
2,KEN0228,ken0228,a second test,ABB002
3,UGA0222,uga0222,a third test,ABC003

issues.csv

project_id,project_name,tracker_id,status_id,subject,description,category_id,start_date,due_date,customfield2
1,BFA0241,4,2,problem1,bigmess,4,2013-05-08,2013-06-12,124552
1,BFA0241,3,1,problem2,small mess,2,2013-05-07,2013-05-08,125667
2,KEN0228,4,3,problem3,mediummess,3,2013-06-08,2013-06-11,130001

users.csv

login,password,firstname,lastname,mail,auth_source_id
bob,pwd1,Bob,Boggins,bob@gmail.com,2
fred,pwd2,Fred,Fredkins,fred@gmail.com,3
sue,pwd3,Sue,Sasson,sue@gmail.com,4

I have already tried converting this to json/xml (using a big excel concatenate function) and importing through the api but it seems to fall down on custom fields, dates, tracker ids etc. Ideally it would be possible to reference projects by name rather than id when posting issues as ids would be generated automatically when I POST new projects meaning I would have to go back and edit my issues.csv to include project ids before importing. I hope that all makes sense?!?

Definitely in terms of user friendliness a script that can deal directly with csv would be the easiest thing for people like me to deal with!

Many thanks for any help you can offer with this!

RE: No import function = giving up on Redmine - Added by Jean-Baptiste Barth over 7 years ago

Preliminary version: https://gist.github.com/jbbarth/5551408

Copy this file in your "lib/tasks/" directory, then run it with:

RAILS_ENV=production rake redmine:import_from_csv users=tmp/users.csv projects=tmp/projects.csv issues=tmp/issues.csv

Some notes:
  • BE SURE TO MAKE A DB BACKUP BEFORE LAUNCHING THE SCRIPT. :)
  • this is not fully functional for now, but I'd like some feedback before continuing (or not). Custom fields import doesn't work especially.
  • those CSV files should not include "id" fields. In the issues.csv file, you should have a "project_identifier" column to reference the project name or identifier to which the issue belongs.
  • user passwords should be 8 chars min. Globally, this script doesn't violate redmine constraints, e.g. the line will display an error if a validation doesn't pass, and the data won't be imported.
  • for now I recommend you do a backup of your database and restore it between each try.

Any feedback is welcome, and please be indulgent it's only a work in progress...

RE: No import function = giving up on Redmine - Added by James R over 7 years ago

thanks for all your work on this. I have not been able to try it out yet but will do so over the next couple of evenings and let you know how things go. Ruby is all a bit of a mystery to me...

RE: No import function = giving up on Redmine - Added by Ryan Hsu over 7 years ago

Hi, Jean-Baptiste Barth,
Thanks for your effort on this. It did help me!
I just post what I have tried and modification to make the import task work.
The command is:

RAILS_ENV=production rake redmine:import_from_csv issues=/tmp/test2.csv

The fields I use:

project_id,author_id,tracker_id,status_id,subject,description,category_id,start_date,due_date,customfield8

customfield is fixed name, "8" is found in database.

attachment:

csv file: test2.csv
database: issue_sql.png
redmine task issue status: import_issue.png

Environment:
Redmine version 2.2.0.stable
Ruby version 1.8.7 (x86_64-linux)
Rails version 3.2.9
Environment production
Database adapter MySQL
Redmine plugins:
redmine_importer 1.0
redmine_tab 0.4.0

issue_sql.png (17.3 KB)

import_issue.png (59.4 KB)

test2.csv Magnifier (183 Bytes)

RE: No import function = giving up on Redmine - Added by James R over 7 years ago

Hi Jean Baptiste!

Sorry for the big delay... family priorities have taken precedence for the last few weeks...

Wow this is excellent! Works like a charm with not problems at all once I figured out how to run a ruby command! Now I need to experiment with the custom fields more...

Thank you so much for this. Definitely should be integrated into a future release as it opens up a whole world of import possibilities!

import issue by project_name - Added by James R over 7 years ago

A useful feature would be the ability to import the issues by project name as often it is difficult to identify the project ID.

I have no idea whether this would possibe?

Here is a modified example of the issues.csv:

project_name,author_id,tracker_id,status_id,subject,assigned_to_id,description,category_id,start_date,due_date
UGA0222,1,2,2,Batty,5,bigmess,4,2013-05-05,2013-06-07
UGA0222,1,3,1,Batty,5,small mess,2,2013-05-05,2013-06-07
UGA0222,1,2,3,Batty,5,mediummess,3,2013-05-05,2013-06-07

At a later stage something similar for stuff like assigned_to_name, author_name would also be great for ease of import from csv files.

grateful for any insights...Or just tell me I am talking the impossible here!

RE: No import function = giving up on Redmine - Added by James R over 7 years ago

Experimenting with custom fields for the project.csv file. As you said this is not working yet

Is there any suggested approach as to how I could even start to get this working?

thanks for any help you can offer here.

RE: No import function = giving up on Redmine - Added by Vincent Olivieri over 7 years ago

This is a great util!

I've installed and tested it in my Redmine eval instance. I'm trying to import projects and issues from an in-house workflow system.

Any progress on custom fields? That would finish this for me and allow us to proceed with the conversion. I'm able to specify the custom fields in my csv and see rows created in custom_values, but with NULL values. I then noticed the "todo" comment in the script.

Thanks so much for your time and effort!

RE: No import function = giving up on Redmine - Added by James R over 7 years ago

Same results for me regarding custom fields. I do not have the skills to develop this further and am trying to find a good work around... maybe using the api?

no luck yet

RE: No import function = giving up on Redmine - Added by Saeed Richardson over 7 years ago

Just thought I'd chime in as an extra contributor. I've used the import for about 70 issues from Pivotal Tracker and it worked well with the exception of not liking the "fixed_version_id" field. I opted to just remove the column and it worked without a hitch.

Thanks for taking the time to create the rake task for this.

RE: No import function = giving up on Redmine - Added by Andrey Anufrienko about 7 years ago

I've written some changes that allow to import issues (only ISSUES!) with custom fields.
Please use attached script or replace 'def run' from your script with the following:

  1. Runs the migration
    def run
    errors = []
    puts
    puts "Starting data import."
    puts
    w(users projects issues).each do |type|
    next unless content[type]
    klass = type.classify.constantize
    print "#{klass}: "
    my_custom_fields = Array.new
    content[type].each do |attributes|
    object = klass.new
    object.tracker = Tracker.first if klass == Issue
    attributes.each_with_index do |value, index|
    field = mappings[type][index]
    if type "issues" &x%x
    field "project_identifier"
    object.project_id = Project.where("name = ? or name = ?", value, value).first
    elsif field.is_a?(String)
    object.send("#{field}=", value)
    else
    #customfield
    v = CustomValue.new :custom_field_id => field.id,
    :value => value
    v.customized_type = "Issue"
    my_custom_fields.push(v)
    end
    end
    if object.valid?
    print "."
    object.save
    #we have to create custom fields after creating issue because we need issue's ID
    my_custom_fields.each do |mcf|
    mcf.customized_id = object.id
    #we have to replace existing default (zero) values with ones from CSV
    cv = CustomValue.where(:customized_type => mcf.customized_type, :custom_field_id => mcf.custom_field_id, :customized_id => mcf.customized_id).first
    if cv.present?
    cv.value = mcf.value
    cv.save
    else
    mcf.save
    end
    end
    else
    print "E"
    errors << "Cannot save following line in #{type}: #{attributes.join(",")}\n => errors: #{object.errors.messages.inspect}\n => object: #{object.inspect}"
    end
    end
    puts
    end
    puts
    if errors.any?
    puts "Errors:"
    errors.each{|e| puts e}
    end
    end

import_from_csv.rake Magnifier - script with some fixes (6.75 KB)

RE: No import function = giving up on Redmine - Added by Juan Fernandez about 7 years ago

My attempt, a Comand line interface Configurable (through yml config file) PHP csv importer, with custom fields.
https://github.com/juanmf/RedmineIssueImporter

I imported 868 Issues from hand made Excel Sheets into redmine Issues, in <10 minutes (once all wrong value mappings wehe memorized).

RE: No import function = giving up on Redmine - Added by Petr Czech over 6 years ago

Is there a easier solution ?

I want to import my exel data, wich looks like(File: "Example issue").

I dont understand that one form Juan...

It would be perfect, then i only click on the one "Import" button in Redmine, chose my file and it would do everything.

Hope someone has a solution and now what i mean

see ja

Example Issue.xlsx - The Example (9.05 KB)

RE: No import function = giving up on Redmine - Added by Jan Niggemann (redmine.org team member) over 6 years ago

Hi Petr,

no, that's not possible I'm afraid.

RE: No import function = giving up on Redmine - Added by Ivan Cenov over 6 years ago

An idea for importer:
  1. Read csv file line by line
  2. use Redmine REST API to import issues.

RE: No import function = giving up on Redmine - Added by Juan Fernandez over 6 years ago

Hi Petr,

I took 10 minutes to make an example config file for you, that follows your xslx data format (except for tue estimated time, you should add it if you plan to use the importer).
check it out here:

https://gist.github.com/juanmf/8f75706af3ec7ed74918

Regards,
Juan

Petr Czech wrote:

Is there a easier solution ?

I want to import my exel data, wich looks like(File: "Example issue").

I dont understand that one form Juan...

It would be perfect, then i only click on the one "Import" button in Redmine, chose my file and it would do everything.

Hope someone has a solution and now what i mean

see ja

RE: No import function = giving up on Redmine - Added by @ go2null over 5 years ago

I have a fork with some fixes - runs on Redmine v2.6.

RE: No import function = giving up on Redmine - Added by Olivier Houdas over 5 years ago

@go2null: I'm not sure how to suggest fixes for your fork in github, so I forked mine (gc-olivierho, but I don't intend to maintain it...). Please check my changes: 2 fixes (including one on your last fix 2 days ago, which missed an "s") + 1 change of behavior to be able to both import new lines and update existing ones.

And I want to thank all those who have contributed to get this feature into Redmine for this great work! This plugin is just going to make our sprints twice faster if not more, with task definition & estimation to be made live in a spreadsheet instead of modifying Redmine issues in the interface (we have a "Task" type) over and over again !

RE: No import function = giving up on Redmine - Added by Korneliusz Moc over 5 years ago

Hey guys,
Since some time I'm struggling with issue importer. I've already tried various plugins and nothing help.
Now I've tried go2null fork, but again no luck (site is returning "internal error"). I've checked production.log the check out the reason and I found "_NoMethodErrror (undefinied method `find_or_create_by_user_id' for #<Class:0x007fbe77d481230>):_"
I assume this is problem with my redmine, not the plugins.
I'm running:

Environment:
  Redmine version                3.0.0.stable
  Ruby version                   2.0.0-p598 (2014-11-13) [x86_64-linux]
  Rails version                  4.2.0
  Environment                    production
  Database adapter               Mysql2
SCM:
  Subversion                     1.8.10
  Git                            1.9.5

Do you have anything that run on Redmine v3.0. I will be greatefull for any advice.

RE: No import function = giving up on Redmine - Added by Juan Fernandez over 5 years ago

have you tried my cli tool? I posted it above.
many have succeded with it.

RE: No import function = giving up on Redmine - Added by Korneliusz Moc over 5 years ago

Yeah, I gave it a try. Enabled API access key on Redmine, and set up everything including config. Seems that everything should be working, parser is properly reading input file, but for unknown reasons I'm getting an error. For test purposes I'm trying to load input file only with one record, so process is returning one error. As I've checked in errors.dump, there is info

type => No response from server, not even an error code, check config data, project name, etc.

This is wierd as server is responding as it should. I've even checked the connection via cURL with the same data I use in config.yml
And it worked. I've received xml as expected. Don't know where else to look for.

RE: No import function = giving up on Redmine - Added by Juan Fernandez over 5 years ago

@Korneliusz Moc

could you update and retry? I added the stringified response to the exception message. So you can see what was actually returned.

Custom Column Import csv into Redmine - Added by santhosh babu over 5 years ago

Dear All,

Is it Possible to Import Customize Column into Redmine through Import csv.

1 2 (1-25/30)