Initial svn fetch_changesets takes 12 hours...

Added by Piers Wombwell over 12 years ago

I thought I'd try redmine with our repository.

So far it has taken 12 hours to do the import, though I do suspect most of this is because of swapping. I'm using a 2GB VM and the process is 2.4GB, but it is only taking 1.7% CPU.

top - 09:13:12 up 1 day, 21:04,  3 users,  load average: 13.98, 13.53, 12.65
 Tasks: 103 total,   2 running, 100 sleeping,   0 stopped,   1 zombie
 Cpu(s):  0.0%us,  1.5%sy,  0.0%ni, 24.9%id, 73.6%wa,  0.0%hi,  0.0%si,  0.0%st
 Mem:   2056800k total,  2003604k used,    53196k free,     2220k buffers
 Swap:  1048572k total,   679020k used,   369552k free,     9700k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18989 root      20   0 2491m 1.7g  544 R  1.7 88.0  12:10.39 ruby

(I ran fetch_changesets as root -- I don't run the server as root)

This is using Ubuntu Oneiric's packaged up redmine, so version 1.1.3, but I can't see anything obvious in the changelogs for newer versions that might make any difference.

Any ideas? The repo is around 2GB with many files. Can I break it down into batches of revisions somehow?

Ruby version 1.8.7 (x86_64-linux)
RubyGems version 1.7.2
Rack version 1.1
Rails version 2.3.14
Active Record version 2.3.14
Active Resource version 2.3.14
Action Mailer version 2.3.14
Active Support version 2.3.14
Application root /usr/share/redmine
Environment production
Database schema version 20101114115359

Replies (2)

RE: Initial svn fetch_changesets takes 12 hours... - Added by Piers Wombwell over 12 years ago

I had to give up as the same machine hosts our svn repo and svn became painfully slow.

I guess there's no solution? It was all time spent in ruby rather than an 'svn' process, so I'm not sure the suggestion in a previous forum post of using libsvn would help -- though I haven't tried it.

RE: Initial svn fetch_changesets takes 12 hours... - Added by Piers Wombwell over 12 years ago

Not that anyone seem to have cared ;-) but I've figured out how to fix it.

Edit /usr/share/redmine/app/models/repository/subversion.rb
In fetch_changesets, modify:

# loads changesets by batches of 200 identifier_to = [identifier_from + 199, scm_revision].min
to

# loads changesets by batches of 1 identifier_to = [identifier_from + 1, scm_revision].min

This reduced the import time from >12 hours (with me aborting) to around five minutes.

(1-2/2)

Project

General

Profile

Redmine