Slow when downloading big SVN files

Added by Andy Sun almost 3 years ago

I use redmine 1.2.1 to manage my project with subversion control.
When downloading files from "Repository", bigger the file is, longer the time it takes to react(After click downloading, it took 142.9s to start actual download for a 40M file).

Here is the log:
Processing RepositoriesController#entry to raw (for 127.0.0.1 at 2011-11-13 00:4
5:43) [GET]
Parameters: {"format"=>"raw", "action"=>"entry", "id"=>"test", "path"=>["calib
re-0.8.24.msi"], "controller"=>"repositories"}
Sending data calibre-0.8.24.msi
Completed in 142917ms (View: 0, DB: 3) | 200 OK [http://localhost/projects/test/
repository/raw/calibre-0.8.24.msi]

database version: Mysql 5.5.17
Ruby version: 1.8.7 -p330
Rails version: 2.3.14
Redmine version: 1.2.1 stable
scm binary version: subversion 1.6.17

Does anyone have the same problem? Anyone know why?

Replies (24)

RE: Slow when downloading big SVN files - Added by Felix Schäfer almost 3 years ago

Redmine first loads the file into memory before beginning to send the file to the client, that probably explains what you are seeing.

RE: Slow when downloading big SVN files - Added by Ivan Cenov almost 3 years ago

Wow, is it possible some kind of streaming?

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

So is there anything I can do to speed up the process? Two minutes waiting is killing group members patient

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

I tried downloading a 40M file from File area and it respond really fast. So why is this just happened with svn?

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

I tried to turn off stream in "actionpack-2.3.14\lib\action_controller\streaming.rb" or increase the cache size. But it took the same 145946ms to respond. Please help me~

RE: Slow when downloading big SVN files - Added by Felix Schäfer almost 3 years ago

Ivan Cenov wrote:

Wow, is it possible some kind of streaming?

Not with the current implementation, I'm afraid.

RE: Slow when downloading big SVN files - Added by Felix Schäfer almost 3 years ago

Andy Sun wrote:

I tried to turn off stream in "actionpack-2.3.14libaction_controllerstreaming.rb" or increase the cache size. But it took the same 145946ms to respond. Please help me~

This is not a problem with ActionController, it does only the sending part, the reading part isn't streamed but waits for the whole file to be in memory before releasing it for sending.

RE: Slow when downloading big SVN files - Added by Felix Schäfer almost 3 years ago

Andy Sun wrote:

So is there anything I can do to speed up the process? Two minutes waiting is killing group members patient

Download the files from svn directly or from the web/dav-frontend provided by svn for apache.

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

Felix Schäfer wrote:

Andy Sun wrote:

I tried to turn off stream in "actionpack-2.3.14libaction_controllerstreaming.rb" or increase the cache size. But it took the same 145946ms to respond. Please help me~

This is not a problem with ActionController, it does only the sending part, the reading part isn't streamed but waits for the whole file to be in memory before releasing it for sending.

I tried using Filesystem in redmine 1.2.2, and it respond in no time after I clicked "download" button( Pop-up a menu asking if you want to download). Can the svn part be modified just like this?

RE: Slow when downloading big SVN files - Added by Felix Schäfer almost 3 years ago

Andy Sun wrote:

Can the svn part be modified just like this?

Probably, but it won't be easy as the response from the SCM shell-outs will have to be streamed to the client.

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

Felix Schäfer wrote:

Andy Sun wrote:

Can the svn part be modified just like this?

Probably, but it won't be easy as the response from the SCM shell-outs will have to be streamed to the client.

I really like the Redmine way to handle SVN. If this slowness can't be fixed, probably I have to find another SVN web server to handle repositories viewings. But the apache itself is so simple...

RE: Slow when downloading big SVN files - Added by Ivan Cenov almost 3 years ago

Andy Sun wrote:

Felix Schäfer wrote:

Andy Sun wrote:

Can the svn part be modified just like this?

Probably, but it won't be easy as the response from the SCM shell-outs will have to be streamed to the client.

I really like the Redmine way to handle SVN. If this slowness can't be fixed, probably I have to find another SVN web server to handle repositories viewings. But the apache itself is so simple...

What is your system? If it is Windows you may use TortoiseSVN for interacting with SVN repositories. Try to download the same file by the command-line client svn (svn.exe) and compare the speeds. This could give some hint about the issue.

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

My OS is win7 x64. I tried using apache weblink to download the same file and it respond in no time. Sorry I'm not familiar with svn command line. But is the apache using the same command as redmine do? What's the command for svn to download a given file?

RE: Slow when downloading big SVN files - Added by Ivan Cenov almost 3 years ago

The command for downloading a file -- without making a work copy:

svn export path_to_file [local file name] --username ARG -- password ARG

I don't know if Redmine uses this command to do the download operation.

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

Ivan Cenov wrote:

The command for downloading a file -- without making a work copy:
[...]
I don't know if Redmine uses this command to do the download operation.

I used the command you mentioned. My test doc-0.8.24.doc file is 42.4MB.If I just use "svn export file:///E:/test/doc-0.8.24.doc" it finished in a blink. If I use "svn export file:///E:/test/doc-0.8.24.doc E:/2" it showed in 2 second with an error saying

svn: can't move “E:svn-F90B340C” to “E:\2”: access denied.

And I got a 42.4MB file "svn-F90B340C" in my E:/ .
I mean, it seems the command line works fine (quickly). How dose redmine handle the same quest?

RE: Slow when downloading big SVN files - Added by Ivan Cenov almost 3 years ago

Andy Sun wrote:

Ivan Cenov wrote:

The command for downloading a file -- without making a work copy:
[...]
I don't know if Redmine uses this command to do the download operation.

I used the command you mentioned. My test doc-0.8.24.doc file is 42.4MB.If I just use "svn export file:///E:/test/doc-0.8.24.doc" it finished in a blink. If I use "svn export file:///E:/test/doc-0.8.24.doc E:/2" it showed in 2 second with an error saying

svn: can't move “E:svn-F90B340C” to “E:”: access denied.

And I got a 42.4MB file "svn-F90B340C" in my E:/ .
I mean, it seems the command line works fine (quickly). How dose redmine handle the same quest?

I'm sorry, I don't know how Redmine deals with downloading files from SVN. May be the implementation is in redmine\lib\redmine\scm\adapters\subversion_adapter.rb but I am not sure. Someone involved in writing the Redmine core could help.

Ivan

RE: Slow when downloading big SVN files - Added by Ivan Cenov almost 3 years ago

Andy Sun wrote:

I used the command you mentioned. My test doc-0.8.24.doc file is 42.4MB.If I just use "svn export file:///E:/test/doc-0.8.24.doc" it finished in a blink. If I use "svn export file:///E:/test/doc-0.8.24.doc E:/2" it showed in 2 second with an error saying

svn: can't move “E:svn-F90B340C” to “E:”: access denied.

What if you try the following instead:

svn export file:///E:/test/doc-0.8.24.doc E:/2/doc-0.8.24.doc (I assume that '2' is a folder.).

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

Ivan Cenov wrote:

Andy Sun wrote:

Ivan Cenov wrote:

The command for downloading a file -- without making a work copy:
[...]
I don't know if Redmine uses this command to do the download operation.

I used the command you mentioned. My test doc-0.8.24.doc file is 42.4MB.If I just use "svn export file:///E:/test/doc-0.8.24.doc" it finished in a blink. If I use "svn export file:///E:/test/doc-0.8.24.doc E:/2" it showed in 2 second with an error saying

svn: can't move “E:svn-F90B340C” to “E:”: access denied.

And I got a 42.4MB file "svn-F90B340C" in my E:/ .
I mean, it seems the command line works fine (quickly). How dose redmine handle the same quest?

I'm sorry, I don't know how Redmine deals with downloading files from SVN. May be the implementation is in *redminelib

edminescmadapterssubversion_adapter.rb* but I am not sure. Someone involved in writing the Redmine core could help.

Ivan

Thank you~ I'll try to look at the file. By the way, have you encountered the same problem in your redmine with any big files? Or changing the OS to linux will solve the problem?

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

Is Jean-Philippe Lang working on the subversion_adapter.rb? I can't find any way to contact him...

RE: Slow when downloading big SVN files - Added by Arnaud Martel almost 3 years ago

Just for information, I can't reproduce your problem: I have a subversion repository with a 210MB file and it just take 2 seconds for the download to start (from redmine with https protocol)...
My subversion repository is hosted on the redmine server and I'm using an url like 'file:///folder/of/my/repository' in the repository tab (in my project settings). Is it the same for you ???

RE: Slow when downloading big SVN files - Added by Andy S almost 3 years ago

We have the same settings. (eg. redmine and subersion installed on the same server, using file protocol).
What is your OS exactly? My server is win server 2008 R2, subversion 1.6.17, redmine 1.2.1(tried 1.2.2 and it's the same). Maybe it is because of the OS?

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

I always got the spam error so I use another account post this

We have the same settings. (eg. redmine and subersion installed on the same server, using file protocol).

What is your OS exactly? My server is win server 2008 R2, subversion 1.6.17, redmine 1.2.1(tried 1.2.2 and it's the same). Maybe it is because of the OS?

RE: Slow when downloading big SVN files - Added by Arnaud Martel almost 3 years ago

My server has 8 cores, 16 GB of RAM, and I installed Linux CentOS x86_64, with subversion 1.6.16 and redmine 1.2.1

RE: Slow when downloading big SVN files - Added by Andy Sun almost 3 years ago

Arnaud Martel wrote:

My server has 8 cores, 16 GB of RAM, and I installed Linux CentOS x86_64, with subversion 1.6.16 and redmine 1.2.1

I tried downloading 140M files on Linux Ubuntu, and it took 8s to start downloading. My server is not as powerful as yours. But it's still faster than working under Windows.

So I think it is because of Windows. Thank you Arnaud. Maybe before this is solved, I'll turn to Linux server.

(1-24/24)