Defect #7047

Git adapter very slow when a commit modifies a lot of files

Added by Jean-Baptiste Barth about 4 years ago. Updated almost 2 years ago.

Status:NewStart date:2010-12-04
Priority:LowDue date:
Assignee:-% Done:

0%

Category:SCM
Target version:-
Resolution: Affected version:

Description

I have a Git repo with thousands of files modified in some commits. Repository index action is very slow (>20s).

At first sight, it seems it's due to Redmine::Scm::Adapters::GitAdapter#revisions uses --raw option, which is not needed in all cases, at least not needed in Repository::Git#latest_changesets. It results in thousands of useless lines parsed after the shell-out.

I'll have a look at that. I admit it's not a common use case, hence the priority.

false_flag_report_last_commit.png (96 KB) Toshi MARUYAMA, 2011-03-03 09:25

true_flag_report_last_commit.png (145 KB) Toshi MARUYAMA, 2011-03-03 09:25


Related issues

Related to Defect #1435: slow display of 'repository' tab for huge repository (pag... Closed 2008-06-12
Related to Defect #5096: Redmine hangs up while browsing Git repository Closed 2010-03-16
Related to Defect #4773: Redmine+Git+PostgresSQL 8.4 fails with linux kernel tree ... Closed 2010-02-09
Related to Feature #6092: Truncate Git revision labels in Activity page/feed and al... Closed 2010-08-09
Related to Defect #5357: Git: SCM revisions ordered by date/time (should be revers... New 2010-04-20
Related to Defect #6013: git tab,browsing, very slow -- even after first time Closed 2010-08-02
Related to Feature #8365: Git: per project setting to report last commit or not in ... Closed 2011-05-13

Associated revisions

Revision 4797
Added by Toshi MARUYAMA almost 4 years ago

scm: change key name of configurable command name (#7517, #6159, #7047).

Revision 4851
Added by Toshi MARUYAMA almost 4 years ago

scm: git: implement find_changeset_by_name (#7047).

SQL "like" is slow.

Revision 4945
Added by Toshi MARUYAMA almost 4 years ago

scm: git: add flag of disable showing last commit in repository tree (#7047).

Revision 5639
Added by Toshi MARUYAMA over 3 years ago

scm: git: split adapter entries() for browsing tree performance (#7047).

Revision 5640
Added by Toshi MARUYAMA over 3 years ago

scm: git: override adapter entry() for browsing tree performance (#7047).

Revision 5641
Added by Toshi MARUYAMA over 3 years ago

scm: git: call entries_git() in adapter entriese() for browsing tree performance (#7047).

Revision 5642
Added by Toshi MARUYAMA over 3 years ago

scm: git: call entries_git() in adapter entry() for browsing tree performance (#7047).

cat/diff/blame link calls adapter entry().
Abstract adapter entry() calls adapter entries().
Git adapter entries() is very slow.

Revision 5654
Added by Toshi MARUYAMA over 3 years ago

scm: git: use report_last_commit option in unit adapter test (#7047).

Revision 5655
Added by Toshi MARUYAMA over 3 years ago

scm: git: override entries() in model for browsing tree performance (#7047).

Revision 5656
Added by Toshi MARUYAMA over 3 years ago

scm: git: add the model method whether reporting last commit in repository tree (#7047).

Revision 5657
Added by Toshi MARUYAMA over 3 years ago

scm: git: use the model value of whether reporting last commit in repository tree (#7047).

And remove entries_git() and the flag in adapter.

Revision 5661
Added by Toshi MARUYAMA over 3 years ago

scm: git: change model report_last_commit() name to extra_report_last_commit() (#7047).

Revision 5762
Added by Toshi MARUYAMA over 3 years ago

scm: add "extra_info" column to repositories table and set serialize (#7146, #7047).

This column is for specific SCM.
Git 7 days problem (#7146) can not resolve unless storing branches info in database.

Revision 5766
Added by Toshi MARUYAMA over 3 years ago

scm: add a repository model method "merge_extra_info()" to merge "extra_info" serializing hash yaml (#7146, #7047).

Revision 5767
Added by Toshi MARUYAMA over 3 years ago

scm: add a test of a repository model method "merge_extra_info()" in unit test (#7146, #7047).

Revision 5768
Added by Toshi MARUYAMA over 3 years ago

scm: git: get "extra_report_last_commit" value from extra_info (#8365, #7047).

Browsing repository tree performance problem is git only.
So, adding new "report_last_commit" column is very expensive.

Revision 5769
Added by Toshi MARUYAMA over 3 years ago

scm: switch to save repository setting params in "extra_" or not (#8365, #7047).

Revision 5770
Added by Toshi MARUYAMA over 3 years ago

scm: use repository merge_extra_info() to save extra_ parameter (#8365, #7047).

Revision 5771
Added by Toshi MARUYAMA over 3 years ago

scm: git: add check box of whether reporting last commit for files and directories in project setting (#8365, #7047).

Browsing repository tree performance problem is git only.
So, adding new "report_last_commit" column is very expensive.

Revision 5772
Added by Toshi MARUYAMA over 3 years ago

scm: git: use i18n text whether reporting last commit for files and directories in project setting (#8365, #7047).

Revision 5773
Added by Toshi MARUYAMA over 3 years ago

scm: git: add Japanese translation of whether reporting last commit for files and directories in project setting (#8365, #7047).

Revision 5774
Added by Toshi MARUYAMA over 3 years ago

scm: update locales of whether reporting last commit for files and directories in project setting (#8365, #7047).

Revision 5775
Added by Toshi MARUYAMA over 3 years ago

scm: git: show only filename and filesize if setting of reporting last commit is disable (#8365, #7047).

History

#3 Updated by Toshi MARUYAMA almost 4 years ago

$ time wget https://www.chiliproject.org/projects/chiliproject/repository
--2011-02-08 10:33:38--  https://www.chiliproject.org/projects/chiliproject/repository
Resolving www.chiliproject.org... 184.171.175.157
Connecting to www.chiliproject.org|184.171.175.157|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30894 (30K) [text/html]
Saving to: `repository'

100%[================================================================================================>] 30,894      --.-K/s   in 0.1s    

2011-02-08 10:33:41 (213 KB/s) - `repository' saved [30894/30894]

real    0m3.378s
user    0m0.063s
sys     0m0.023s

$ time wget https://www.chiliproject.org/projects/chiliproject/repository/revisions/master/show/db/migrate
--2011-02-08 10:33:54--  https://www.chiliproject.org/projects/chiliproject/repository/revisions/master/show/db/migrate
Resolving www.chiliproject.org... 184.171.175.157
Connecting to www.chiliproject.org|184.171.175.157|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 139043 (136K) [text/html]
Saving to: `migrate'

100%[================================================================================================>] 139,043      216K/s   in 0.6s    

2011-02-08 10:34:20 (216 KB/s) - `migrate' saved [139043/139043]

real    0m26.036s
user    0m0.068s
sys     0m0.024s

#4 Updated by Etienne Massip almost 4 years ago

  • Assignee changed from Jean-Baptiste Barth to Toshi MARUYAMA

#5 Updated by Toshi MARUYAMA almost 4 years ago

The issue cannot be solved to no use of wrap git command such as Yuya's Mercurial extension.

source:trunk/lib/redmine/scm/adapters/mercurial/redminehelper.py

#6 Updated by Toshi MARUYAMA almost 4 years ago

I added flag to switch showing last commit at source:trunk/lib/redmine/scm/adapters/git_adapter.rb@4992#L25.

true

$ time wget http://localhost:3000/projects/redmine-git-00/repository/revisions/master/show/db/migrate
--2011-02-24 21:17:23--  http://localhost:3000/projects/redmine-git-00/repository/revisions/master/show/db/migrate
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 139135 (136K) [text/html]
Saving to: `migrate'

100%[===================================================================================>] 139,135     --.-K/s   in 0.009s  

2011-02-24 21:18:11 (15.5 MB/s) - `migrate' saved [139135/139135]

real    0m47.799s
user    0m0.004s
sys    0m0.013s

false

$ time wget http://localhost:3000/projects/redmine-git-00/repository/revisions/master/show/db/migrate
--2011-02-24 21:20:43--  http://localhost:3000/projects/redmine-git-00/repository/revisions/master/show/db/migrate
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 100345 (98K) [text/html]
Saving to: `migrate'

100%[===================================================================================>] 100,345     --.-K/s   in 0.007s  

2011-02-24 21:20:45 (14.7 MB/s) - `migrate' saved [100345/100345]

real    0m2.076s
user    0m0.005s
sys    0m0.007s

I don't have a plan to refactor git adapter.

#8 Updated by Jean-Philippe Lang almost 2 years ago

  • Status changed from 7 to New

Assigned issue with no assignee back to New status.

Also available in: Atom PDF