Feature #4455

Mercurial overhaul

Added by Peter Fern almost 5 years ago. Updated over 3 years ago.

Status:ClosedStart date:2009-12-21
Priority:NormalDue date:
Assignee:Toshi MARUYAMA% Done:

0%

Category:SCM
Target version:-
Resolution:

Description

The current mercurial support in Redmine has a number of issues, that this patch attempts to address:
Features:
  • Tag/branch support (#1981)
Bugs:
  • Browsing a revision does not display file size (requires the Mercurial Size extension) (#3421)
  • Browsing a revision does not display correct identifier for files at that revision
  • Repository tab takes a long time to display with large/complex repositories (#3449)
  • File contents fail
Changes:
  • Revision numbers are far too brittle in mercurial, and commit IDs should be used instead (#3724)

The last is problematic, since people are likely already referring to changesets via the revision number (though if they do any complex work with the repository, those are likely broken already), so we need a way to update all issues/journals for projects with Mercurial repos that have references to revisions with corresponding reverences to commits. This would have to be a custom migration job, where the map is built, and issue/journal references are updated, then the changesets are updated with the new revision and scmid values. I have not written this yet, and would welcome contributions in this area.

Also, I imagine that there are additional unit tests required for SCMs supporting branches/tags, and that the test repo would need updating for this? Again, contributions welcome here.

Where I've asked for contributions, they'd be greatly appreciated however this functionality is too important to let languish, so if no one else will help with it, I'll try and find the time to do the remaining steps myself, though it won't be until some time into the new year.

The only thing outstanding for me is that I'd like to speed up retrieving the correct commit IDs entries at the current revision, since this can be a bit expensive for large/binary-heavy repositories, but I can't think of a more efficient method that retains accuracy, however I think accuracy is more important in this case.

redmine-mercurial.patch Magnifier - Mercurial overhaul (22.3 KB) Peter Fern, 2009-12-21 01:27

redmine-mercurial-01.patch Magnifier (1.02 KB) Toshi MARUYAMA, 2010-01-19 23:29

hg-redmine-00.png (15.6 KB) Toshi MARUYAMA, 2010-01-22 16:56

TortoiseHg-00.png (28.6 KB) Toshi MARUYAMA, 2010-01-22 16:56

hg-win32-extraline.patch Magnifier - Trim spurious empty line from hg on win32 (670 Bytes) Peter Fern, 2010-02-13 11:43

hg-win32-extraline-attempt2.patch Magnifier - Don't use single quotes for shellout on win32 (541 Bytes) Peter Fern, 2010-02-13 11:48

hg-overhaul-0.9.3.patch Magnifier (65.3 KB) Toshi MARUYAMA, 2010-03-07 11:19

overhaul.py Magnifier - a draft of an helper hg extension (832 Bytes) Alessio Franceschelli, 2010-03-18 16:25

revision-no-migration.png - Repo browser screenshot (23 KB) Yuya Nishihara, 2010-03-23 16:31

revision-no-migration-2.png - Repo browser screenshot 2 (30.3 KB) Yuya Nishihara, 2010-03-23 16:31

ya-hg-overhaul-0.9-stable-2010-03-27.patch Magnifier - Yet another overhaul patch for 0.9-stable (52 KB) Yuya Nishihara, 2010-03-27 14:29

redmine-issue4455-2010-05-17.jpg (12.8 KB) Yuya Nishihara, 2010-05-17 15:14

ya-hg-overhaul-0.9-stable-2010-05-17.patch Magnifier - Yet another overhaul patch for 0.9-stable (64.1 KB) Yuya Nishihara, 2010-05-17 15:14

isodatesec.patch Magnifier (1.31 KB) Toshi MARUYAMA, 2010-05-31 18:01

mq-applied-tags.png (18.5 KB) Toshi MARUYAMA, 2010-06-11 13:31

mq-applied-revisions.png (44.3 KB) Toshi MARUYAMA, 2010-06-11 13:31

unit-test-for-isodatesec.diff Magnifier (833 Bytes) Toshi MARUYAMA, 2010-07-23 16:35

unit-test-by-nodeid.diff Magnifier (881 Bytes) Toshi MARUYAMA, 2010-07-27 10:11

functional-by-nodeid.diff Magnifier (929 Bytes) Toshi MARUYAMA, 2010-07-27 10:11

merge_users_stats.diff Magnifier - Parent a52c9dd45ef4b42acd99d423844ac4c43bd712c5 (1.93 KB) Ammler _, 2010-07-28 09:55

hg-changeset-order.patch Magnifier (12.7 KB) Yuya Nishihara, 2010-07-29 19:05

hg-isodatesec.patch Magnifier (2.13 KB) Yuya Nishihara, 2010-07-29 19:05

TortoiseHg-PyQt-en.png (74.1 KB) Toshi MARUYAMA, 2010-07-30 09:02

TortoiseHg-PyQt-ja.png (50.3 KB) Toshi MARUYAMA, 2010-07-30 09:02


Related issues

Related to Defect #3421: Mercurial reads files from working dir instead of changesets Closed 2009-05-26
Related to Defect #3724: Mercurial repositories display revision ID instead of cha... Closed 2009-08-10
Related to Feature #1981: display mercurial tags Closed 2008-10-02
Related to Feature #7246: Handle "named branch" for mercurial Closed 2011-01-07
Related to Feature #2799: Support for Bazaar's shared reposetories (created with in... New 2009-02-21
Related to Feature #5386: Branch/Tags in Changeset Description New 2010-04-27
Related to Feature #3909: Mercurial: show repository graphic history Closed 2009-09-23
Related to Feature #6515: Mercurial hgrc support with issues auto-syncronizaton New 2010-09-28
Related to Patch #7494: Patch for support http protocol wirh mercurial repository Closed 2011-01-30 2011-01-30
Related to Feature #6892: Automatic repository creation. New 2010-11-15

Associated revisions

Revision 4824
Added by Toshi MARUYAMA over 3 years ago

scm: catch CommandFailed during bulk Repository.fetch_changesets (#4455).

Contributed by Yuya Nishihara.

Revision 4830
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add new method 'hg' to wrap shellout (#4455).

Contributed by Yuya Nishihara.

Revision 4833
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add Mercurial helper extension (#4455).

  • 'rhsummary' and 'rhmanifest' for reducing the number of hg command calls.
  • 'rhdiff' for compatibility with Mercurial < 1.1.

Also renamed TEMPLATES_DIR to HELPERS_DIR because the directory now contains
templates and a helper extension.

Original version was written by Alessio Franceschelli,
downloaded from http://www.redmine.org/attachments/3395/overhaul.py

Contributed by Alessio Franceschelli and Yuya Nishihara.

Revision 4834
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: ignore redminehelper.pyc and redminehelper.pyo for Git and Mercurial (#4455).

Revision 4837
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: ignore redminehelper.pyc and redminehelper.pyo for Subversion (#4455).

Revision 4839
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: change "--cwd" option to "-R" for running "hg" (#4455).

Revision 4840
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add "hgtarget" method (#4455).

Abstract adapter shell quotes paths.

Revision 4841
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: refactor "annotate" by using hg helper method (#4455).

Contributed by Yuya Nishihara.

Revision 4842
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: refactor "cat" by using hg helper method (#4455).

Contributed by Yuya Nishihara.

Revision 4844
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: switch root_url or url in "hg" method (#4455).

"info" sets root_url from url.

Revision 4845
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add "summary" method in adapter (#4455).

Contributed by Yuya Nishihara.

Revision 4846
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: rewrite MercurialAdapter#info by using helper extention (#4455).

Contributed by Yuya Nishihara.

Revision 4848
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: rewrite MercurialAdapter#revisions as an iterator (#4455).

Now it uses XmlMini.parse() in place of slow REXML.

Contributed by Yuya Nishihara.

Revision 4849
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: refactor Repository::Mercurial#fetch_changesets (#4455).

Contributed by Yuya Nishihara.

Revision 4854
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: rewrite MercurialAdapter#diff by using helper extension (#4455, #7518).

fix incompatibility of diff with Mercurial < 1.1.

Contributed by Yuya Nishihara.

Revision 4856
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: rewrite MercurialAdapter#entries to show per-file change log and size (#3421, #4455).

Contributed by Yuya Nishihara.

Revision 4868
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: improvement latest_changesets without supporting tags and named branches (#4455).

Based on latest-changesets-improvements-2.diff of #6159.
http://www.redmine.org/attachments/4332/latest-changesets-improvements-2.diff

Contributed by Yuya Nishihara.

Revision 4871
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: temporary disable tags and branches at model (#1981, #7246, #4455).

Until latest_changesets supports tags and branches.

Revision 4876
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: fix exception operation in adapter "summary" method (#4455).

Revision 4878
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: simplify fixing exception operation in adapter "summary" method (#4455).

Revision 4879
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: fix exception operation in adapter all methods using xml and io.read (#4455).

Revision 4934
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: replace urllib.unquote to urllib.unquote_plus in helper (#4455).

Revision 4989
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: wrap revison, tag and branch with URL encoding for entries (#4455, #1981, #7246).

Revision 4994
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: remove localtime from adapter (#4455).

Revision 4997
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: update test repository for path contains space (#2664, #4455).

Revision 4998
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add unit adapter test repository for path contains space (#2664, #4455).

Revision 5101
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add new "rhlog()" function in helper extension (#7246, #4455).

For latest changesets supporting named branch.

Revision 5102
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add new "nodes_in_branch()" method in adapter (#7246, #4455).

For latest changesets supporting named branch.

Revision 5103
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: add tests for "nodes_in_branch()" method in adapter (#7246, #4455).

Revision 5108
Added by Toshi MARUYAMA over 3 years ago

scm: mercurial: remove unused parameters from "nodes_in_branch()" method in adapter (#7246, #4455).

History

#1 Updated by Peter Fern almost 5 years ago

  • File contents fail

should be:

  • File contents fail to be displayed when the work dir has changed from the last revision (#3421)

#2 Updated by Luke Hoersten almost 5 years ago

Looks good. I'll definitely be giving this a try.

#3 Updated by Toshi MARUYAMA almost 5 years ago

I tested this patch.
Branch has no problem,
but tag lost '.' and '-'.

For example, "tag-111.222" becomes "tag".

#4 Updated by Peter Fern almost 5 years ago

Toshi Maruyama wrote:

I tested this patch.
Branch has no problem,
but tag lost '.' and '-'.

For example, "tag-111.222" becomes "tag".

Should be an easy fix, do you know the valid characters for tags off-hand, or should I look them up?

#5 Updated by Peter Fern almost 5 years ago

Peter Fern wrote:

Should be an easy fix, do you know the valid characters for tags off-hand, or should I look them up?

Actually, never mind, I'll just admit anything that's not whitespace. Patch on it's way tomorrow.

#6 Updated by Toshi MARUYAMA almost 5 years ago

Peter Fern wrote:

Toshi Maruyama wrote:

I tested this patch.
Branch has no problem,
but tag lost '.' and '-'.

For example, "tag-111.222" becomes "tag".

Should be an easy fix, do you know the valid characters for tags off-hand, or should I look them up?

http://mercurial.selenic.com/wiki/Tag

It can contain any characters except ":" (colon),
"\r" (Carriage Return) or "\n" (Line Feed).

And I test.

$ hg branches
branch_ aaaa bbb                  31:61f3b4c8e7d0

$ hg tags
tip                               31:61f3b4c8e7d0
aaaa aaaa                         29:44823ece1572

#7 Updated by Toshi MARUYAMA almost 5 years ago

I created a patch.

#8 Updated by Toshi MARUYAMA almost 5 years ago

Task redmine:fetch_changesets fails.

rake aborted!
undefined method `scmid' for nil:NilClass
r:/Ruby/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/whiny_nil.rb:52:in `method_missing'
c:/redmine/app/models/repository/mercurial.rb:48:in `fetch_changesets'
r:/Ruby/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/core_ext/symbol.rb:11:in `__send__'
r:/Ruby/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/core_ext/symbol.rb:11:in `to_proc'
c:/redmine/app/models/repository.rb:166:in `each'
c:/redmine/app/models/repository.rb:166:in `fetch_changesets'
c:/redmine/lib/tasks/fetch_changesets.rake:22
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
r:/Ruby/lib/ruby/1.8/monitor.rb:242:in `synchronize'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
r:/Ruby/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31
r:/Ruby/bin/rake:19:in `load'
r:/Ruby/bin/rake:19
(in c:/redmine)
** Invoke redmine:fetch_changesets (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute redmine:fetch_changesets

#9 Updated by Peter Fern almost 5 years ago

Sorry, been rather busy this past week. This error does not occur for me, however I'll resync with trunk and try again in the next few days.

#10 Updated by Toshi MARUYAMA almost 5 years ago

In log/test.log.


Shelling out: hg --debug --encoding utf8
 -R "c:\test\test00/" --cwd "c:\test\test00/" log
 --style "c:/redmine/lib/redmine/scm/adapters/mercurial
 /hg-template-1.0-lite.tmpl" -r "f5ead3bc1788":"0" --limit 1
uninitialized constant REXML::Document

So, I added require 'rexml/document' in lib/redmine/scm/adapters/mercurial_adapter.rb.
Then, Task redmine:fetch_changesets finished successfully.

#11 Updated by Toshi MARUYAMA almost 5 years ago

This patch uses wc command. So there is a same probrem with #3836.

#12 Updated by Peter Fern almost 5 years ago

Toshi Maruyama wrote:

This patch uses wc command. So there is a same probrem with #3836.

Toshi, thanks for all your testing. I based parts of this on the existing git support, hence the duplicate bug. Using 'wc' was a poor choice, however the same fix will work here. I'll update to trunk and incorporate these changes over the weekend, and check for any other *nix specific shell execution.

#13 Updated by Toshi MARUYAMA almost 5 years ago

Because I am Japanese, I have made a big typo "probrem".
"problem" is correct.

MSysGit has wc.exe in "C:\Program Files\Git\bin".
But TortoiseHg does not have wc.exe in "C:\Program Files\TortoiseHg".

#14 Updated by Toshi MARUYAMA almost 5 years ago

This patch is great. But, new problem occurs.
As described at #3567, Mercurial allows to commit a changeset for earlier date.
But redmine sorts revisions by "committed_on".

I attach redmine and TortoiseHg image.
And "hg glog" outputs.

$ hg glog
o  changeset:   3:654c854c6911
|  tag:         tip
|  parent:      1:024578a359f8
|  user:        test00@example.com
|  date:        Mon Jan 01 00:00:00 2007 +0900
|  summary:     2007-01-01
|
| @  changeset:   2:43687839e1e7
| |  parent:      0:5f33311e7327
| |  user:        test01@example.com
| |  date:        Tue Jan 01 00:00:00 2008 +0900
| |  summary:     2008-01-01
| |
o |  changeset:   1:024578a359f8
|/   user:        test00@example.com
|    date:        Thu Jan 01 00:00:00 2009 +0900
|    summary:     2009-01-01
|
o  changeset:   0:5f33311e7327
   user:        test00@example.com
   date:        Sat Jan 23 00:16:11 2010 +0900
   summary:     Initial revision.

#15 Updated by Toshi MARUYAMA over 4 years ago

I think migration is impossible because "revision number" and "commit id" are written directly in journals table and wiki_contents table.
Should this feature separate from existing Mercurial adapter and add as new adapter?

I think #3169 "Multiple repositories for projects" is related.

#17 Updated by Peter Fern over 4 years ago

Toshi Maruyama wrote:

This patch is great. But, new problem occurs.
As described at #3567, Mercurial allows to commit a changeset for earlier date.
But redmine sorts revisions by "committed_on".

This is an existing bug, and honestly, I'm not sure what to do about it. It is particularly difficult to deal with since with this patch, we do not store the incremental revision numbers in the database at all. I'm open to suggestions on how to resolve this, but I think for now I will not deal with it until everything else is signed off.

Toshi Maruyama wrote:

I think migration is impossible because "revision number" and "commit id" are written directly in journals table and wiki_contents table.
Should this feature separate from existing Mercurial adapter and add as new adapter?

I don't like the idea of creating a separate adapter, however I think that the journals and wiki_contents references could be updated with a db migration? That was my original plan. I will see if I can get Eric or one of the other devs to make some comments on this once the other issues are resolved.

Toshi Maruyama wrote:

Test units source:trunk/test/unit/mercurial_adapter_test.rb and source:trunk/test/unit/repository_mercurial_test.rb are incompatible.
What should I do?

I'll work through all the tests soon (hopefully next week), and update them to test all the new functionality, as well as to work with the new id format.

#18 Updated by Eric Davis over 4 years ago

Peter Fern wrote:

I don't like the idea of creating a separate adapter, however I think that the journals and wiki_contents references could be updated with a db migration? That was my original plan. I will see if I can get Eric or one of the other devs to make some comments on this once the other issues are resolved.

Just let me know once you are ready. I'm watching this issue now but I've only used Mercurial once (I use git mostly).

#19 Updated by Toshi MARUYAMA over 4 years ago

Peter Fern wrote:

I'll work through all the tests soon (hopefully next week), and update them to test all the new functionality, as well as to work with the new id format.

Peter, I found your git repository on github.
Can I use this?

#21 Updated by Peter Fern over 4 years ago

Toshi Maruyama wrote:

I update source:trunk/test/unit/repository_mercurial_test.rb.
And I create git repository on github.
http://github.com/marutosi/redmine/commits/marutosi

Toshi, sorry for the lack of updates here - I wrote a large reply some time ago, but it looks like it didn't post correctly. It detailed the changes I had pushed to my Github fork, it's location, and my strategy for moving forward.

I started to look at the tests over the weekend, but the current test repository does not cover all the features that the new code does. My idea was to base a new Mercurial test suite on the Git tests (though they also seem a little inadequate for testing all the Git functionality), but the repository would need recreation from scratch as simply converting the current Git repository will not work due to the differences in branching with Mercurial vs Git. If you want to move forward on updating the Mercurial tests, this is the route I would suggest.

As to correct sorting, the only way I can see of doing it would be to add a field `scmsort` and store the revision numbers there. This should work with some other minor modifications, but as it requires a schema change, I'm a little reluctant to implement it that way... thoughts?

#22 Updated by Toshi MARUYAMA over 4 years ago

I created wiki converter rake task.
http://github.com/marutosi/redmine/commit/ec8d2de04876f2a77aeb2ca88f4407c0e1db8b21

Wikis have revisions on DB, so old revisions are not changed and converted text are saved as new revision.

#23 Updated by Toshi MARUYAMA over 4 years ago

I found the reason of #3449 "Redmine Takes Too Long" everytime.
The reason is "db_revision" and "scm_revision" are not equal everytime at Repository::Mercurial.fetch_changesets.

  • "db_revision" is latest changeset because this is sorted by "committed_on".
  • "scm_revision" is tip and tip is not latest changeset.

There are two case.

  1. Child changeset has earlier date than parent.
  2. Multiple Heads. http://mercurial.selenic.com/wiki/MultipleHeads

This condition occurs frequently.

I tried to resolve this problem referring patch of #3449.


class Repository::Mercurial < Repository
.
.
.
  def latest_changeset
    @latest_changeset ||= changesets.find (
         :first ,
         :order => "LPAD(revision,10,'0000000000') DESC" 
      )
  end

But, SQLite does not support "LPAD".

Adding a field at "changesets" table is only way to resolve this performance problem.
So I agree to add "scmsort" field, but I think "scm_number" or "scm_no" or "hg_revno" is better.

Additionally, git rabase changeset has earlier date than parent.

#24 Updated by Toshi MARUYAMA over 4 years ago

Toshi Maruyama wrote:

I tried to resolve this problem referring patch of #3449.

I mistake issue number. Correct is #3567 - "Sorting for changesets might go wrong on Mercurial repos".

#25 Updated by Toshi MARUYAMA over 4 years ago

I reconsidered above performance problem.
Mercurial revision number is sequential from 0.
So, "changesets.count - 1" is latest Mercurial revision number in DB.

I create a patch and attach at #3449 - Redmine Takes Too Long On Large Mercurial Repository.

#26 Updated by Peter Fern over 4 years ago

Toshi Maruyama wrote:

Adding a field at "changesets" table is only way to resolve this performance problem.
So I agree to add "scmsort" field, but I think "scm_number" or "scm_no" or "hg_revno" is better.

Additionally, git rabase changeset has earlier date than parent.

The reason I suggested scmsort (maybe scm_order) was so that it could be used for other SCMs that require an order other than committed_on - in Mercurial that would be the revision number, but in some other SCM, you might use a unix timestamp or some such...

Toshi Maruyama wrote:

I reconsidered above performance problem.
Mercurial revision number is sequential from 0.
So, "changesets.count - 1" is latest Mercurial revision number in DB.

I create a patch and attach at #3449 - Redmine Takes Too Long On Large Mercurial Repository.

Be careful - with history editing, the current code does not clear missing revisions, so your count will not reflect the actual number of commits, or the current revision number, in the repository.

#27 Updated by Toshi MARUYAMA over 4 years ago

Peter Fern wrote:

Be careful - with history editing, the current code does not clear missing revisions, so your count will not reflect the actual number of commits, or the current revision number, in the repository.

Thanks. I test "hg strip". Changesets ramain on DB.

#28 Updated by Toshi MARUYAMA over 4 years ago

History editing is rare case on shared repository. I think everytime check of histoy editing is high cost. For this case, should Redmine provide another form?

#29 Updated by Peter Fern over 4 years ago

Toshi Maruyama wrote:

History editing is rare case on shared repository. I think everytime check of histoy editing is high cost. For this case, should Redmine provide another form?

This overhaul already takes care of it - if the tip id and number of changesets in the repository and database match, no sync is done, otherwise we match off scmids against the log and prune or add as necessary. It's not that heavy really, and it's necessary for consistency... Git does the same too.

#30 Updated by Toshi MARUYAMA over 4 years ago

Peter Fern wrote:

Toshi Maruyama wrote:

History editing is rare case on shared repository. I think everytime check of histoy editing is high cost. For this case, should Redmine provide another form?

This overhaul already takes care of it - if the tip id and number of changesets in the repository and database match, no sync is done, otherwise we match off scmids against the log and prune or add as necessary. It's not that heavy really, and it's necessary for consistency... Git does the same too.

I test Mercurial crew repository.
http://hg.intevation.org/mercurial/crew

Completed in 177313ms (View: 777, DB: 5245) | 200 OK [http://localhost/projects/hgcrew/repository]
Completed in 175350ms (View: 792, DB: 5132) | 200 OK [http://localhost/projects/hgcrew/repository]
Completed in 175149ms (View: 792, DB: 5136) | 200 OK [http://localhost/projects/hgcrew/repository]
$ du -sh .hg
25M     .hg
$ du -sh .
41M     .
$ find  .hg -type f | wc
   1488    1488   63468
$ find  . -type f | wc
   3038    3038  120048

$ hg log  -l2
changeset:   10425:4cfd0d56be6d
tag:         tip
user:        XXXXXXXXXXXXXXXXXXXX
date:        Thu Feb 11 21:11:59 2010 +0100
summary:     doc: add missing documentation for http_proxy.always

changeset:   10424:677f15da38c1
user:        YYYYYYYYYYYYYYYYYYYY
date:        Thu Feb 11 20:42:20 2010 +0100
summary:     url: proxy handling, simplify and correctly deal with IPv6

#31 Updated by Peter Fern over 4 years ago

Toshi Maruyama wrote:

I test Mercurial crew repository.
http://hg.intevation.org/mercurial/crew

Most of this time is spent fetching details for individual metadata for items (parallelizing the metadata retrieval might be a good idea to improve display time):

Completed in 13612ms (View: 661, DB: 54) | 200 OK [http://localhost/projects/hgcrew/repository]
Completed in 14331ms (View: 601, DB: 21) | 200 OK [http://localhost/projects/hgcrew/repository]
Completed in 13540ms (View: 605, DB: 23) | 200 OK [http://localhost/projects/hgcrew/repository]

Check your error log - I had some problems importing that repository the first time, since when I created the database I did not have the default charset set to UTF-8, and some commit logs contain non-latin characters. I was seeing this:

  Changeset Load Including Associations (0.0ms)   Mysql::Error: Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '=': SELECT `changesets`.`id` AS t0_r0, `changesets`.`repository_id` AS t0_r1, `changesets`.`revision` AS t0_r2, `changesets`.`committer` AS t0_r3, `changesets`.`committed_on` AS t0_r4, `changesets`.`comments` AS t0_r5, `changesets`.`commit_date` AS t0_r6, `changesets`.`scmid` AS t0_r7, `changesets`.`user_id` AS t0_r8, `users`.`id` AS t1_r0, `users`.`login` AS t1_r1, `users`.`hashed_password` AS t1_r2, `users`.`firstname` AS t1_r3, `users`.`lastname` AS t1_r4, `users`.`mail` AS t1_r5, `users`.`mail_notification` AS t1_r6, `users`.`admin` AS t1_r7, `users`.`status` AS t1_r8, `users`.`last_login_on` AS t1_r9, `users`.`language` AS t1_r10, `users`.`auth_source_id` AS t1_r11, `users`.`created_on` AS t1_r12, `users`.`updated_on` AS t1_r13, `users`.`type` AS t1_r14, `users`.`identity_url` AS t1_r15 FROM `changesets` LEFT OUTER JOIN `users` ON `users`.id = `changesets`.user_id AND (`users`.`type` = 'User' OR `users`.`type` = 'AnonymousUser' ) WHERE (`changesets`.repository_id = 1 AND (`changesets`.`committer` = 'Wojciech Miłkowski <user@domain.tld>')) ORDER BY changesets.committed_on DESC, changesets.id DESC LIMIT 1
  SQL (0.1ms)   ROLLBACK

I fixed it by converting all tables to UTF-8:

echo 'show tables' |mysql -u<redminedbuser> -p<redminedbpass> <redminedbname> |grep -v Tables_in_redmine_dev_mercurial |xargs -i{} echo 'ALTER TABLE {} CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;' |mysql --u<redminedbuser> -p<redminedbpass> <redminedbname>

I should probably commit the changesets in batches, rather than all at once since we could use a lot of memory, and an error causes all the changesets to roll back, which hurts on initial import...

#32 Updated by Peter Fern over 4 years ago

Fix should be:

echo 'show tables' |mysql -u<redminedbuser> -p<redminedbpass> <redminedbname> |grep -v Tables_in_<redminedbname> |xargs -i{} echo 'ALTER TABLE {} CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;' |mysql --u<redminedbuser> -p<redminedbpass> <redminedbname>

Where you replace the parameters <redminedbuser>, <redminedbpass>, and <redminedbname>..

#33 Updated by Toshi MARUYAMA over 4 years ago

I use native windows ruby (not cygwin) and SQLite.

#34 Updated by Peter Fern over 4 years ago

Toshi Maruyama wrote:

I use native windows ruby (not cygwin) and SQLite.

Does SQLite deal with UTF-8? Do you have any errors in your logs?

There won't be many (any?) production installs running on SQLite, but I can't believe there could be an order of magnitude difference in performance for this operation, and I've confirmed that no update occurs when the revision count matches in DB and repository, and the tip IDs match. There might be a performance gain on merging changes by moving from 'NOT IN' to 'IN' when deleting, but I'd have to test, and it can't account for what you're seeing.

I would suggest that you must be encountering an error importing the repository, causing Redmine to attempt to re-import each time... what is changelogs.count for that repository?

#35 Updated by Peter Fern over 4 years ago

Peter Fern wrote:

what is changelogs.count for that repository?

Err, I mean changesets.count

#36 Updated by Toshi MARUYAMA over 4 years ago

SELECT count(*) AS count_all FROM "changesets" WHERE ("changesets".repository_id = 15)
  => 10426
  def fetch_changesets
    scm_info = scm.info
    return unless scm_info or scm_info.lastrev.nil?

    db_revision = latest_changeset ? latest_changeset.scmid.to_s : 0
    scm_revision = scm_info.lastrev.scmid.to_s
    # Save ourselves an expensive operation if we're already up to date

    scm_revcount = scm.num_revisions
    db_revcount = changesets.count

    logger.debug( "scm_revcount :" + scm_revcount.to_s )
    logger.debug( "db_revcount  :" + db_revcount.to_s )

Shelling out: hg -R "R:\WEB-DOWN\hg-crew/" log -r :tip --template='
'
  SQL (0.0ms)   SELECT count(*) AS count_all FROM "changesets" WHERE ("changesets".repository_id = 15) 
scm_revcount :10427
db_revcount  :10426
Shelling out: hg -R "R:\WEB-DOWN\hg-crew/" log -r :tip --template='
'
  CACHE (0.0ms)   SELECT count(*) AS count_all FROM "changesets" WHERE ("changesets".repository_id = 15) 
Shelling out: hg --debug --encoding utf8 -R "R:\WEB-DOWN\hg-crew/" --cwd "R:\WEB-DOWN\hg-crew/" log --style "S:/redmine-00/lib/redmine/scm/adapters/mercurial/hg-template-1.0-lite.tmpl" -r :"4cfd0d56be6dcbcfac90bf95e8f8750ab66fd9aa" 
  Changeset Load (3369.6ms)   SELECT * FROM "changesets" WHERE ("changesets".repository_id = 15) ORDER BY changesets.committed_on DESC, changesets.id DESC
  Changeset Delete all (249.6ms)   DELETE FROM "changesets" WHERE (scmid NOT IN ('9117c6561b0bd7792fa13b50d28239d51b78e51f',

#37 Updated by Peter Fern over 4 years ago

Hmm, so, yours is off-by-one... I wonder if it's a single commit that is not being inserted into the database correctly? You should get an error... can you upload your log as an attachment here?

Mine syncs fine on MySQL (with UTF-8):

DB rev: 1c50a954a5244a8da8d6d2311dd12d6a5f6df357, SCM rev: 1c50a954a5244a8da8d6d2311dd12d6a5f6df357

  SQL (1.1ms)   SELECT count(*) AS count_all FROM `changesets` WHERE (`changesets`.repository_id = 1) 
Shelling out: hg -R '/home/user/workspace/crew/' log -r :tip --template='
'

DB count: 10430, SCM count: 10430

  CACHE (0.0ms)   SELECT count(*) AS count_all FROM `changesets` WHERE (`changesets`.repository_id = 1) 
Shelling out: hg -R '/home/user/workspace/crew/' log -r :tip --template='

#38 Updated by Toshi MARUYAMA over 4 years ago

On MSYS/MinGW .

$ hg -R "R:\WEB-DOWN\hg-crew/" log -r :tip --template='\n' | wc
  10426       0   10426

$ which wc
/bin/wc.exe

#39 Updated by Peter Fern over 4 years ago

Are you using the following code in lib/redmine/scm/adapters/mercurial_adapter.rb?

        def num_revisions
          cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='\n'" 
          shellout(cmd) {|io| io.readlines.size}
        end

#40 Updated by Toshi MARUYAMA over 4 years ago

Peter Fern wrote:

Are you using the following code in lib/redmine/scm/adapters/mercurial_adapter.rb?
[...]

Yes.

#41 Updated by Toshi MARUYAMA over 4 years ago

Toshi Maruyama wrote:

Peter Fern wrote:

Are you using the following code in lib/redmine/scm/adapters/mercurial_adapter.rb?
[...]

Yes.

http://github.com/marutosi/redmine/blob/marutosi/lib/redmine/scm/adapters/mercurial_adapter.rb#L150

#42 Updated by Peter Fern over 4 years ago

Well... that's odd - for some reason io.readlines.size is one larger for you on Windows, perhaps there is an extra line of output generated somehow. I'm not quite sure how to resolve that since I can't reproduce it.

#43 Updated by Toshi MARUYAMA over 4 years ago

        def num_revisions
          # cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='\n'" 
          # shellout(cmd) {|io| io.readlines.size}
          cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='{rev}\n'" 
          i = 0
          logger.debug ( "test start"  )
          shellout(cmd) do |io|
            io.each_line do |line|
              i += 1
              logger.debug ( "test:" + line )
            end
          end
          logger.debug ( "test finish i: " + i.to_s  )
        end

test start
Shelling out: hg -R "R:\WEB-DOWN\REDMINE\hg-test-repo\redmine-test-mercurial-repository/" log -r :tip --template='{rev}
'
test:'0
test:''1
test:''2
test:''3
test:''4
test:''5
test:'
test finish i: 7
  CACHE (0.0ms)   SELECT count(*) AS count_all FROM "changesets" WHERE ("changesets".repository_id = 19) 

#44 Updated by Peter Fern over 4 years ago

Hmm, so you have a trailing line of output... try the attached patch.

#45 Updated by Peter Fern over 4 years ago

Peter Fern wrote:

Hmm, so you have a trailing line of output... try the attached patch.

Actually, that won't work. I just noticed all the "'" quotes, so something about the shell there is interpreting the quotes literally. Give me a moment.

#46 Updated by Peter Fern over 4 years ago

Try this instead.

If this works, we'll have to go through all the other @shellout@s and make sure we're not using single quotes there too.

#47 Updated by Peter Fern over 4 years ago

Actually, should probably just shell_quote it

#48 Updated by Toshi MARUYAMA over 4 years ago

        def num_revisions
          # cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='\n'" 
          # shellout(cmd) {|io| io.readlines.size}
          # cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='{rev}\n'" 
          cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template=\"{rev}\n\"" 
          i = 0
          logger.debug ( "test start"  )
          shellout(cmd) do |io|
            io.each_line do |line|
              i += 1
              logger.debug ( "test:" + line )
            end
          end
          logger.debug ( "test finish i: " + i.to_s  )
          i
        end
Shelling out: hg -R "R:\WEB-DOWN\REDMINE\hg-test-repo\redmine-test-mercurial-repository/" log -r :tip --template="{rev}
" 
test:0
test:1
test:2
test:3
test:4
test:5
test finish i: 6
  CACHE (0.0ms)   SELECT count(*) AS count_all FROM "changesets" WHERE ("changesets".repository_id = 19) 

#50 Updated by Toshi MARUYAMA over 4 years ago

        def num_revisions
          # cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='\n'" 
          # shellout(cmd) {|io| io.readlines.size}
          # cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template='{rev}\n'" 
          cmd = "#{HG_BIN} -R #{target('')} log -r :tip --template=#{shell_quote('{rev}\n')}" 
          i = 0
          logger.debug ( "test start"  )
          shellout(cmd) do |io|
            io.each_line do |line|
              i += 1
              logger.debug ( "test:" + line )
            end
          end
          logger.debug ( "test finish i: " + i.to_s  )
          i
        end

test start
Shelling out: hg -R "R:\WEB-DOWN\REDMINE\hg-test-repo\redmine-test-mercurial-repository/" log -r :tip --template="{rev}\n" 
test:0
test:1
test:2
test:3
test:4
test:5
test finish i: 6

#51 Updated by Peter Fern over 4 years ago

Alright, that should improve your load times significantly. There are probably still some other optimizations to be done, but I believe it's important to first be accurate, then be fast.

#52 Updated by Toshi MARUYAMA over 4 years ago

Git has this single quotes problem too (#3836 - Git repository support fails on Windows).

source:/tags/0.9.2/lib/redmine/scm/adapters/git_adapter.rb#L113

So, I tried to fix it.
But, "GitAdapter#num_revisions" method removed from trunk by r3394 and logic changed.

#53 Updated by Toshi MARUYAMA over 4 years ago

Peter Fern wrote:

Alright, that should improve your load times significantly. There are probably still some other optimizations to be done, but I believe it's important to first be accurate, then be fast.

New Repository::Git#fetch_changesets is good for performance.
Mercurial has sequential revision number, so this process is easy and fast.

  1. From database get revision number and hash id ordered by revision number from tip.
  2. If revision number and hash id are not equal, remove changeset from database.
  3. If revision number and hash id are equal, loop finish.
  4. Read new changesets from repository and insert into database.

#54 Updated by Toshi MARUYAMA over 4 years ago

I finished coding and unit test with DB migrate of adding "scm_order" field.
http://github.com/marutosi/redmine/commits/c72615fac2f99be855a3189e28fa78ed15ad2c96

Functional test "test_directory_entry" of repositories_mercurial_controller_test.rb fails.
I remove "MercurialAdapter#entry(path=nil, identifier=nil)" method, then this test finished with no error.

#56 Updated by Toshi MARUYAMA over 4 years ago

I create new git branch for Redmine 0.9 stable.
http://github.com/marutosi/redmine/commits/hg-overhaul-0.9
And I create and attach patch for Redmine 0.9.3.

This patch includes git binary patch.
Binary files are Mercurial bundle files.
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9/test/fixtures/repositories/mercurial/
To apply git binary patch, run following commands.

$ git apply hg-overhaul-0.9.3.patch
$ hg import --no-commit hg-overhaul-0.9.3.patch

#57 Updated by Toshi MARUYAMA over 4 years ago

I got a feedback on github and fixed some bugs.
I create new tag "hg-overhaul-0.9.3-20100310" for Redmine 0.9.3
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9.3-20100310

You can download tar ball or zip at following link.
http://github.com/marutosi/redmine/tarball/hg-overhaul-0.9.3-20100310
http://github.com/marutosi/redmine/zipball/hg-overhaul-0.9.3-20100310

#58 Updated by Alessio Caiazza over 4 years ago

Toshi Maruyama wrote:

I got a feedback on github and fixed some bugs.
I create new tag "hg-overhaul-0.9.3-20100310" for Redmine 0.9.3

Executing ruby script/runner "Repository.fetch_changesets" -e production no longer work with these patches.
Issue tracker where not updated if you don't visit manually the repository page.

By the way you did an excellent work ;)

#59 Updated by Toshi MARUYAMA over 4 years ago

Alessio Caiazza wrote:

Toshi Maruyama wrote:

I got a feedback on github and fixed some bugs.
I create new tag "hg-overhaul-0.9.3-20100310" for Redmine 0.9.3

Executing ruby script/runner "Repository.fetch_changesets" -e production no longer work with these patches.
Issue tracker where not updated if you don't visit manually the repository page.

By the way you did an excellent work ;)

Thank you for your feedback. Can you use git? Please pull and use latest revision of "hg-overhaul-0.9" branch.

#60 Updated by Alessio Caiazza over 4 years ago

Toshi Maruyama wrote:

Thank you for your feedback. Can you use git? Please pull and use latest revision of "hg-overhaul-0.9" branch.

git? :(
Yes the "hg-overhaul-0.9" branch fixed the problem.

Today I made a patch, based on this branch, for editing some options of .hg/hgrc inside project repository settings.

This patch needs some more work, but I'll publish it very soon (I hope).

#61 Updated by Peter Fern over 4 years ago

@Toshi - I apologise for my lack of attention to this issue, I've been out of town for most of the last 6 weeks or so, but it's excellent to see that you've been able continue with this in my absence. I will try and set aside some time in the next few days for some review and testing.

#62 Updated by Alessio Caiazza over 4 years ago

Alessio Caiazza wrote:

Toshi Maruyama wrote:

I got a feedback on github and fixed some bugs.
I create new tag "hg-overhaul-0.9.3-20100310" for Redmine 0.9.3

Executing ruby script/runner "Repository.fetch_changesets" -e production no longer work with these patches.
Issue tracker where not updated if you don't visit manually the repository page.

By the way you did an excellent work ;)

I've found an issue, but I'm not sure if it's related to your work or not.
If you change a repo url, the root_url attribute didn't get updated, so executing Repository.fetch_changesets didn't get any updates, but viewing the Repository page updates all the issues.

#63 Updated by Toshi MARUYAMA over 4 years ago

Alessio Caiazza wrote:

I've found an issue, but I'm not sure if it's related to your work or not.
If you change a repo url, the root_url attribute didn't get updated, so executing Repository.fetch_changesets didn't get any updates, but viewing the Repository page updates all the issues.

Thanks. I change repo url editable.
http://github.com/marutosi/redmine/blob/d54a7f267a2f48112e268751c4347a0476b0cea2/app/helpers/repositories_helper.rb#L169
I start an investigation.

#64 Updated by Paul Rivier over 4 years ago

for what it is worth, mercurial 1.5 has been released last week and - finally - can output log in xml :

http://mercurial.selenic.com/wiki/WhatsNew

#65 Updated by Toshi MARUYAMA over 4 years ago

Alessio Caiazza wrote:

I've found an issue, but I'm not sure if it's related to your work or not.
If you change a repo url, the root_url attribute didn't get updated, so executing Repository.fetch_changesets didn't get any updates, but viewing the Repository page updates all the issues.

I fixed and pushed.
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9
http://github.com/marutosi/redmine/commit/2f69eb14648bb3f3da773df407186922fd55c02d
Can you try this?

#66 Updated by Alessio Caiazza over 4 years ago

Toshi Maruyama wrote:

Alessio Caiazza wrote:

I've found an issue, but I'm not sure if it's related to your work or not.
If you change a repo url, the root_url attribute didn't get updated, so executing Repository.fetch_changesets didn't get any updates, but viewing the Repository page updates all the issues.

I fixed and pushed.
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9
http://github.com/marutosi/redmine/commit/2f69eb14648bb3f3da773df407186922fd55c02d
Can you try this?

cd /var/www/nolith/rails/redmine && ruby script/runner "Repository.fetch_changesets" -e production
/var/www/nolith/rails/redmine-hg/vendor/rails/railties/lib/commands/runner.rb:48: undefined method `[]' for nil:NilClass (NoMethodError)
        from /var/www/nolith/rails/redmine-hg/lib/redmine/scm/adapters/mercurial_adapter.rb:47:in `hgversion'
        from /var/www/nolith/rails/redmine-hg/lib/redmine/scm/adapters/mercurial_adapter.rb:37:in `client_version'
        from /var/www/nolith/rails/redmine-hg/lib/redmine/scm/adapters/mercurial_adapter.rb:62:in `lite_template_path'
        from /var/www/nolith/rails/redmine-hg/lib/redmine/scm/adapters/mercurial_adapter.rb:261:in `revisions'
        from /var/www/nolith/rails/redmine-hg/lib/redmine/scm/adapters/mercurial_adapter.rb:155:in `lastrev'
        from /var/www/nolith/rails/redmine-hg/app/models/repository/mercurial.rb:124:in `fetch_changesets'
        from /var/www/nolith/rails/redmine-hg/vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:217:in `send'
        from /var/www/nolith/rails/redmine-hg/vendor/rails/activerecord/lib/active_record/associations/association_proxy.rb:217:in `method_missing'
        from /var/www/nolith/rails/redmine-hg/app/models/repository.rb:183:in `fetch_changesets'
        from /var/www/nolith/rails/redmine-hg/app/models/repository.rb:181:in `each'
        from /var/www/nolith/rails/redmine-hg/app/models/repository.rb:181:in `fetch_changesets'
        from (eval):1
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `eval'
        from /var/www/nolith/rails/redmine-hg/vendor/rails/railties/lib/commands/runner.rb:48
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from script/runner:3

Maybe you should set root_url = url

#67 Updated by Toshi MARUYAMA over 4 years ago

Alessio Caiazza wrote:

Toshi Maruyama wrote:

Alessio Caiazza wrote:

I've found an issue, but I'm not sure if it's related to your work or not.
If you change a repo url, the root_url attribute didn't get updated, so executing Repository.fetch_changesets didn't get any updates, but viewing the Repository page updates all the issues.

I fixed and pushed.
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9
http://github.com/marutosi/redmine/commit/2f69eb14648bb3f3da773df407186922fd55c02d
Can you try this?

[...]

Maybe you should set root_url = url

I fixed error handling and pushed github.
Your trace means version check error.
Do you upgrade Mercurial and hg in PATH?
Or, "hg --version" does not output "version"?

On Fedora 12.

$ LANG=it_IT.UTF-8 hg --version
Mercurial Distributed SCM (version 1.5)

$ LANG=ja_JP.UTF-8 hg --version
Mercurial Distributed SCM (version 1.5)

$ rpm -q mercurial
mercurial-1.5-2.fc12.i686

#68 Updated by Toshi MARUYAMA over 4 years ago

Alessio Caiazza wrote:

Maybe you should set root_url = url

root_url set at
source:tags/0.9.3/lib/redmine/scm/adapters/abstract_adapter.rb#L50 .

#69 Updated by Alessio Caiazza over 4 years ago

Toshi Maruyama wrote:

Alessio Caiazza wrote:

Toshi Maruyama wrote:

Alessio Caiazza wrote:

I've found an issue, but I'm not sure if it's related to your work or not.
If you change a repo url, the root_url attribute didn't get updated, so executing Repository.fetch_changesets didn't get any updates, but viewing the Repository page updates all the issues.

I fixed and pushed.
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9
http://github.com/marutosi/redmine/commit/2f69eb14648bb3f3da773df407186922fd55c02d
Can you try this?

[...]

Maybe you should set root_url = url

I fixed error handling and pushed github.
Your trace means version check error.

I'm sorry. I didn't checked the error, only cut and pasted it. :(

Do you upgrade Mercurial and hg in PATH?

That's strage, I didn't....

Or, "hg --version" does not output "version"?
[...]

 $ locale | grep LANG
LANG=it_IT.UTF-8

 $ hg --version
Mercurial SCM Distribuito (versione 1.4.3)

Copyright (C) 2005-2010 Matt Mackall <mpm@selenic.com> e altri
Questo è software libero; vedere i sorgenti per le condizioni di copia.
Non c'è alcuna garanzia; neppure di COMMERCIABILITÀ o IDONEITÀ AD UNO
SCOPO PARTICOLARE.

Italian localization uses "versione" instead of "version"

By the way I pulled your commits and now it works.
Thanks a lot.

#70 Updated by Alessio Franceschelli over 4 years ago

I'll really appreciate an updated patch.
An svn patch, built against trunk, would be easy to deploy when redmine installation is a svn checkout.

Thanks!

#71 Updated by Toshi MARUYAMA over 4 years ago

Ale Franz wrote:

I'll really appreciate an updated patch.
An svn patch, built against trunk, would be easy to deploy when redmine installation is a svn checkout.

Thanks!

Thanks. This overhaul includes DB migrate. So, patch for SVN trunk is difficult.
I created git branch for SVN trunk.
http://github.com/marutosi/redmine/tree/hg-overhaul-svn-trunk
But, now main branch is hg-overhaul-0.9
http://github.com/marutosi/redmine/tree/hg-overhaul-0.9

#72 Updated by Alessio Franceschelli over 4 years ago

OK. I checked out branch hg-overhaul-0.9
In the repository tree, I don't have information (date, author, ..) about the files/folder in the root.
btw, it works fine in subfolders.
any idea? I have already tried to remove and add again the repository in redmine

thanks!

#73 Updated by Toshi MARUYAMA over 4 years ago

Ale Franz wrote:

OK. I checked out branch hg-overhaul-0.9
In the repository tree, I don't have information (date, author, ..) about the files/folder in the root.
btw, it works fine in subfolders.
any idea? I have already tried to remove and add again the repository in redmine

thanks!

I start to write announce, now this include Japanese.
http://bitbucket.org/marutosi/redmine-test-mercurial-repository/src/cda5c85f3ee7/announce.hg-overhaul.txt
I will describe details later.

I hardcode limiter of "hg log -l1 FILE".
because this is very heavy.

git has a same problem.

You can change limiter "@@limit_include_file_revs" by hand.
http://github.com/marutosi/redmine/blob/hg-overhaul-0.9/lib/redmine/scm/adapters/mercurial_adapter.rb#L32
This limiter is number of files per directory.

Mercurial branch is "named branch".
So, if named brance stored in DB, I think browsing is more fast.

#74 Updated by Toshi MARUYAMA over 4 years ago

Toshi Maruyama wrote:

Mercurial branch is "named branch".
So, if named brance stored in DB, I think browsing is more fast.

"brance" => branch"

Mercurial branch is "named branch".
So, if named branch stored in DB, I think browsing is more fast.

#75 Updated by Alessio Franceschelli over 4 years ago

I think a solution to have informations about all the files, without the need of getting the log for each file and parse it, could be to use the mercurial API, then simply get the short hex of the changeset for each file.
You could make a mercurial extension, with this custom command.

The info on every changeset are already in the db (loaded during fetch), so the information needed are just the changeset and the filezise.

btw i have no python background; I just tryed to play a little to write this extension (overhaul.py) attached. (on linux, just put it on hgext and enable it on hgrc)
it adds the command "hg overhaul [rev]", where "rev" could be also tip or a branch, as usual. (default is working copy, as every hg command)
The output can be splitted on "tab".
I don't know if it could be possible to limit it to a single folder.

maybe you can take a quick look at this extension and find out if it could help you.

#76 Updated by Toshi MARUYAMA over 4 years ago

Thanks Ale. Your suggestion and hg extension are great.
Redmine repository browser require information of "file" or "dir".
source:/tags/0.9.3/lib/redmine/scm/adapters/abstract_adapter.rb#L238
Mercurial can not handle directory.
So, this is difficult.

This overhaul support branches and this code is based on git code.
Git branch is a pointer of head. So, git need to run "git log -n 1".
Mercurial branch is "named branch" and named branch fixed on changesets.
And, I think Entry#lastrev requires last revision of branch.

Redmine 0.9 Repository::Mercurial build lastrev from DB.
source:/tags/0.9.3/app/models/repository/mercurial.rb#L32
But branch information are not stored in DB.
So, Redmine need to run "hg log -l1 --only-branch BRANCH FILE".

#77 Updated by Toshi MARUYAMA over 4 years ago

First stage of my future plans.

  • Tag is a hash of name and shortid, so redmine can use only DB info with shortid of tag.
  • If branch is single including closed branch, redmine can use only DB info.
  • Branch support can be selectable.
    Personally, I don't want to use named branch aggressively. Because named branch can not delete.

#78 Updated by Yuya Nishihara over 4 years ago

Hi, last week, I discussed about this overhaul with Toshi, and we agreed, at least, we need to resolve the following issues:

  1. It lacks performance: Because the overhauled backend spawns bunch of hg commands on every request, it's really slow.
  2. It changes database scheme with no migration: The new implementation uses Changeset.revision as "node id", but currently it's "revision number".
    It seems confusing. See revision-no-migration.png

And possible solutions (my opinion):

  1. We need to reduce the number of hg calls. Ale's idea seems great.
    Once packing all hg calls into single command, the performance should be terribly improved.
  2. We can bring back "node id" with no database change.
    Since "scmid" already contains it, all we need to do is switching to "scmid" in place of "revision".
    For details, check my patch series, http://bitbucket.org/yuja/redmine-mq/, and screenshot, revision-no-migration-2.png

My patches don't contain a number of improvements by Peter, Toshi and others, but I'll follow them in the near future. :)

#79 Updated by Alessio Franceschelli over 4 years ago

Yuya Nishihara wrote:

  1. We need to reduce the number of hg calls. Ale's idea seems great.
    Once packing all hg calls into single command, the performance should be terribly improved.

If you let me know which information you need to retrieve from the repository, I'll update the extension to meet you requirements.
I'm quite sure you need: # filtered output on a per directory basis # what else?

#80 Updated by Yuya Nishihara over 4 years ago

Ale Franz wrote:

If you let me know which information you need to retrieve from the repository, I'll update the extension to meet you requirements.
I'm quite sure you need:
  1. filtered output on a per directory basis
  2. what else?

Hi, it's really helpful! :)
I'm yet reached that point, hacking around "node id" issue now, but maybe Toshi has some idea?

#81 Updated by Yuya Nishihara over 4 years ago

Hi, I read the latest overhaul patch by Peter, Toshi, etc., then created another one:

ya-hg-overhaul-0.9-stable-2010-03-27.patch (for 0.9-stable series)

It does never change database schema, so we don't need to migrate db.
And it reduces the number of hg calls, which means it must be as fast as the 0.9-stable.

It implements or fixes:

  • Tag/branch support, #1981
  • File size issue, #3421
  • Node ID instead of revision number, #3724
  • Diff of changeset can be wrong if the previous changeset isn't the parent.

I'm not sure about #3449, so it's not included yet.

Limitations:

  • No test cases are updated yet.
  • Some pages shows revision number instead of rev:nodeid combo.
  • Some codes are badly implemented. I need to clean up them.
  • Maybe it lacks compatibility with older Mercurial clients.
    It uses helper extension to fetch file entries, sizes, tags, branches, etc. at once, but it's yet tested, say, with Mercurial 0.9.5.
    BTW, this extension is based on Alessio's, thanks!; but I mangled it too heavily. :)
  • We can reduce bunch of database queries.

Series of MQ patches are available at:

hg qclone http://bitbucket.org/yuja/redmine-mq/ ; hg qselect 0.9; hg qpush -a

#82 Updated by Yuya Nishihara over 4 years ago

Yuya Nishihara wrote:

I'm not sure about #3449, so it's not included yet.

I talked with Toshi, and found the patch happened to fix #3449.
The underlaying problem is latest_changeset does not returns the latest revision. Since my patch changes the order, now it should return the correct one.

#83 Updated by Ammler _ over 4 years ago

Any chance to reconsider #3724?

I am not the only one who prefers revision number. I really would like to see this as an option.
http://www.redmine.org/issues/3724#note-14 (or my note-20)

#84 Updated by Yuya Nishihara over 4 years ago

Marcel Gmür wrote:

I am not the only one who prefers revision number. I really would like to see this as an option.
http://www.redmine.org/issues/3724#note-14 (or my note-20)

As Toshi suggested at #3724, I tried to keep revision numbers as is. See redmine-issue4455-2010-05-17.jpg - it shows changeset id as "rev:nodeid" style. The URL uses nodeid by default, but also accepts revision number.

I uploaded the latest snapshot of my patch series: ya-hg-overhaul-0.9-stable-2010-05-17.patch - including many bug fixes since 2010-03-27, thanks to Toshi.
It's also available at http://bitbucket.org/yuja/redmine-mq-issue4455/src

#85 Updated by Ammler _ over 4 years ago

It is very nice, that the patch queue doesn't need db migration, so I can run both versions at same time. Progress is quite awesome in this respect.

One little glitch, the fetch_changesets somehow does ignore the seconds from commit dates, this causes unordered activity lists if the dev commits multiple changesets in same minute. The database already does support seconds, so this should be fixeable, I hope. :-)

I hope for some reposman support like creating of the repo after project creating and choosing mercurial as VCS. I already use Redmine togehter with hgredmine.

Greets and Thanks so far...
-Ammler http://dev.openttdcoop.org (or http://new.dev.openttdcoop.org with this mq)

#86 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

One little glitch, the fetch_changesets somehow does ignore the seconds from commit dates, this causes unordered activity lists if the dev commits multiple changesets in same minute. The database already does support seconds, so this should be fixeable, I hope. :-)

I referred Mercurial: The Definitive Guide and Mercurial issue 1799.
'isodatesec' supported at Mercuiral revision 8999d1249171

This patch is for Redmine 0.9-stable.

#87 Updated by Yuya Nishihara over 4 years ago

Toshi Maruyama wrote:

'isodatesec' supported at Mercuiral revision 8999d1249171

It means 'isodatesec' is available since Mercurial 1.0. :)

This patch is for Redmine 0.9-stable.

Thanks, now mercurial backend stores precise committed_on and commit_date.
I imported your patch to my mq-issue4455 repo, but IMHO, it's safe enough to be got in upstream separately.

#88 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

I hope for some reposman support like creating of the repo after project creating and choosing mercurial as VCS. I already use Redmine togehter with hgredmine.

I don't know what hgredmine is.
Reposman has already enabled to create repositories and register to Redmine with a following command.

$ ruby extra/svn/reposman.rb -s /tmp/hg-repo  -u /tmp/hg-repo \
    -r localhost:3000 -o foo -g foo --scm mercurial --command "hg init" 

#89 Updated by Toshi MARUYAMA over 4 years ago

Yuya's patch fails in "Activity" of Subversion project.

NoMethodError in Projects#activity

Showing app/views/projects/activity.rhtml where line #13 raised:

undefined method `truncate_long_revision_name' for RepositoriesHelper:Module

Extracted source (around line #13):

10:     <%= avatar(e.event_author, :size => "24") if e.respond_to?(:event_author) %>
11:   <span class="time"><%= format_time(e.event_datetime, false) %></span>
12:   <%= content_tag('span', h(e.project), :class => 'project') if @project.nil? || @project != e.project %>
13:   <%= link_to format_activity_title(e.event_title), e.event_url %></dt>
14:   <dd><span class="description"><%= format_activity_description(e.event_description) %></span>
15:   <span class="author"><%= e.event_author if e.respond_to?(:event_author) %></span></dd>
16: <% end -%>

/somewhere/app/helpers/repositories_helper.rb:32:in `format_revision'
/somewhere/app/models/changeset.rb:26
/somewhere/app/views/projects/activity.rhtml:13:in `_run_rhtml_app47views47projects47activity46rhtml'
/somewhere/app/views/projects/activity.rhtml:8:in `each'
/somewhere/app/views/projects/activity.rhtml:8:in `_run_rhtml_app47views47projects47activity46rhtml'
/somewhere/app/views/projects/activity.rhtml:5:in `each'
/somewhere/app/views/projects/activity.rhtml:5:in `_run_rhtml_app47views47projects47activity46rhtml'

#90 Updated by Yuya Nishihara over 4 years ago

Toshi Maruyama wrote:

Yuya's patch fails in "Activity" of Subversion project.

Fixed as http://bitbucket.org/yuja/redmine-mq-issue4455/changeset/e5b02f53025a , though it's workaround yet.

Thanks for reporting it.

#91 Updated by Ammler _ over 4 years ago

it also fails on overall activity, rss feed works.

#92 Updated by Yuya Nishihara over 4 years ago

Marcel Gmür wrote:

it also fails on overall activity, rss feed works.

Yes, the previous patch had a problem with non-hg repos, including Subversion.
If it occurs on the latest one from http://bitbucket.org/yuja/redmine-mq-issue4455 , it would be another problem.

#93 Updated by Ammler _ over 4 years ago

Well, I also get an error if I try to fetch the changesets of a hg repo:

/usr/lib64/ruby/gems/1.8/gems/rails-2.3.5/lib/commands/runner.rb:48: #<REXML::ParseException: No close tag for /log> (REXML::ParseException)                                                        

#94 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

Well, I also get an error if I try to fetch the changesets of a hg repo:
[...]

Please tell me and Yuya where hg repo is.

#95 Updated by Ammler _ over 4 years ago

I don't know, what you exactly need, here my full output of the fetch changeset run:

http://pastebin.com/FkVeC5N0

The error only happens, if there is something fetch.

I run now both apps parallel, one is the svn vanilla 0.9-stable, the other the qclone from bitbucket.org, My public address is the uses your patch queue, but I still fetch changesets with the svn version, so I don't get seconds, but it works.

Greets
Ammler

#96 Updated by Yuya Nishihara over 4 years ago

Marcel Gmür wrote:

I don't know, what you exactly need, here my full output of the fetch changeset run:

http://pastebin.com/FkVeC5N0

Thanks for testing the patches. Could you give us the Mercurial version?
I eliminated "# HG doesn't close the XML Document..." hack temporarily, and yet checked what version have the problem:

http://bitbucket.org/yuja/redmine-mq-issue4455/src/tip/hg/revisions.diff

#97 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

I don't know, what you exactly need, here my full output of the fetch changeset run:

http://pastebin.com/FkVeC5N0

The error only happens, if there is something fetch.

I run now both apps parallel, one is the svn vanilla 0.9-stable, the other the qclone from bitbucket.org, My public address is the uses your patch queue, but I still fetch changesets with the svn version, so I don't get seconds, but it works.

Greets
Ammler

Original Redmine does not have "Repository.find_by_url".
Are you using another patch?

#98 Updated by Yuya Nishihara over 4 years ago

Original Redmine does not have "Repository.find_by_url".

Yes, it has. It's provided by ActiveRecord:

http://www.ruby-forum.com/topic/94068

#99 Updated by Toshi MARUYAMA over 4 years ago

hg 1.4 fails.

$ hg update 1.4
$ make clean
$ make local

$ hg --version
Mercurial - 分散構成管理ツール(バージョン 1.4)
$ LANG=C ruby script/runner -e test "Repository.find_by_url('/WEB-DOWN/hg-repo/tortoisehg').fetch_changesets" 
/usr/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/rails/gem_dependency.rb:119:Warning: Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010.  Use #requirement
/usr/lib/ruby/gems/1.8/gems/rails-2.3.5/lib/commands/runner.rb:48: #<REXML::ParseException: No close tag for /log> (REXML::ParseException)
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:28:in `parse'
/usr/lib/ruby/1.8/rexml/document.rb:227:in `build'
/usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'

#100 Updated by Yuya Nishihara over 4 years ago

Toshi Maruyama wrote:

hg 1.4 fails.

Oops, found that "footer" is really new feature:
http://bitbucket.org/mirror/mercurial/changeset/56284451a22c

I'll fix it tomorrow. Thanks for your clarification.

#101 Updated by Yuya Nishihara over 4 years ago

Fixed "missing </log>" error on Mercurial < 1.5.
The latest patches are available at http://bitbucket.org/yuja/redmine-mq-issue4455/src

Thanks Toshi and Marcel.

#102 Updated by Ammler _ over 4 years ago

http://pastebin.com/AKj5tXXX

Mercurial 1.3.1

dev@salieri:~/redmine> hg parent
changeset: 3780:c8f523aaa8fe
branch: 0.9-stable
tag: qtip
tag: hg/version.diff
user: Yuya Nishihara <>
date: Mon Mar 22 16:30:33 2010 +0900
summary: mercurial: get client_version correctly even if LANG is set

hg 1.3 is given by the standard repos of openSUSE 11.2, I try now with 1.5

#103 Updated by Ammler _ over 4 years ago

just like to mention, that after update to 1.5.4, it works how it should.

#104 Updated by Ammler _ over 4 years ago

how would you manage patch queues with redmine?

#105 Updated by Toshi MARUYAMA over 4 years ago

Patch Branches for Mercurial is interesting.
But pbranch can not import MQ.
And I don't know how to manage target for Redmine svn trunk and 0.9 stable with pbranch.

#106 Updated by Toshi MARUYAMA over 4 years ago

If we don't use MQ and we use normal hg changesets, I think we can use Transplant extension for Redmine svn trunk and 0.9 stable.

#107 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

how would you manage patch queues with redmine?

Do you mean MQ applied hg repository on Redmine?
http://dev.openttdcoop.org/projects/redmine/repository

I and Yuya agreed Redmine do not need to take care of strip revision of shared repository.

#108 Updated by Ammler _ over 4 years ago

The problem is that redmine does fetch the changesets to the repo I qpushed from the patch queue, what happens, if I pop some out again?

Or if some patches got committed to the branch? Maybe the only solution is not to publish the repo, which the queue is based on?

#109 Updated by Yuya Nishihara over 4 years ago

Marcel Gmür wrote:

Maybe the only solution is not to publish the repo, which the queue is based on?

Indeed. As MQ is designed for private use, you shouldn't publish MQ-applied repository.

#110 Updated by Ammler _ over 4 years ago

I agree with that, but we still need possiblity for qclone, my suggestion is a new SCM type, MQ for example. With that type reposman creates the 2 repos. Repository view automatically shows only the repo in .hg/patches, the other one is ignored from redmine mainly.

#111 Updated by Toshi MARUYAMA over 4 years ago

I think there is no need to create new SCM type.
One repositry per one project is Redmine limitation #779.
And "stable" and "developement" repository is popular with Mercruial.
Please see #4924 .

#112 Updated by Ammler _ over 4 years ago

Toshi Maruyama wrote:

One repositry per one project is Redmine limitation #779.

Yes, but MQ are 2 repos, which you need only .hg/patches to be public, redmine doesn't need to care about the "base repo", but you need somehow to distinguish for reposman.

Any suggestions, how I can allow my people to create MQ without ssh access?

#113 Updated by Toshi MARUYAMA over 4 years ago

".hg/patches/.hg" is normal hg repository.
So, you can clone only it.
And "hg qclone" has "-p" option.

$ hg qclone --help

 -p --patches       location of source patch repository
$ hg clone -U http://bitbucket.org/yuja/redmine-mq/.hg/patches yuya-mq
$ ll yuya-mq/.hg
合計 6
-rw-rw-r-- 2 maruyama maruyama   57 2010-06-18 13:16 00changelog.i
-rw-rw-r-- 1 maruyama maruyama   94 2010-06-18 13:17 branchheads.cache
-rw-rw-r-- 1 maruyama maruyama   67 2010-06-18 13:17 hgrc
-rw-rw-r-- 2 maruyama maruyama   23 2010-06-18 13:16 requires
drwxrwxr-x 3 maruyama maruyama 1024 2010-06-18 13:21 store
-rw-rw-r-- 1 maruyama maruyama    7 2010-06-18 13:17 undo.branch
-rw-rw-r-- 1 maruyama maruyama    0 2010-06-18 13:17 undo.dirstate

$ hg qclone -U http://bitbucket.org/svn/redmine -p ssh://localhost/`pwd`/yuya-mq
requesting all changes
adding changesets
adding manifests
adding file changes
added 3757 changesets with 20386 changes to 3133 files (+29 heads)
requesting all changes
adding changesets
adding manifests
adding file changes
added 122 changesets with 292 changes to 73 files

Are you using redmine checkout plugin?
And do you want to show "hg qlone URL" on redmine?
This is redmine checkout plugin issue, not redmine issue.

#114 Updated by Toshi MARUYAMA over 4 years ago

Toshi Maruyama wrote:

Are you using redmine checkout plugin?
And do you want to show "hg qlone URL" on redmine?
This is redmine checkout plugin issue, not redmine issue.

Sorry, fix typo "qlone" -> "qclone".

Are you using redmine checkout plugin?
And do you want to show "hg qclone URL" on redmine?
This is redmine checkout plugin issue, not redmine issue.

#115 Updated by Ammler _ over 4 years ago

I still don't get, how you will qclone from a redmine project.

#116 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

I still don't get, how you will qclone from a redmine project.

Make two projects on redmine

One is normal hg repository, another is MQ repositry.

  1. tortoisehg-pygtk
  2. tortoisehg-pygtk-mq

"hg init" and set repositories at redmine

$ hg init /REDMINE/WORK-DIR-NO-RAID/THG-REPOS/tortoisehg-pygtk
$ hg init /REDMINE/WORK-DIR-NO-RAID/THG-REPOS/tortoisehg-pygtk-mq

push to repositories which redmine manages

$ mkdir /tmp/redmine-demo
$ hg clone -U http://bitbucket.org/marutosi/tortoisehg-pygtk \
    /tmp/redmine-demo/tortoisehg-pygtk
$ hg clone -U http://bitbucket.org/marutosi/tortoisehg-pygtk-mq/.hg/patches \
    /tmp/redmine-demo/tortoisehg-pygtk-mq
$ hg -R /tmp/redmine-demo/tortoisehg-pygtk push \
    /REDMINE/WORK-DIR-NO-RAID/THG-REPOS/tortoisehg-pygtk
$ hg -R /tmp/redmine-demo/tortoisehg-pygtk-mq push \
    /REDMINE/WORK-DIR-NO-RAID/THG-REPOS/tortoisehg-pygtk-mq

hg qclone

$ hg qclone \
      -U /REDMINE/WORK-DIR-NO-RAID/THG-REPOS/tortoisehg-pygtk \
      -p /REDMINE/WORK-DIR-NO-RAID/THG-REPOS/tortoisehg-pygtk-mq
$ hg log -l1
changeset:   6588:6995d295e6f3
tag:         tip
user:        Toshi MARUYAMA <XXXXXXXXXXXXXXXXXXXXX>
date:        Thu Jun 24 15:00:06 2010 +0900
summary:     restore original build env.

$ hg log -l1 --mq
changeset:   1:40bf2cc11197
tag:         tip
user:        Toshi Maruyama <XXXXXXXXXXXXXXXXXXXXX>
date:        Tue Jun 22 19:12:12 2010 +0900
summary:     Manage series.

#117 Updated by Eric Davis over 4 years ago

Is this ready to be reviewed to be included in trunk?

I'm concerned because the latest patches are are targeting 0.9 (should target trunk) and Ammler from IRC pointed me to http://bitbucket.org/yuja/redmine-mq-issue4455/src which looks like a bunch of patches.

#118 Updated by Holger Just over 4 years ago

BTW, in the current incarnation of the checkout plugin, you can overwrite the command shown in the plugin configuration.

The next version (which is dues soon) will resemble a more github-like UI. So you are then able to define multiple URLs / Protocols.

#119 Updated by Ammler _ over 4 years ago

I would also quite much support the switch to trunk and would of course also continue to test the patch queue on my productive system.

#120 Updated by Toshi MARUYAMA over 4 years ago

Eric Davis wrote:

I'm concerned because the latest patches are are targeting 0.9 (should target trunk) and Ammler from IRC pointed me to http://bitbucket.org/yuja/redmine-mq-issue4455/src which looks like a bunch of patches.

Yuya's MQ targets both of SVN trunk and 0.9 stable.

Following is SVN trunk applying commands.

$ hg qclone -U http://bitbucket.org/yuja/redmine-mq-issue4455
$ cd redmine-mq-issue4455
$ hg update tip --mq

$ hg qselect -s
+0.9
+experimental
+issue-2664

$ hg qselect
no active guards

$ hg update tip
$ hg branch
default

$ hg pare
changeset:   3809:eb19c68290e6
tag:         tip
user:        edavis10@e93f8b46-1217-0410-a6f0-8f06a7374b81
date:        Wed Jun 30 03:32:18 2010 +0000
summary:     Set @project so macros will work on the welcome and project list. #5781

$ hg qpush -a
applying hg/isodatesec.patch
applying git/changeset-order.diff
applying sort-changesets-by-id.diff
applying changeset-identifier.diff
applying changeset-query-identifier.diff
applying better-format-revision.diff
applying wiki-commit-link-changeset-identifier.diff
applying issue-auto-close-revision-format.diff
applying activity-changeset-identifier.diff
applying repoview-nodeid.diff
applying revision-view-input-field.diff
applying fetch-changesets-catch-command-failed.diff
applying hgext/helper-by-ale.diff
applying hgext/helper-by-me.diff
applying hg/cmd.diff
applying hg/diff.diff
applying hg/annotate.diff
applying hg/revisions.diff
applying hg/summary.diff
applying hg/fetch-changesets.diff
applying hg/latest-changesets.diff
applying hg/tags.diff
applying hg/entries.diff
applying hg/branches.diff
applying hg/version.diff
skipping hg/plain.diff - guarded by ['+experimental']
now at: hg/version.diff

#121 Updated by Toshi MARUYAMA over 4 years ago

You can export git patch with following command.

$ hg export qbase:qtip --git

#122 Updated by Toshi MARUYAMA over 4 years ago

I tried to push Yuya's MQ to my github with hg-git, but I could not clone git repository with same following problem.

hg-git issue 117 "Exception on first clone"
http://github.com/schacon/hg-git/issues#issue/117

#123 Updated by Yuya Nishihara over 4 years ago

Eric Davis wrote:

Is this ready to be reviewed to be included in trunk?

Honestly, not yet. Though most parts are okay, some patches contain rough edges like hard-coded SQL, incompatibility with git plugin, etc.

I'm concerned because the latest patches are are targeting 0.9

As Toshi mentioned, it's actually trunk-based. It just includes some changes from trunk to be applied to 0.9 series.

which looks like a bunch of patches.

I'll fold them into two or three when it's ready.

#124 Updated by Toshi MARUYAMA over 4 years ago

Eric, as mentioned at note-86 and note-87, isodatesec.patch has no risk.
Could you commit this patch?

#125 Updated by Ammler _ over 4 years ago

Maybe you should create own tickets about finished independent patches.

I have also a urgent bug to report:

Projects without any tracker don't work, workaround is to add a tracker, but that isn't needed on vanilla redmine.

Greets
Ammler

#126 Updated by Ammler _ over 4 years ago

snippet from log:

ActionView::TemplateError (Validation failed: Tracker can't be blank, Tracker can't be blank) on line #45 of app/views/layouts/base.rhtml:
42:                                                                                                                                       
43:     <% if display_main_menu?(@project) %>                                                                                             
44:     <div id="main-menu">                                                                                                              
45:         <%= render_main_menu(@project) %>                                                                                             
46:     </div>                                                                                                                            
47:     <% end %>                                                                                                                         
48: </div>                                                                                                                                

    vendor/plugins/redmine_code_review/app/models/code_review_project_setting.rb:37:in `find_or_create'
    vendor/plugins/redmine_code_review/init.rb:52:in `evaluate_init_rb'                                
    lib/redmine/menu_manager.rb:282:in `call'                                                          
    lib/redmine/menu_manager.rb:282:in `allowed_node?'                                                 
    lib/redmine/menu_manager.rb:252:in `menu_items_for'                                                
    lib/redmine/menu_manager.rb:251:in `each'
    lib/redmine/menu_manager.rb:251:in `menu_items_for'
    lib/redmine/menu_manager.rb:176:in `render_menu'
    lib/redmine/menu_manager.rb:166:in `render_main_menu'
    app/views/layouts/base.rhtml:45:in `_run_rhtml_app47views47layouts47base46rhtml'
    passenger (2.2.7) lib/phusion_passenger/rack/request_handler.rb:95:in `process_request'
    passenger (2.2.7) lib/phusion_passenger/abstract_request_handler.rb:207:in `main_loop'
    passenger (2.2.7) lib/phusion_passenger/railz/application_spawner.rb:374:in `start_request_handler'
    passenger (2.2.7) lib/phusion_passenger/railz/application_spawner.rb:332:in `handle_spawn_application'
    passenger (2.2.7) lib/phusion_passenger/utils.rb:184:in `safe_fork'
    passenger (2.2.7) lib/phusion_passenger/railz/application_spawner.rb:330:in `handle_spawn_application'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:352:in `__send__'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:352:in `main_loop'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:196:in `start_synchronously'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:163:in `start'
    passenger (2.2.7) lib/phusion_passenger/railz/application_spawner.rb:209:in `start'
    passenger (2.2.7) lib/phusion_passenger/spawn_manager.rb:262:in `spawn_rails_application'
    passenger (2.2.7) lib/phusion_passenger/abstract_server_collection.rb:126:in `lookup_or_add'
    passenger (2.2.7) lib/phusion_passenger/spawn_manager.rb:256:in `spawn_rails_application'
    passenger (2.2.7) lib/phusion_passenger/abstract_server_collection.rb:80:in `synchronize'
    passenger (2.2.7) lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
    passenger (2.2.7) lib/phusion_passenger/spawn_manager.rb:255:in `spawn_rails_application'
    passenger (2.2.7) lib/phusion_passenger/spawn_manager.rb:154:in `spawn_application'
    passenger (2.2.7) lib/phusion_passenger/spawn_manager.rb:287:in `handle_spawn_application'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:352:in `__send__'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:352:in `main_loop'
    passenger (2.2.7) lib/phusion_passenger/abstract_server.rb:196:in `start_synchronously'

Rendering /home/dev/redmine.hg/public/500.html (500 Internal Server Error)

#127 Updated by Yuya Nishihara over 4 years ago

Marcel Gmür wrote:

Projects without any tracker don't work, workaround is to add a tracker, but that isn't needed on vanilla redmine.

As your log said, it's redmine_code_review's bug, and they've already released the fixed version. See http://www.r-labs.org/issues/392

#128 Updated by Ammler _ over 4 years ago

:-$ shame on me :'-(

#129 Updated by Toshi MARUYAMA over 4 years ago

Yuya, could you tell us your thought of #5117.

#130 Updated by Toshi MARUYAMA over 4 years ago

Yuya Nishihara wrote:

incompatibility with git plugin,

Do you mean #5357 ?

#131 Updated by Yuya Nishihara over 4 years ago

Yuya, could you tell us your thought of #5117.

It seems good. IMHO, there's room to clean up more, though.

incompatibility with git plugin,

Do you mean #5357 ?

Yes, git adapter doesn't consider the insertion order.

#132 Updated by Ammler _ over 4 years ago

order by id is fine, just be aware that adding the commits to the db is correct, for example test with commits with exactly same timestamp

#133 Updated by Ammler _ over 4 years ago

Redmine should use the revision number for links per default or else compare the hashes of imported changes with the repo.

As you could rollback, rebase or strip changesets from repos, the hashes and revision numbers can change or the hash could go at all.

#134 Updated by Toshi MARUYAMA over 4 years ago

Marcel Gmür wrote:

Redmine should use the revision number for links per default or else compare the hashes of imported changes with the repo.

#135 Updated by Toshi MARUYAMA over 4 years ago

Redmine mercurial mirror on bitbucket stopped updating at r3840. Because it is made by hgsubversion , you can pull from SVN only after r3841 to this hg repository.
I described commands at Open discussion: redmine mercurial mirror on bitbucket .

#136 Updated by Alessio Franceschelli over 4 years ago

what we need to do to propose it to 1.0.1?
any issues pending?

#137 Updated by Eric Davis over 4 years ago

Alessio Franceschelli wrote:

what we need to do to propose it to 1.0.1?

It won't go into 1.0.1 but once it's ready it can go into 1.1.0.

#139 Updated by Toshi MARUYAMA about 4 years ago

Unit and functional tests by nodeid.

#140 Updated by Yuya Nishihara about 4 years ago

Thanks Toshi, I've imported these three tests.
And I decided to enable issue tracker at bitbucket to receive these kind of patches. This thread looks quite long, so I don't want to mess up here just for my patch.

http://bitbucket.org/yuja/redmine-mq-issue4455/issues

#141 Updated by Ammler _ about 4 years ago

I have another proposal for this patch queue:
#2624 (Repository statistics should honour user-mapping)

It isn't mercurial only issue, but it happens quite likely with Mercurial after the user setup new system and might change the username, add email or change email etc....

I already use it locally in the mq and it seems to work nicely. I'll add my patch here but you might with your interest also relate the ticket it belongs to...

#142 Updated by Ammler _ about 4 years ago

Eric Davis wrote:

Alessio Franceschelli wrote:

what we need to do to propose it to 1.0.1?

It won't go into 1.0.1 but once it's ready it can go into 1.1.0.

I guess, chances to get this in trunk might be better to split the queue in some categories and propose those with its own ticket for inclusion. Maybe you can manage that with your subfolders, you already have?

#143 Updated by Eric Davis about 4 years ago

Marcel Gmür wrote:

I guess, chances to get this in trunk might be better to split the queue in some categories and propose those with its own ticket for inclusion. Maybe you can manage that with your subfolders, you already have?

If you can convert the code it into multiple patches I can review and apply individually, then that would be great. For example:

  • one patch for branch support
  • one patch for tag support
  • one patch to fix bug 3421
  • ...

If that's possible, some of the bug fixes might be accepted into 1.0.1. The 1.0.x series can't get major features or rewrites since it's "stable" but bug fixes and minor features can still be added.

#144 Updated by Yuya Nishihara about 4 years ago

Currently these two patches are ready. Could you review them?

And I think fix_mercurial_localization_issue.diff of #5117 is good candidate.

I'll prepare repository on github if it's convenient than patches as attachments.

#145 Updated by Toshi MARUYAMA about 4 years ago

Yuya Nishihara wrote:

I'll prepare repository on github if it's convenient than patches as attachments.

I pushed my github branch.
http://github.com/marutosi/redmine/commits/hg-patches-svntrunk

#146 Updated by Toshi MARUYAMA about 4 years ago

I confirm it's fine.

$ hg clone -U http://bitbucket.org/tortoisehg/thg -r 55bb727fe2fade2187
$ hg glog -l3
o  changeset:   7945:55bb727fe2fa
|  tag:         tip
|  user:        Toshi MARUYAMA <XXXXXXXXXXXXXXXXXXX>
|  date:        Mon Jul 26 18:44:02 2010 +0900
|  summary:     tortoisehg.spec: remove hgtk.
|
o  changeset:   7944:40cd959737d9
|  user:        Toshi MARUYAMA <XXXXXXXXXXXXXXXXXXXXX>
|  date:        Mon Jul 26 21:36:07 2010 +0900
|  summary:     nautilus: PyQt(thg) support.
|
o  changeset:   7943:cef10e7ab0e5
|  user:        Yuya Nishihara <YYYYYYYYYYYYYYYYYYYYY>
|  date:        Sun Jul 25 17:04:10 2010 +0900
|  summary:     setup: add PyQt4 dependencies instead of PyGtk's
|

#147 Updated by Toshi MARUYAMA about 4 years ago

Yuya Nishihara wrote:

Eric Davis wrote:

Is this ready to be reviewed to be included in trunk?

Honestly, not yet. Though most parts are okay, some patches contain rough edges like hard-coded SQL, incompatibility with git plugin, etc.

Is this related #6019 and #6159 ?
svn_latest_changesets_improvement.diff at #6159 seems same approach of Yuya's SQL.

#148 Updated by Yuya Nishihara about 4 years ago

Toshi MARUYAMA wrote:

Is this related #6019 and #6159 ?
svn_latest_changesets_improvement.diff at #6159 seems same approach of Yuya's SQL.

Yes. Thanks to it, I was able to rewrite the "hard-coded SQL" patch without sub-query, see #6159.

#149 Updated by Gabriele Garuglieri about 4 years ago

Each time one is pulling with rebase both the revisions and hashes of local changesets changes and the repository view shows the rebased changests multiple times with new and old revisons and hashes.
Besides the old revisions are now overriding some pulled revisions that never shows in the repository view.
Are these patches also addressing this problem?

Thanks,
Gabriele

#150 Updated by Toshi MARUYAMA about 4 years ago

Gabriele Garuglieri wrote:

Each time one is pulling with rebase both the revisions and hashes of local changesets changes and the repository view shows the rebased changests multiple times with new and old revisons and hashes.
Besides the old revisions are now overriding some pulled revisions that never shows in the repository view.
Are these patches also addressing this problem?

I described this problem at http://www.redmine.org/issues/3724#note-18 .
I tried to resolve it, but it is very difficult.
Because 'revision' of database is not unique.
I and Yuya agreed that history editing is rare case for public(shared) repository and Redmine does not need treat history editing for public(shared) repository.

#151 Updated by Gabriele Garuglieri about 4 years ago

I understand that given the constraints with which we have to live, i.e. that the revision and hash, that allow to uniquely identify a changeset in mercurial, can change, the problem of static links in wikis and issues cannot be solved with the current implementation. Perhaps we could think of linking a changeset through a sort of indirection with an internal identifier that could be a hash of committer, date and time of commit, comment and changes that could be used, albeit slowly, to identify and relink again the correct changeset should the history change. But, beside the breaking change in db that this would introduce, this would not solve yet the problem of the static name of the link in pages and i don't know how high could be the probability of hashes collision given the few elements to hash.

Anyhow to begin with i think it could be helpful if it could be integrated in redmine, unless it has some adverse effect that at the moment i don't know, what i do now manually.
After rebase i detect the first out of place revision and then run the following queries:

delete from changes 
where changes.changeset_id in
    (select changesets.id from changesets 
    where changesets.repository_id = <repository_id>
    and changesets.id >= 
        (select changesets.id from changesets
        where changesets.repository_id = <repository_id>
        and revision = 'rev#'
        )
    );

delete from changesets_issues 
where changesets_issues.changeset_id in
    (select changesets.id from changesets 
    where changesets.repository_id = <repository_id>
    and changesets.id >= 
        (select changesets.id from changesets
        where changesets.repository_id = <repository_id>
        and revision = 'rev#'
        )
    );

delete from changesets 
where changesets.repository_id = <repository_id>
and changesets.id >= 
    (select changesets.id from changesets
    where changesets.repository_id = <repository_id>
    and revision = 'rev#'
    );

The next time i visualize the repository it will refetch the missing revisions in the right order. It's a hack but surely faster than redefining the repo and refetching all the revisions.
They will still be visualized in the wrong order since the only sort available is by date but i think this problem is being addressed by one of your patches.

I think that if one knows exactly what he is doing and knows where the history of two repo clones diverged perhaps this could also partially help (static links in pages excluded) for the issue in http://www.redmine.org/issues/3724#note-18 you mentioned first.

Best regards,
Gabriele

#152 Updated by Toshi MARUYAMA about 4 years ago

Gabriele Garuglieri wrote:

They will still be visualized in the wrong order since the only sort available is by date but i think this problem is being addressed by one of your patches.

Sorry, my patches at #3724 are obsolete.
I am supporting Yuya, now.

#153 Updated by Toshi MARUYAMA about 4 years ago

Gabriele, please try yuya's MQ.
Yuya's MQ resolves problems that you mention.
http://bitbucket.org/yuja/redmine-mq-issue4455

#154 Updated by Toshi MARUYAMA about 4 years ago

Toshi MARUYAMA wrote:

Sorry, my patches at #3724 are obsolete.

I deleted my obsolete remote branches of github.
I don't know why does github show http://github.com/marutosi/redmine/commit/30845a8c8ed51a71beb6e1d4df4dfbcf3a55c444 that I described at http://www.redmine.org/issues/3724#note-19 .

#155 Updated by Toshi MARUYAMA about 4 years ago

As far as I know, redmine 1.0 sets relations between issues and changesets in only case of fetching changesets.
There is a issue "Feature #2009 Manually add related revisions".

If '#issue_number" is written in commit log, redmine add 'rNN' at a journal.
I think it is a problem for Mercurial.
Yuya's MQ changes from 'rNN' to 'commit:AAA' for it at http://bitbucket.org/yuja/redmine-mq-issue4455/src/beeb014b9a89/hg-use-scmid.patch/issue-auto-close.diff.

#156 Updated by Ammler _ about 4 years ago

bitbucket.org/svn/redmine seems stalled

I had running my own hg convert already at http://hg.openttdcoop.org/redmine and could push it to bitbucket.org/OpenTTD/redmine, maybe you like to use that, we could also use our redmine, which is based on the MQ for the project management, if you like.

http://dev.openttdcoop.org/projects/redmine
we have already HGRedmine running for push authentification

Please tell me, if there is interest...

#157 Updated by Toshi MARUYAMA about 4 years ago

Thanks, Marcel.

I don't like "hg convert", because we need to keep .hg/shamap as described at http://www.redmine.org/boards/1/topics/12312#message-15863 .
And I don't need old branches.

I start cron job of redmine svn mirror and pushed following repositories.

But my cron job works only when my Fedora is living.

#158 Updated by Ammler _ about 4 years ago

something else:

tip is not always head of default, it can also be head of a branch.

It should somehow be viewable, which branch we currently see, maybe by showing always a branch in the branch menu item, as there isn't a empty branch anyway.

#159 Updated by Moritz Voss about 4 years ago

http://dev.openttdcoop.org/projects/redmine
we have already HGRedmine running for push authentification

Please tell me, if there is interest...

Absolutely.

#160 Updated by Alessio Caiazza about 4 years ago

Alessio Caiazza wrote:

Today I made a patch, based on this branch, for editing some options of .hg/hgrc inside project repository settings.

This patch needs some more work, but I'll publish it very soon (I hope).

... 7 months later ...

I'm too fast ;)
As I promised, Feature #6515 (hgrc editor)

#161 Updated by Toshi MARUYAMA about 4 years ago

Marcel Gmür wrote:

Maybe you should create own tickets about finished independent patches.

I created new issue for hg-isodatesec.patch. New issue is #6656 .

#162 Updated by Toshi MARUYAMA about 4 years ago

Yuya Nishihara wrote:

Currently these two patches are ready. Could you review them?

And I think fix_mercurial_localization_issue.diff of #5117 is good candidate.

I'll prepare repository on github if it's convenient than patches as attachments.

I finished updating these issues status.
All these patches include unit tests and independent for each other.
Please set target of these issues to next stable 1.0.3 or 1.1.

1. Mercurial adapter loses seconds of commit times

Issue MQ Github revision Github pull request

2. Sort changesets by revision and fix fetching performance problem

Issue MQ

Github revision

Because I pushed by hg-git, author and commit date are 2010-07-29.
So, github commit ordering is strange.
This is same problem with #5357.

Github pull request

3. Mercurial_adapter should ensure the right LANG environment variable

Issue

MQ Github revision Github pull request

#163 Updated by Eric Davis about 4 years ago

  • Assignee set to Eric Davis
  • Target version set to Unplanned

You think it's ready to be included in Redmine 1.1?

Setting this to Unplanned and assigning it to myself so I can remember to review the pull requests.

#164 Updated by Toshi MARUYAMA about 4 years ago

Eric Davis wrote:

You think it's ready to be included in Redmine 1.1?

Yes.

"1. Mercurial adapter loses seconds of commit times (#6656)" and "3. Mercurial_adapter should ensure the right LANG environment variable(#5117)" are bug fix with no behaviour change, so I wish to set target next stable 1.0.3.

"2. Sort changesets by revision and fix fetching performance problem (#3449, #3567)" is bug fix with behaviour change.
But this patch effect only for Mercrurial and does not effect other SCM.
I think all Mercurial user can accept this behaviour change.
Please judge set target 1.0.3 or 1.1.

This overhaul(#4455) has many issues.
http://bitbucket.org/yuja/redmine-mq-issue4455/issues?status=new&status=open
We still have many works.

#165 Updated by Toshi MARUYAMA about 4 years ago

Toshi MARUYAMA wrote:

As far as I know, redmine 1.0 sets relations between issues and changesets in only case of fetching changesets.
There is a issue "Feature #2009 Manually add related revisions".

If '#issue_number" is written in commit log, redmine add 'rNN' at a journal.
I think it is a problem for Mercurial.
Yuya's MQ changes from 'rNN' to 'commit:AAA' for it at http://bitbucket.org/yuja/redmine-mq-issue4455/src/beeb014b9a89/hg-use-scmid.patch/issue-auto-close.diff.

I created new issue for this problem. New issue is #6681.

#166 Updated by Eric Davis almost 4 years ago

  • Assignee deleted (Eric Davis)

#167 Updated by Sam Kuper almost 4 years ago

I believe issue #6892 should be taken into consideration as part of the overhaul.

#168 Updated by Toshi MARUYAMA almost 4 years ago

Sam Kuper wrote:

I believe issue #6892 should be taken into consideration as part of the overhaul.

reposman.rb supports to create Mercurial repository.
Please see note-88 of this issue.

#169 Updated by Ammler _ almost 4 years ago

shouldn't you apply this mq to the repo http://bitbucket.org/marutosi/redmine-hgsubversion-allbranches-clean since the current one is "dead"?

#170 Updated by Toshi MARUYAMA almost 4 years ago

Marcel Gmür wrote:

shouldn't you apply this mq to the repo http://bitbucket.org/marutosi/redmine-hgsubversion-allbranches-clean since the current one is "dead"?

My hourly cron job is running only while my machine is alive.
Now I have executed this job by hand.

#171 Updated by Toshi MARUYAMA almost 4 years ago

Marcel, are you synchronizing my repository and your repository?
Following commit id are same.

The following is main part of my hourly cron job.

hg -q -R ${ALLDIR} pull http://bitbucket.org/${BBUSER}/redmine-hgsubversion-allbranches-clean 
hg -q -R ${ALLDIR} svn rebuildmeta http://redmine.rubyforge.org/svn/
hg -q -R ${ALLDIR} pull http://redmine.rubyforge.org/svn/
hg -q -R ${ALLDIR} push https://${BBUSER}:${BBPW}@bitbucket.org/${BBUSER}/redmine-hgsubversion-allbranches-clean
hg -q -R ${ALLDIR} push -b default https://${BBUSER}:${BBPW}@bitbucket.org/${BBUSER}/redmine-hgsubversion-svntrunk-clean
hg -q -R ${ALLDIR} push -b 1.0-stable https://${BBUSER}:${BBPW}@bitbucket.org/${BBUSER}/redmine-hgsubversion-1.0-clean

#172 Updated by Ammler _ almost 4 years ago

Yes, I changed my paths now and sync your repo and the MQ only (no qclone anymore)

dev.openttdcoop.org runs on a quite stable public server, as it is the official place for many addons of OpenTTD, if you like we could run a convert there and push to bitbucket.

I don't care to use hg convert or hgsubversion, both need a support file, btw.

#173 Updated by Ammler _ almost 4 years ago

Edit:

http://svn.dev.openttdcoop.org is the clean svn (trunk) version of Redmine
http://dev.openttdcoop.org is mainly this MQ and some minor patches
(both use the same DB)

#174 Updated by Toshi MARUYAMA almost 4 years ago

Marcel Gmür wrote:

dev.openttdcoop.org runs on a quite stable public server, as it is the official place for many addons of OpenTTD, if you like we could run a convert there and push to bitbucket.

I wish so.

#175 Updated by Ammler _ almost 4 years ago

I setup now a new hgsubversion convert to https://bitbucket.org/redmine/redmine, also the hgsubversion metdadate are uploaded, so it would be quite easy for someone else to continue the convert, whenever something on my side fails.

Also the user redmine is made just for this, if someone of the officials feels, I am happy to handover that account.

the sync runs in hourly base, I don't think, we need the branches alone, as you can use "hg pull -b <branch>", but if you like clean branches, tell me.

I have also included the redmine convert to our Redmine which does use this MQ, if you wish, you are welcome to develop there, I can give you access to our redmine which does give you the ability to create own projects and repos...

Something else? Please tell or mail to

#176 Updated by Ammler _ almost 4 years ago

stupid me, didn't see that you can rebuild the metadata..., fixed and also a trunk and 1.0 repo will be synced...

do you now create the mq from there?

#177 Updated by Toshi MARUYAMA almost 4 years ago

Revision 0 of original hgsubversion mirror and my repository are 711945aa86b2 .
Revision 0 of your repository is 108d86396bc7 .
Did you create new revision from revesion 0?

Are you using author map?
Commit user of original hgsubversion mirror is "jplang@e93f8b46-1217-0410-a6f0-8f06a7374b81".
And are you using another setting? Can you publish it?

Python PEP 385 "Migrating from svn to Mercurial" publishes scripts of hgsubversion at http://hg.python.org/pymigr .
And this scripts use author-map .

#178 Updated by Toshi MARUYAMA almost 4 years ago

I am using no setting in current hgsubversion pulling.
So, commit user is "jplang@e93f8b46-1217-0410-a6f0-8f06a7374b81".
Redmine Subversion branch layout is standard which is trunk, branches and tags.

#179 Updated by bo ye almost 4 years ago

I think this issue should be a feature tracker, rather than a patch tracker.

#180 Updated by Ammler _ almost 4 years ago

Toshi: hmm, true, didn't think it is necessary to have same hashes, so I removed the (imo) unnecessary host uid, my settings are:
https://bitbucket.org/redmine/redmine/wiki (no authormap, just removing the host)

bo ye: this issue is not just one patch, it is a quite big Patch Queue ;-)

#181 Updated by Ammler _ almost 4 years ago

(the uuid can still be found in .hg/svn/uuid)

#182 Updated by Toshi MARUYAMA almost 4 years ago

I found new original Redmine problem and create new issue #7064.

#183 Updated by bo ye almost 4 years ago

Marcel Gmür wrote:

bo ye: this issue is not just one patch, it is a quite big Patch Queue ;-)

I just wish people could see this issue on the roadmap page of redmine(no need to click into unplanned page), so there would be more up up votes. :)

#184 Updated by Ammler _ almost 4 years ago

The problem is simply from what I get in the IRC channel, no active contributor uses mecurial, so this really needs to work quite well for any progress. But Toshi and Yuya do a very good job on maintaining the MQ. I run the MQ on our productive system now for quite some time already without any issues. (Thanks to no db changes, mostly.)

#185 Updated by Toshi MARUYAMA almost 4 years ago

Marcel Gmür wrote:

(Thanks to no db changes, mostly.)

Patch at note-2 of #7064 has db changes.

#186 Updated by Toshi MARUYAMA almost 4 years ago

Mercurial test repository at note-2 of #7064 contains two branches and two tags.
And it has 'Ü' in latin-1 not UTF-8.
And it has '%' and '_' for SQL 'like' escape test.

I wish to be committed the patch at note-2 of #7064 to Redmine SVN trunk firstly.
Please review it.

#187 Updated by Yuya Nishihara almost 4 years ago

Hi, thanks to the effort of Toshi and Marcel, we provide the snapshot release of Redmine with whole patches applied:

hg clone https://bitbucket.org/redmine/redmine-issue4455-snapshot/#1.0-issue4455

I'll keep it up-to-date until these patches can be merged into Redmine 1.1 or 1.2?

#188 Updated by Ammler _ almost 4 years ago

r4539:r4542 will break some patches in the queue (hg/cmd & hg/diff at least)

#189 Updated by Yuya Nishihara almost 4 years ago

Marcel Gmür wrote:

r4539:r4542 will break some patches in the queue (hg/cmd & hg/diff at least)

Rebased onto r4542. Please try the latest MQ patches.

#190 Updated by Toshi MARUYAMA almost 4 years ago

I saw plans that Redmine 1.1 will be released in the next few weeks at Plans for next Redmine releases .

I wish to set target of following two patches to Redmine 1.1.
These patches have unit tests and functional tests.

1. Sort changesets by revision, fetching performance problem and file name encoding

Patch

Issue

2. Changing revision label and identifier at SCM adapter level

Patch

Issue

#191 Updated by Yuya Nishihara almost 4 years ago

Toshi MARUYAMA wrote:

I wish to set target of following two patches to Redmine 1.1.
These patches have unit tests and functional tests.

1. Sort changesets by revision, fetching performance problem and file name encoding

Patch

Hi, they're different issue.
"sort changesets by revision" is simple enough for 1.1, but IMO "file name encoding" isn't.

#192 Updated by Toshi MARUYAMA almost 4 years ago

  • Assignee set to Toshi MARUYAMA
  • % Done changed from 90 to 0

#193 Updated by Toshi MARUYAMA almost 4 years ago

  • Tracker changed from Patch to Feature

#194 Updated by Toshi MARUYAMA almost 4 years ago

I can see watchers list of issues.
I notice this issue watchers list is empty.
Why?

#195 Updated by Jean-Philippe Lang almost 4 years ago

Toshi Maruyama wrote:

I can see watchers list of issues.
I notice this issue watchers list is empty.
Why?

I see 3 observers in the sidebar for this ticket.

#196 Updated by Toshi MARUYAMA almost 4 years ago

Jean-Philippe Lang wrote:

Toshi Maruyama wrote:

I can see watchers list of issues.
I notice this issue watchers list is empty.
Why?

I see 3 observers in the sidebar for this ticket.

Yes.
At I wrote note 194, watchers list was empty.
So, I wrote a message at http://www.redmine.org/boards/1/topics/12312?r=20538#message-20538 .

#197 Updated by Ammler _ almost 4 years ago

He :-)

I don't get, why empty watchers list is an error, personally I read the rss feed from this ticket...

#198 Updated by Toshi MARUYAMA almost 4 years ago

I finished committing to Redmine 1.1 about Mercurial.
For a while, I focus other SCMs (Darcs, CVS and Bazaar).
Yuya, please update your MQ.

#199 Updated by Jean-Philippe Lang almost 4 years ago

Yes, I think we're done with 1.1 (release planned this week-end).
Can we close this ticket and split what is left in specific tickets? This thread becomes hard to read.

#200 Updated by Toshi MARUYAMA almost 4 years ago

Jean-Philippe Lang wrote:

Can we close this ticket and split what is left in specific tickets? This thread becomes hard to read.

I think so, too. I will summarize Mercurial issues.

#201 Updated by Yuya Nishihara almost 4 years ago

Toshi MARUYAMA wrote:

Yuya, please update your MQ.

Thanks. Rebased MQ to SVN trunk r4651 and dropped empty/needless patches.
Note that MQ patches no longer support 1.0-stable. And it can be broken due to possible bad conflict resolution.

https://bitbucket.org/yuja/redmine-mq-issue4455/

Jean-Philippe Lang wrote:

Can we close this ticket and split what is left in specific tickets? This thread becomes hard to read.

+1

#202 Updated by Toshi MARUYAMA almost 4 years ago

Mercurial overhaul next steps

  1. Defect #3724 change revision label and identifier
  2. Defect #3421 browse directory tree
  3. Feature #1981 tag
  4. Feature #7246 "named branch"

Please add notes.

#203 Updated by Ammler _ almost 4 years ago

A little proposal for the Mercurial overhaul MQ: instead using guard for the branch 1.1-stable, I would use also branch in the MQ, it does make it easier for keeping them working for branches.

#204 Updated by Toshi MARUYAMA almost 4 years ago

I will close #3724 and focus #3421.
I have some questions about redminehelper.py .

Yuya's MQ contains fixing #3421 and latest changesets improvement.
Could you change order in series?

#205 Updated by Yuya Nishihara almost 4 years ago

Marcel Gmür wrote:

A little proposal for the Mercurial overhaul MQ: instead using guard for the branch 1.1-stable, I would use also branch in the MQ, it does make it easier for keeping them working for branches.

Hmm, it could be, but I'm too lazy to manage multiple branches.

Toshi MARUYAMA wrote:

I have some questions about redminehelper.py .

  • Is copyright correct?

I think it's correct. If your concern is wording, it can be changed to follow the Redmine's coding style.

How to deal redminehelper.pyo and redminehelper.pyc?
What happens if redmine process does not have permission to write directory?

Without write permission, python silently generates nothing.

Could you change order in series?

Yeah, I should do it. But, as I mentioned before, I want to rewrite redminehelper.py to use XML instead of original format.

#206 Updated by Ammler _ almost 4 years ago

Yuya Nishihara wrote:

Marcel Gmür wrote:

A little proposal for the Mercurial overhaul MQ: instead using guard for the branch 1.1-stable, I would use also branch in the MQ, it does make it easier for keeping them working for branches.

Hmm, it could be, but I'm too lazy to manage multiple branches.

you really think, it is easier to mess with guards?

Somthing elese related to the the branches and tags menu: Sorting should be DESC, newest branch/tag on top, I guess like bitbucket...

#207 Updated by Toshi MARUYAMA almost 4 years ago

If fixing #3421 requires redminehelper.py, "extra/mercurial/redminehelper.py" path is strange.
Should we relocate path as following?

lib/redmine/scm/adapters
mercurial
├── ext
│   └── redminehelper.py
└── xml
    ├── hg-template-0.9.5.tmpl
    └── hg-template-1.0.tmpl

#208 Updated by Yuya Nishihara over 3 years ago

Marcel Gmür wrote:

Somthing elese related to the the branches and tags menu: Sorting should be DESC, newest branch/tag on top, I guess like bitbucket...

Oops, it looks to be a random order because of Hash. I'll fix it later, thanks!

Toshi MARUYAMA wrote:

lib/redmine/scm/adapters

mercurial
├── ext
│   └── redminehelper.py
└── xml
    ├── hg-template-0.9.5.tmpl
    └── hg-template-1.0.tmpl

Not sure, but isn't it enough like this?

mercurial/
+ redminehelper.py
+ hg-template-0.9.5.tmpl
+ hg-template-1.0.tmpl

#209 Updated by Yuya Nishihara over 3 years ago

Marcel Gmür wrote:

A little proposal for the Mercurial overhaul MQ: instead using guard for the branch 1.1-stable, I would use also branch in the MQ, it does make it easier for keeping them working for branches.

I forked MQ patches for 1.1-stable because it becomes headache to keep them up-to-date. :)
From now, patches for 1.1-stable are maintained separately, no improvements will go for them.

#210 Updated by Toshi MARUYAMA over 3 years ago

I create new issue #7597.

#211 Updated by Yuya Nishihara over 3 years ago

I want to rewrite redminehelper.py to use XML instead of original format.

Now redminehelper.py generates XML and updated mercurial_adapter.rb to use ActiveSupport::XmlMini:
https://bitbucket.org/yuja/redmine-mq-issue4455/

Should I move redminehelper.py under lib/redmine/scm/adapters/mercurial?

#212 Updated by Toshi MARUYAMA over 3 years ago

Yuya Nishihara wrote:

Should I move redminehelper.py under lib/redmine/scm/adapters/mercurial?

I think note-208 is much better.

#213 Updated by Yuya Nishihara over 3 years ago

Toshi MARUYAMA wrote:

I think note-208 is much better.

of which?

lib/redmine/scm/adapters/mercurial/ext/redminehelper.py
or
lib/redmine/scm/adapters/mercurial/redminehelper.py ?

#214 Updated by Toshi MARUYAMA over 3 years ago

Yuya Nishihara wrote:

of which?

lib/redmine/scm/adapters/mercurial/ext/redminehelper.py
or
lib/redmine/scm/adapters/mercurial/redminehelper.py ?

lib/redmine/scm/adapters/mercurial/redminehelper.py

#215 Updated by Toshi MARUYAMA over 3 years ago

I create new issue #7618.

#216 Updated by Yuya Nishihara over 3 years ago

Toshi MARUYAMA wrote:

lib/redmine/scm/adapters/mercurial/redminehelper.py

Thanks for clarify. Moved it under adapters/mercurial directory.

#217 Updated by Toshi MARUYAMA over 3 years ago

I committed 'hg' method in r4830.
Does "--cwd" work fine in Windows UNC path such as "\\server\folder"?

source:trunk/lib/redmine/scm/adapters/mercurial_adapter.rb@4830#L234

#218 Updated by Yuya Nishihara over 3 years ago

Toshi MARUYAMA wrote:

Does "--cwd" work fine in Windows UNC path such as "\\server\folder"?

It works for me, on Mercurial 1.7.2. What error you've got on log?

#220 Updated by Toshi MARUYAMA over 3 years ago

Redmine 1.1 uses only one "--cwd".
source:tags/1.1.1/lib/redmine/scm/adapters/mercurial_adapter.rb#L83

In general, changing directory is very dangerous.

#221 Updated by Ammler _ over 3 years ago

why is --cwd needed at all since you don't work with working copies at all

#222 Updated by Yuya Nishihara over 3 years ago

Toshi MARUYAMA wrote:

TortoiseHg shellext checks UNC or not.

Hmm, but I don't think -R accepts UNC but --cwd doesn't. Isn't it strange?

In general, changing directory is very dangerous.

by its context. It's pretty safe to change the working directory of child 'hg' process.
There's no threading issue.

Marcel Gmür wrote:

why is --cwd needed at all

To avoid concatenation of repo filesystem path and inner-repo path:

$ hg -R /path/to/repo log /path/to/repo/foo
...
$ hg --cwd /path/to/repo log foo
...

#223 Updated by Toshi MARUYAMA over 3 years ago

I changed "--cwd" to "-R", and all adapter lib tests passes.
The remaining work is that latest_changesets supports tags and branches.

I start to hack #2664.
Non UTF-8 sub directory (EUC-JP) can not show latest_changesets.
I don't know the reason.

#224 Updated by Toshi MARUYAMA over 3 years ago

As I wrote in r5091 comment, MySQL test on CI server fails.

http://www.redmine.org/builds/index.html

  1) Failure:
test_latest_changesets(RepositoryMercurialTest) [/test/unit/repository_mercurial_test.rb:90]:
<["28", "27"]> expected but was
<["0", "1"]>.

  2) Failure:
test_latest_changesets_with_limit(RepositoryMercurialTest) [/test/unit/repository_mercurial_test.rb:198]:
<[#<Changeset id: 797, repository_id: 64, revision: "28", committer: "test Ü <test00@foo.bar>", committed_on: "2007-10-31 17:00:00", comments: "commit default branch.", commit_date: "2007-10-31", scmid: "3ae45e2d177d", user_id: nil>,
 #<Changeset id: 796, repository_id: 64, revision: "27", committer: "jsmith <jsmith@foo.bar>", committed_on: "1980-12-31 17:00:00", comments: "copy latin-1 path files to latin-1 subdir.", commit_date: "1980-12-31", scmid: "7bbf4c738e71", user_id: 2>]> expected but was
<[#<Changeset id: 769, repository_id: 64, revision: "0", committer: "jsmith <jsmith@foo.bar>", committed_on: "2007-12-14 10:22:52", comments: "Initial import.\nThe repository contains 3 files.", commit_date: "2007-12-14", scmid: "0885933ad4f6", user_id: 2>,
 #<Changeset id: 770, repository_id: 64, revision: "1", committer: "jsmith <jsmith@foo.bar>", committed_on: "2007-12-14 10:24:01", comments: "Added 2 files and modified one.", commit_date: "2007-12-14", scmid: "9d5b5b004199", user_id: 2>]>.

#225 Updated by Jean-Philippe Lang over 3 years ago

  • Status changed from New to Closed

I'm closing this ticket as proposed above. The development forum may be a better place to continue this discussion.

#226 Updated by Toshi MARUYAMA over 3 years ago

  • Target version deleted (Unplanned)

Also available in: Atom PDF