Defect #10887

Can't fetch Mercurial changesets from repository

Added by Jens Bäckman over 10 years ago. Updated about 5 years ago.

Status:NewStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:SCM
Target version:-
Resolution: Affected version:1.4.1

Description

I get this backtrace when trying to view a repository:

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:206:in `each_revision'
  app/models/repository/mercurial.rb:141:in `fetch_changesets'
  app/models/repository/mercurial.rb:140:in `step'
  app/models/repository/mercurial.rb:140:in `fetch_changesets'
  app/controllers/repositories_controller.rb:91:in `show'
  /usr/lib/ruby/1.8/rack/adapter/rails.rb:74:in `call'
  /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process'
  /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch'
  /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process'
  /usr/lib/ruby/1.8/thin/connection.rb:57:in `process'
  /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data'
  /usr/lib/ruby/1.8/eventmachine.rb:257:in `run_machine'
  /usr/lib/ruby/1.8/eventmachine.rb:257:in `run'
  /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start'
  /usr/lib/ruby/1.8/thin/server.rb:156:in `start'
  /usr/lib/ruby/1.8/thin/controllers/controller.rb:80:in `start'
  /usr/lib/ruby/1.8/thin/runner.rb:174:in `send'
  /usr/lib/ruby/1.8/thin/runner.rb:174:in `run_command'
  /usr/lib/ruby/1.8/thin/runner.rb:140:in `run!'
  /usr/bin/thin1.8:6

The repository in question has over 3000 changesets.

Output from script/about:

About your application's environment
Ruby version              1.8.7 (x86_64-linux)
RubyGems version          1.3.7
Rack version              1.1.3
Rails version             2.3.14
Active Record version     2.3.14
Active Resource version   2.3.14
Action Mailer version     2.3.14
Active Support version    2.3.14
Application root          /var/lib/redmine
Environment               production
Database adapter          mysql
Database schema version   20120301153455

Running Redmine 1.4-stable -> thin -> nginx.

History

#1 Updated by Vasiliy Marchuk over 10 years ago

I get this too:

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:206:in `each_revision'
  app/models/repository/mercurial.rb:141:in `fetch_changesets'
  app/models/repository/mercurial.rb:140:in `step'
  app/models/repository/mercurial.rb:140:in `fetch_changesets'
  app/controllers/repositories_controller.rb:114:in `show'

Repository only has 14 commits:

@  14[tip]   2272237fef14   2012-03-11 20:34 +0200   styx
|    Fixed sending signature to Qbix main site from reports script
|
o  13   3f3b5c39359b   2012-02-17 02:36 +0200   styx
|    Reports cron script
|
o  12   f79cf76626fe   2011-07-30 12:01 +0300   styx
|    disable sql-dump into errors.log
|
o  11   3297edd6c8dc   2011-07-30 11:58 +0300   styx
|    little fix
|
o  10   fdeb01db8881   2011-07-30 11:48 +0300   styx
|    support for 1.3 app version, added BundleVersion and SystemVersion into Pings' table
|
o  9   526ab3d6615f   2011-04-14 01:37 -0400   gregory
|    redirecting to qbix.com/calendar now
|
o  8   a11db5f4d074   2011-04-13 04:38 -0400   gregory
|    chmod 770
|
o  7   aecda1a96b7a   2011-03-10 18:06 +0200   styx
|    Support for ping's messages
|
o    6:5,4   4e597fb30296   2011-03-07 20:02 +0200   styx
|\     Merge with main
| |
| o  5:3   556e586c1c6d   2011-03-06 17:55 +0200   styx
| |    Web side for Calendar pings
| |
o |  4   e879e2095c7d   2011-03-06 13:55 -0500   greg
|/     testing if we can commit
|
o  3   8f001bc4f6d9   2011-03-03 06:07 -0500   gregory
|    changed route
|
o  2   d46010c6b2ec   2011-03-03 06:06 -0500   gregory
|    renamed GroupsApi to CalendarApi
|
o  1   f893d8a34085   2011-03-03 06:06 -0500   gregory
|    added support for pings
|
o  0   dddfadc3157b   2011-03-03 05:53 -0500   gregory
     calendar app

Running Redmine 2.0.0.stable -> thin -> nginx

script/about:

Environment:
  Redmine version                          2.0.0.stable
  Ruby version                             1.8.7 (i386-linux)
  Rails version                            3.2.3
  Environment                              production
  Database adapter                         MySQL
Redmine plugins:
  no plugin installed

Mercurial 2.0

#2 Updated by Anonymous about 10 years ago

I also get this issue, with a repository with almost 18000 commits. On the web interface, it shows all commits up to and including 14199.

Environment:
Redmine version                          2.0.3.stable
Ruby version                             1.9.3 (x86_64-linux)
Rails version                            3.2.6
Environment                              production
Database adapter                         Mysql2
Redmine plugins:
redmine_auto_percent                     0.0.1
redmine_favourite_projects               0.5
redmine_latex_mathjax                    0.1.0
redmine_openid_selector                  0.0.1
redmine_recaptcha                        0.1.0

Mercurial 2.2.2

#3 Updated by Anonymous about 10 years ago

Actually, I should clarify: I get an issue that looks very similar, but I am not 100% sure if it is the same. Namely, when I run ./script/rails runner "Repository.fetch_changesets" -e production, I get the following error (with some paths slightly "anonymized"):

/PATH-USER/gems/railties-3.2.6/lib/rails/commands/runner.rb:53:in `eval': undefined method `[]' for nil:NilClass (NoMethodError)
    from /PATH-WWW/app/models/repository/mercurial.rb:141:in `block in fetch_changesets'
    from /PATH-WWW/app/models/repository/mercurial.rb:140:in `step'
    from /PATH-WWW/app/models/repository/mercurial.rb:140:in `fetch_changesets'
    from /PATH-WWW/app/models/repository.rb:306:in `block (2 levels) in fetch_changesets'
    from /PATH-USER/gems/activerecord-3.2.6/lib/active_record/associations/collection_proxy.rb:89:in `each'
    from /PATH-USER/gems/activerecord-3.2.6/lib/active_record/associations/collection_proxy.rb:89:in `method_missing'
    from /PATH-WWW/app/models/repository.rb:304:in `block in fetch_changesets'
    from /PATH-WWW/app/models/repository.rb:303:in `each'
    from /PATH-WWW/app/models/repository.rb:303:in `fetch_changesets'
    from (eval):1:in `<top (required)>'
    from /PATH-USER/gems/railties-3.2.6/lib/rails/commands/runner.rb:53:in `eval'
    from /PATH-USER/gems/railties-3.2.6/lib/rails/commands/runner.rb:53:in `<top (required)>'
    from /PATH-USER/gems/railties-3.2.6/lib/rails/commands.rb:64:in `require'
    from /PATH-USER/gems/railties-3.2.6/lib/rails/commands.rb:64:in `<top (required)>'
    from ./script/rails:6:in `require'
    from ./script/rails:6:in `<main>'

#4 Updated by Vadim Kotov almost 10 years ago

Got the same error with Redmine 2.0.4 and Mercurial 2.3.1. Deleting repo in the project's settings and adding it again did not help. Any ideas why is this happening? SCM bindings are completely unusable due to this error.

#5 Updated by Vadim Kotov almost 10 years ago

Ok, I've done a small research and got this thing fixed. In my case there were additional lines from Mercurial in the hg log output, reporting that repo's hgrc file is not trusting (due to permission issue). Thats why Redmine failed to parse that output and changeset collection was nil.
I've added hgrc owner to the trusted section, and changesets were parsed successfully (refer to Mercurial Trust). Also, this issue affects 2.x Redmine branch. I guess there should be a fallback if there are some wrong lines in the log.
Hope this info could help someone!

#6 Updated by Anonymous over 9 years ago

While I am also seeing these messages about hg not trusting some hgrc file, this proved to be a red herring in my case -- I successfully turned off the warning, but it didn't help.

But I finally figured out that the problem was a specific commit that redmine choked on. First, to figure out what commit was causing the problem, I modified app/models/repository/mercurial.rb by changing FETCH_AT_ONCE from 100 to 1 -- this is slower, but this way, I was able to pinpoint the precise commit redmine was choking on.

Then I had a closer look. At first I was stumped, but then I changed the XML parser being used to libxml2, which seems to be more strict / give better diagnostics. This required installing the libxml-ruby gem, and then editing lib/redmine/scm/adapters/abstract_adapter.rb where I inserted the line

ActiveSupport::XmlMini.backend = 'LibXML'

right before
ActiveSupport::XmlMini.parse(xml)

After that, I triggered the repository import from the command line, as described in the Redmine user guide. I.e. via the command
./script/rails runner "Repository.fetch_changesets" -e production

This time it died with the message
Fatal error: PCDATA invalid Char value 24 at :10.

Value 24 is a non-printing ASCII character, which somehow snuck into the commit message of the offending commit. (I checked this by piping the output of "hg log" for that commit into a file and then inspecting this with a hex editor). In my case, the one-time solution thus was the following: I edited lib/redmine/scm/adapters/mercurial_adapter.rb and inserted
            output.delete!("\x18")

right before this (which is at line 201 in Redmine 2.2.3; beware, there are more parse_xml call in that file):
           begin
             # Mercurial < 1.5 does not support footer template for '</log>'
             parse_xml("#{output}</log>")['log']
           rescue
           end

In words: I told Redmine to throw away any occurrences of the bad character.

My guess is that for other people, the problem might also be caused by problematic characters in the log message. It might be a different character, though, but I hope that my instructions will at least help some people with this problem.

Of course it would be nice if there was a "proper" fix for this. I imagine this could be similar to what I did hear, but should probably filter out more bad characters... or perhaps hg or the Redmine mercurial imported could be modified to generate XML using CDATA, instead of relying on the data to be PCDATA...

#7 Updated by Ian German Mesner over 9 years ago

I can report this is still occurring due to certain characters, and I used Max Horn's solution.

I am unfamiliar with the redmine codebase and ruby, but I can see a bug with the xml parsing in mercurial_adapter.rb. Basically, any exception thrown by parse_xml is ignored, with nothing logged. It seems to me that an empty object should be returned (or a check for nil afterwards), and the offending scm log entry be logged by redmine. That way, the entire repository fetch won't be in error if a single scm log entry is corrupt (in my case an accidental DLE character was inserted). In addition, it would be nice to strip offending characters out of the log message.

195           log = hg(*hg_args) do |io|
196             output = io.read
197             if output.respond_to?(:force_encoding)
198               output.force_encoding('UTF-8')
199             end
200             begin
201               # Mercurial < 1.5 does not support footer template for '</log>'
202               parse_xml("#{output}</log>")['log']
203             rescue
204             end
205           end

#8 Updated by Marc Schlaich almost 9 years ago

Seeing a similar issue

hg: error during getting info: undefined method `[]' for nil:NilClass
Completed 500 Internal Server Error in 310ms

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:161:in `entries'
  app/models/repository.rb:192:in `entries'
  app/controllers/repositories_controller.rb:116:in `show'

#9 Updated by Egor Dolgalev almost 9 years ago

I have the same issue like Marc Schlaich

hg: error during getting info: undefined method `[]' for nil:NilClass
Completed 500 Internal Server Error in 310ms

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:161:in `entries'
  app/models/repository.rb:192:in `entries'
  app/controllers/repositories_controller.rb:116:in `show'

I choose debug log level and get:

Shelling out: "hg" "-R" "d:\tmp\1\test-branch\" "--encoding" "utf-8" "--config" "extensions.redminehelper=e:/redmine-2.4.1/lib/redmine/scm/adapters/mercurial/redminehelper.py" "--config" "diff.git=false" "rhmanifest" "-r" "tip" "" 
Completed 500 Internal Server Error in 397.0ms

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:161:in `entries'
  app/models/repository.rb:192:in `entries'
  app/controllers/repositories_controller.rb:116:in `show'

Then I tried to run above command manually and get next log:

<?xml version="1.0"?>
<rhmanifest>
<repository root="d%3A%5Ctmp%5C1%5Ctest-branch">
<manifest revision="23" path="">
<file name=".hgtags" revision="22" node="dbecb64f3572" time="1379692530" size="224"/>
<file name="file.txt" revision="20" node="d6dc83e4d9a8" time="1379692454" size="101"/>
</manifest>
</repository>
</rhmanifest>

#10 Updated by Jérôme Mestres about 5 years ago

Late response, but if I can help...

Same issue for me :

NoMethodError (undefined method `[]' for nil:NilClass):
  lib/redmine/scm/adapters/mercurial_adapter.rb:161:in `entries'
  app/models/repository.rb:192:in `entries'
  app/controllers/repositories_controller.rb:116:in `show'

Watch in "lib/redmine/scm/adapters/mercurial_adapter.rb:194" and I tried to execute manually the command at the top of methode :

hg log --debug -C --style /redmine/lib/redmine/scm/adapters/mercurial/hg-template-1.0.tmpl /repo/my-repo > output.log

The file is expect to be XML, but on first line (before <xml>) I have this :

ignoring untrusted configuration option hooks.changegroup [...]

So redmine, can not parse the output as XML.

Change my hgrc file to disable the hook, and It's work !!

I this can help...

Also available in: Atom PDF