[Solved] Can't login suddenly

Added by Carsten Schurig almost 3 years ago

Hi,

I have a strange problem: suddenly I can't login to my redmine instance anymore (or anyone else). I go to the login page enter my credentials and then I get:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, xxx and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.
Apache/2.2.14 (Ubuntu) Server at xxx Port 443

I'm a bit out of ideas now. The log files, redmines and apaches, don't show anything sensible! Everything looks fine there even if I turn on highest debug levels...

The only strange message is

 Giving :session_key to SessionStore is deprecated, please use :key instead.

but this I got even when it worked all right.

For users still logged in everything continues to work fine but once you logged out, you're out of luck.

Does anyone have an idea where to look?

Thanks,
Carsten

Replies (18)

RE: Can't login suddenly - Added by Carsten Schurig almost 3 years ago

Ok, found it. I updated the server running redmine to subversion 1.7 and that's the problem...

RE: Can't login suddenly - Added by Mischa The Evil almost 3 years ago

Carsten Schurig wrote:

Ok, found it. I updated the server running redmine to subversion 1.7 and that's the problem...

Can you please explain more precise what issues are caused by your switch to Subversion 1.7? I don't see any correlation between the Subversion update and the fact that users can't login to Redmine suddenly without any info written to the logs about it. Redmine uses its own authentication system thus there shouldn't be any interference with 3rd party software...

RE: Can't login suddenly - Added by Carsten Schurig almost 3 years ago

Actually I don't understand this as well. But it is the symptoms:

  • updated the server running redmine to SVN 1.7 => fail after login page without a sensible error message
  • reinstalled SVN 1.6 => redmine works again

And SVN 1.7 did work ok on the server as server and as client.

And it's not the authentication as it seems. Access to the mysql database seems not to be the problem.

One thing that might happen: redmine checks for some versions on start. In lib/redmine/version.rb e.g. an existing .svn directory is accessed directly and parsed for the revision (which is not so good btw, as it is heavily dependent on WC format). But actually this code doesn't look as it could break fatally on WC format change, it would just return a wrong revision.

So I suspect there is a svn command called somewhere in the startup, maybe on loading plugins, that fails quite silently. Unfortunately I didn't find it, as I'm not familiar with RoR code...

RE: Can't login suddenly - Added by Etienne Massip almost 3 years ago

Can you try with no plugin?

RE: Can't login suddenly - Added by Eric BELROSE over 2 years ago

I have the same issue since I've updated to SVN 1.7
Something came up ?

RE: Can't login suddenly - Added by Eric BELROSE over 2 years ago

Completing my previous post.

Since the update every POST request return an 500 internal error from Apache (not the Redmine one)
Nothing in production.log, nothing in Apache vhost or global error.log
Just one line in the vhost access.log saying that Apache returned 500 for the POST request.

RE: Can't login suddenly - Added by Matthias Fechner over 2 years ago

Same problem for me. Please see here my post:
http://www.redmine.org/boards/2/topics/26984

RE: Can't login suddenly - Added by Eric BELROSE over 2 years ago

Hi,
I confirm this is related with SVN 1.7 or more exactly the dav_svn module for apache
a2dismod dav_svn
and you get redmine working just fine !

RE: Can't login suddenly - Added by Matthias Fechner over 2 years ago

I had to remove the following two lines from the apache config:
LoadModule dav_svn_module libexec/apache22/mod_dav_svn.so
LoadModule authz_svn_module libexec/apache22/mod_authz_svn.so

That a least push redmine back into a working status.
Thanks a lot for this tip.

Is this a bug in subversion so we should create there a bug for this?

RE: Can't login suddenly - Added by Mischa The Evil over 2 years ago

Please take a look at #9517. Its solution might be related with this. Summarizing the issue: svn => 1.7 doesn't play nice with Phusion Passenger.

RE: Can't login suddenly - Added by Eric BELROSE over 2 years ago

Not realy the same thing there...
Before I had svn on every /svn location configured directly in mod dav_svn.conf file
So I tried to have a seperate vhost just with the svn conf that was in dav_svn.conf
Deleted dav_svn.conf and restarting Apache.
SVN work fine on the dedicated vhost but I'm still having the internal server error with Redmine
I can't access SVN from my Redmine vhost anymore as I seperated it but Redmine is still not working !
So it's not a vhost issue it's definitely a compatibility issue between redmine (passenger ?) and svn mod for apache

RE: Can't login suddenly - Added by Mischa The Evil over 2 years ago

Because Subversion 1.7.x compatibility seems persistently broken (this thread, #9517 and #9541) I've taken some time to do some research on these issues.

The culprit causing the issues as described here and in #9541 can be traced-back to a defect in the mod_dav_svn Apache module provided by Subversion 1.7.0 and 1.7.1. It was initially reported on the Subversion users mailinglist on the 18th of october, by Jean-Daniel Dupas. After some discussion the issue was confirmed as:

mod_dav_svn.c:dav_svn__handler passes all POST requests to dav_svn__method_post and that function will return an error if the request does not have SVNPath or SVNParentPath set since it will fail to open a repository.

AFAICS this defect affects all 1.7.x releases at the moment. A fix for the defect is applied (as mentioned in mailinglist message from committer Philip Martin) in Subversion trunk r11876951 on 2011-10-22T11:32:38.465109Z which is proposed for inclusion in Subversion 1.7.x (1.7.2 as soonest depending on their release-policy, see http://svn.apache.org/repos/asf/subversion/branches/1.7.x/STATUS).

So, summarizing this all: the mentioned issues with Subversion 1.7.x on Redmine are related to issues in Subversion 1.7.x itself. So, no issues here in Redmine nor in Phusion Passenger.
As a hotfix I can suggest to:
  • disable the mod_dav_svn Apache module until Subversion merged the fix in the 1.7-branch, included it in an upcoming 1.7.x-release and you've upgraded to this new release
  • downgrade back to Subversion 1.6.x until the 1.7.x-branch has become more stable
  • build Subversion from (trunk-)source yourself

Note: #9517 seems to be an exception to the above. Though, it can be "solved" easily if someone is suffering the same issue. I've tried to reproduce it anyways but wasn't able to do so, so it seems to be very depending on the configured environment at least also.

Note to self: add some info about these upstream SCM issues to Redmine documentation (RedmineInstall and/or RedmineRepositories)

1 commit-message: "mod_dav_svn should not raise an error on POST requests outside a
Subversion location as these are not directed at Subversion."

RE: Can't login suddenly - Added by Etienne Massip over 2 years ago

Woot. Very nice job!

RE: Can't login suddenly - Added by Mischa The Evil over 2 years ago

Mischa The Evil wrote:

Note to self: add some info about these upstream SCM issues to Redmine documentation (RedmineInstall and/or RedmineRepositories)

Done, added it to where it is already: RedmineRepositories.

Etienne Massip wrote:

Woot. Very nice job!

Thanks... The pleasure is all mine... ;)

RE: Can't login suddenly - Added by Terence Mill over 2 years ago

Is there a issue in a bug tracker i can track at which svn version this is fixed?
Can't find "r1187695" in status site till now.

RE: Can't login suddenly - Added by Eric BELROSE over 2 years ago

Seems to be fixed with yesterday 1.7.2 release
Changelog : * make mod_dav_svn ignore non-Subversion POST requests (r1187695)

RE: Can't login suddenly - Added by Carsten Schurig over 2 years ago

Well, I've just been bold and installed svn 1.7.2 on the server and lukily it does seem to work as expected...

RE: Can't login suddenly - Added by Mischa The Evil over 2 years ago

Terence Mill wrote:

Can't find "r1187695" in status site till now.

Terence, in the Subversion process that means that the commit(s) were merged into the release-branch; from that moment the changes should show up on the changelog (http://svn.apache.org/repos/asf/subversion/branches/1.7.x/CHANGES).

@ all: this is indeed solved in Subversion 1.7.2 (release announcement: http://svn.haxx.se/dev/archive-2011-12/0025.shtml).
I'll modify the topic subject to indicate the status, since this topic is referenced in issue #9541.

(1-18/18)