Feature #37807


Allow access to /robots.txt even if logins are required

Added by Holger Just over 1 year ago. Updated over 1 year ago.

Permissions and roles
Target version:
Start date:
Due date:
% Done:


Estimated time:


Right now, if logins are globally required, the /robots.txt path is not accessible for search engines since the Welcome#robots path also observes this setting. Requests to /robots.txt will thus receive an empty HTTP 401 in this case today.

The attached patch series improves this behavior in multiple ways. The patches were extracted from Planio.

  • 0001-Render-all-visible-projects-in-robots.txt-including-.patch - While initially not strictly related to the issue, this patch extends the list of projects included in the /robots.txt to not only list active projects but also closed projects (which are still visible to anonymous). This ensures that the list is correct for cases where the Redmine does not enforce logins. Still, as the /robots.txt file is intended to be consumed by search engines rather than logged-in users, we only list projects which are visible to Anonymous now. The /robots.txt output thus does not distinguish between the project visibility of the current user but will only output projects visible to Anonymous.
  • 0002-Always-allow-access-to-robots.txt-for-Anonymous.patch - This patch allows Anonymous to always access /robots.txt, regardless of the Setting.login_required setting. Previously, this would have been denied if logins are required.
  • 0003-Disallow-all-in-robots.txt-if-login-is-required.patch - With a required login, Anonymous should not be able to view ANY project information. Even in case some routes are manually excluded from this restriction, we still don't want those to be index by search engines. As such, with required logins, we just instruct all robots to not index anything. This patch also makes sure that we are not leaking any information about public projects in case logins are required. Before this patch (but after the previous patches), we would include a list of all public projects there, even if Anonymous could not see them without a login.


Actions #1

Updated by Go MAEDA over 1 year ago

  • Target version set to Candidate for next major release
Actions #2

Updated by Go MAEDA over 1 year ago

  • Target version changed from Candidate for next major release to 5.1.0

Setting the target version to 5.1.0.

Actions #3

Updated by Go MAEDA over 1 year ago

  • Tracker changed from Patch to Feature
  • Status changed from New to Closed
  • Assignee set to Go MAEDA
  • Resolution set to Fixed

Committed the patches. Thank you for your contribution.


Also available in: Atom PDF