https://www.redmine.org/
https://www.redmine.org/favicon.ico?1679302129
2017-12-28T20:55:41Z
Redmine
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82799
2017-12-28T20:55:41Z
Toshi MARUYAMA
<ul><li><strong>File</strong> <a href="/attachments/19805">robots.diff</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/19805/robots.diff">robots.diff</a> added</li><li><strong>Status</strong> changed from <i>New</i> to <i>Confirmed</i></li><li><strong>Target version</strong> set to <i>4.0.0</i></li></ul><p>Try this patch.</p>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82800
2017-12-28T21:28:12Z
Grischa Zengel
<ul></ul><p>I tested your patch and it works like expected.<br />It's clever to use the existing functions instead of assembling some string.<br />Thanks.</p>
<p>I got:</p>
<pre>
User-agent: *
Disallow: /redmine/projects/support/repository
Disallow: /redmine/projects/support/issues
Disallow: /redmine/projects/support/activity
Disallow: /redmine/issues/gantt
Disallow: /redmine/issues/calendar
Disallow: /redmine/activity
Disallow: /redmine/search
</pre>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82807
2017-12-29T12:08:03Z
Toshi MARUYAMA
<ul><li><strong>Status</strong> changed from <i>Confirmed</i> to <i>New</i></li><li><strong>Target version</strong> deleted (<del><i>4.0.0</i></del>)</li></ul><p><a class="external" href="https://webmasters.stackexchange.com/questions/89395/robots-txt-should-be-in-the-root-directory-or-can-be-in-sub-directory">https://webmasters.stackexchange.com/questions/89395/robots-txt-should-be-in-the-root-directory-or-can-be-in-sub-directory</a></p>
<blockquote>
<p>web crawlers will not read or obey a robots.txt file in a subdirectory.<br /><a class="external" href="http://www.robotstxt.org/robotstxt.html">http://www.robotstxt.org/robotstxt.html</a></p>
</blockquote>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82808
2017-12-29T12:24:15Z
Grischa Zengel
<ul></ul><p>Sure?</p>
<blockquote>
<p>RedirectMatch permanent ^/robots.txt$ /redmine/robots.txt</p>
</blockquote>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82809
2017-12-29T12:25:42Z
Grischa Zengel
<ul></ul><p>I would like to post my apache.log which show that all bots redirect properly, but I've been blocked always because of spam.</p>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82810
2017-12-29T12:29:13Z
Grischa Zengel
<ul></ul><pre>
redmine.:443 88.198.55.175 - - [29/Dec/2017:08:10:23 +0000] "GET /robots.txt HTTP/1.1" 301 3962 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8;)"
redmine.:443 88.198.55.175 - - [29/Dec/2017:08:10:23 +0000] "GET /redmine/robots.txt HTTP/1.1" 200 4126 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8;)"
redmine.:443 66.249.76.118 - - [29/Dec/2017:09:48:48 +0000] "GET /robots.txt HTTP/1.1" 301 3962 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;)"
redmine.:443 66.249.76.118 - - [29/Dec/2017:09:48:48 +0000] "GET /redmine/robots.txt HTTP/1.1" 200 841 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;)"
redmine.:443 5.255.251.125 - - [29/Dec/2017:10:51:21 +0000] "GET /robots.txt HTTP/1.1" 301 3967 "-" "Mozilla/5.0 (compatible; YandexBot/3.0;)"
redmine.:443 87.250.233.120 - - [29/Dec/2017:10:51:21 +0000] "GET /redmine/robots.txt HTTP/1.1" 200 4187 "-" "Mozilla/5.0 (compatible; YandexBot/3.0;)"
</pre>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82811
2017-12-29T12:44:59Z
Grischa Zengel
<ul></ul><p>Some more:</p>
<pre>
redmine.:443 54.36.150.157 - - [28/Dec/2017:05:03:01 +0000] "GET /robots.txt HTTP/1.1" 301 3711 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2;)"
redmine.:443 54.36.150.157 - - [28/Dec/2017:05:03:02 +0000] "GET /redmine/robots.txt HTTP/1.1" 200 919 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2;)"
redmine.:443 157.55.39.25 - - [29/Dec/2017:01:35:11 +0000] "GET /robots.txt HTTP/1.1" 301 3942 "-" "Mozilla/5.0 (compatible; bingbot/2.0;)"
redmine.:443 157.55.39.25 - - [29/Dec/2017:01:35:12 +0000] "GET /redmine/robots.txt HTTP/1.1" 200 4012 "-" "Mozilla/5.0 (compatible; bingbot/2.0;)"
redmine.:443 194.187.170.123 - - [21/Dec/2017:22:26:10 +0000] "GET /robots.txt HTTP/1.0" 301 3825 "-" "Mozilla/5.0 (compatible; Qwantify/2.4w;)/2.4w"
redmine.:443 194.187.170.123 - - [21/Dec/2017:22:26:10 +0000] "GET /redmine/robots.txt HTTP/1.0" 200 1072 "-" "Mozilla/5.0 (compatible; Qwantify/2.4w;)/2.4w"
redmine.:443 18.195.89.56 - - [26/Dec/2017:00:37:01 +0000] "GET /robots.txt HTTP/1.1" 301 3711 "-" "Mozilla/5.0 (compatible; Cliqzbot/2.0;)"
redmine.:443 18.195.89.56 - - [26/Dec/2017:00:37:04 +0000] "GET /redmine/robots.txt HTTP/1.1" 200 782 "-" "Mozilla/5.0 (compatible; Cliqzbot/2.0;)"
</pre>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82813
2017-12-29T12:51:41Z
Toshi MARUYAMA
<ul></ul><p>Grischa Zengel wrote:</p>
<blockquote>
<p>Sure?</p>
<blockquote>
<p>RedirectMatch permanent ^/robots.txt$ /redmine/robots.txt</p>
</blockquote></blockquote>
<p>How will you do if your web site has plural subdirectories?</p>
E.g.:
<ul>
<li>example.com/redmine1</li>
<li>example.com/redmine2</li>
</ul>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82814
2017-12-29T12:58:50Z
Grischa Zengel
<ul></ul><p>For better compatibility I will change to:</p>
<p>$ cat /etc/cron.hourly/robots</p>
<p>#!/bin/sh<br />wget <a class="external" href="https://redmine/redmine/robots.txt">https://redmine/redmine/robots.txt</a> -O /var/www/html/robots.txt</p>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82815
2017-12-29T13:01:05Z
Grischa Zengel
<ul></ul><blockquote>
<p>How will you do if your web site has plural subdirectories?</p>
</blockquote>
<p>Than you have to use the cron solution and concatenate the results.<br />But how many servers will host more than one redmine instance? 1%?</p>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82816
2017-12-29T13:16:28Z
Toshi MARUYAMA
<ul></ul><p>Grischa Zengel wrote:</p>
<blockquote><blockquote>
<p>How will you do if your web site has plural subdirectories?</p>
</blockquote>
<p>Than you have to use the cron solution and concatenate the results.</p>
</blockquote>
<pre>
$ wget http://localhost:3100/test1/robots.txt -O - > ~/Desktop/robots.txt
$ wget http://localhost:3100/test2/robots.txt -O - | grep '^Disallow:' >> ~/Desktop/robots.txt
</pre>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82817
2017-12-29T13:29:31Z
Grischa Zengel
<ul></ul><p>It doesn't matter how you post process the robots.txt from subdirectories, redmine has to generate a valid robots.txt and your patch works.<br />So please add it to next sub release, so I don't have to remember to patch manually. It doesn't break anything.</p>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=82818
2017-12-29T14:59:07Z
Toshi MARUYAMA
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li><li><strong>Target version</strong> set to <i>4.0.0</i></li><li><strong>Resolution</strong> set to <i>Fixed</i></li></ul><p>I have committed in <a class="changeset" title="use relative url at robots.txt (#27865)" href="https://www.redmine.org/projects/redmine/repository/svn/revisions/17135">r17135</a>.</p>
<p>Grischa Zengel wrote:</p>
<blockquote>
<p>So please add it to next sub release</p>
</blockquote>
<p>I don't want to change behaviour in sub version.</p>
Redmine - Defect #27865: RailsBaseURI ignored while creating robots.txt
https://www.redmine.org/issues/27865?journal_id=87387
2018-09-17T13:39:26Z
Go MAEDA
<ul><li><strong>Category</strong> set to <i>SEO</i></li></ul>