Added by George Notaras over 6 years ago
When robots.txt is requested, it is returned with the content type
text/html instead of
text/plain. Is this the case in your installations? Could you please test?
According to my experience this results in the robots.txt file being ignored by the vast majority of indexing bots, even the most popular ones, and thus crawlers index sections that should normally be excluded. This also results in significant increase of the server load.