tree 2a02147bf42a94cd2c6b67c1a83ae209a3f41c25
parent 23993b6d7a6cfcc85238bc6ba3170cd55fa88289
author Marc Kupietz <kupietz@ids-mannheim.de> 1787302065 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1787302065 +0200

Turn away the crawlers that collect training data

Every URL with parameters is a result computed for one word, there is one
for every word of the vocabulary, and none of it can be answered from a
cache, so a crawler that follows them keeps the machine busy for nothing.
The instance on corpora.ids-mannheim.de answers about 21000 such requests
a day.

Requests whose User-Agent matches a list of such crawlers now get 403 in
before_dispatch, i.e. before any computation. The list can be replaced in
the configuration file, robots => {block_user_agents => '...'}, and an
empty value switches the blocking off. Ordinary search engines are not on
it.

For the crawlers that do read robots.txt, every response to a request
with parameters carries X-Robots-Tag: noindex, nofollow, result pages
carry the matching meta element, and robots.txt is served at every path -
which only helps where derekovecs owns its host name, a proxy that mounts
it below a path still has to serve the site wide one itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Id68487ef716459e0ea9f045060edc34edf85065e