Turn away the crawlers that collect training data
Every URL with parameters is a result computed for one word, there is one
for every word of the vocabulary, and none of it can be answered from a
cache, so a crawler that follows them keeps the machine busy for nothing.
The instance on corpora.ids-mannheim.de answers about 21000 such requests
a day.
Requests whose User-Agent matches a list of such crawlers now get 403 in
before_dispatch, i.e. before any computation. The list can be replaced in
the configuration file, robots => {block_user_agents => '...'}, and an
empty value switches the blocking off. Ordinary search engines are not on
it.
For the crawlers that do read robots.txt, every response to a request
with parameters carries X-Robots-Tag: noindex, nofollow, result pages
carry the matching meta element, and robots.txt is served at every path -
which only helps where derekovecs owns its host name, a proxy that mounts
it below a path still has to serve the site wide one itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Id68487ef716459e0ea9f045060edc34edf85065e
diff --git a/example.conf b/example.conf
index 34d6297..e999b3d 100644
--- a/example.conf
+++ b/example.conf
@@ -21,6 +21,14 @@
profiles_max_keys => 200 # similar profiles (JSON)
},
+ # Crawlers that collect training data for language models are answered with
+ # 403 before anything is computed. The value is a regular expression that is
+ # matched against the User-Agent header; the built in list is in
+ # script/derekovecs-server. An empty string switches the blocking off.
+ # robots => {
+ # block_user_agents => 'GPTBot|ClaudeBot|Meta-ExternalAgent|CCBot'
+ # },
+
w2v => {
vecs => "example-models/wpd19_10000/wpd19_10000.vecs",
# compare_to => "https://corpora.ids-mannheim.de/openlab/derekovecs", # compare results to this derekovecs instance