Turn away the crawlers that collect training data
Every URL with parameters is a result computed for one word, there is one
for every word of the vocabulary, and none of it can be answered from a
cache, so a crawler that follows them keeps the machine busy for nothing.
The instance on corpora.ids-mannheim.de answers about 21000 such requests
a day.
Requests whose User-Agent matches a list of such crawlers now get 403 in
before_dispatch, i.e. before any computation. The list can be replaced in
the configuration file, robots => {block_user_agents => '...'}, and an
empty value switches the blocking off. Ordinary search engines are not on
it.
For the crawlers that do read robots.txt, every response to a request
with parameters carries X-Robots-Tag: noindex, nofollow, result pages
carry the matching meta element, and robots.txt is served at every path -
which only helps where derekovecs owns its host name, a proxy that mounts
it below a path still has to serve the site wide one itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Id68487ef716459e0ea9f045060edc34edf85065e
diff --git a/Changelog.md b/Changelog.md
index f72fb13..652474b 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -1,5 +1,20 @@
# Changelog
+## [0.97] - 2026-08-21
+
+- crawlers that collect training data for language models are answered with
+ `403` before anything is computed. They walk the parameterised URLs, which
+ are a computed result per word of the vocabulary and cannot be cached - the
+ instance on corpora.ids-mannheim.de answers about 21000 of them a day. The
+ list of user agents can be replaced with `robots => {block_user_agents => '...'}`
+ in the configuration file, an empty value switches the blocking off
+- every response to a request with parameters carries
+ `X-Robots-Tag: noindex, nofollow`, and the result pages carry the matching
+ `<meta name="robots">`, so that search engines index the entry page only
+- `robots.txt` is served at every path, for installations that own their host
+ name; behind a proxy that mounts derekovecs below a path, the site wide
+ robots.txt still has to say it
+
## [0.96] - 2026-07-31
- the docker image runs as any user now. It only started as root before,
diff --git a/README.md b/README.md
index b6e0af3..346c4ea 100644
--- a/README.md
+++ b/README.md
@@ -78,6 +78,47 @@
one vector per neighbour, so `max_keys` should be kept small. Setting a limit
to `0` disables the respective cache, as does starting the server with `-C`.
+### Crawlers
+
+Every URL of this server with parameters is a result computed for one word,
+there is one for every word of the vocabulary, and none of it can be answered
+from a cache. Crawlers that collect training data for language models walk that
+space, which is expensive and pointless - the corpus they are after is
+published as a corpus.
+
+The server therefore answers such a crawler with `403` before it computes
+anything, sends `X-Robots-Tag: noindex, nofollow` and a matching `<meta>` tag
+for every request with parameters, so that only the entry page is indexed, and
+serves a `robots.txt` of its own. Ordinary search engines are not blocked.
+
+The list of crawlers is a regular expression matched against the `User-Agent`
+header and can be replaced in the configuration file:
+
+```perl
+robots => {
+ block_user_agents => 'GPTBot|ClaudeBot|Meta-ExternalAgent|CCBot'
+},
+```
+
+An empty string switches the blocking off. The built in list is in
+[script/derekovecs-server](script/derekovecs-server); it contains
+`facebookexternalhit`, which Meta uses for collecting data as well as for the
+link previews of Facebook, Instagram and WhatsApp - remove it if those matter.
+
+Crawlers only ever ask for `/robots.txt` at the root of a host, so the built in
+one is only read where derekovecs owns its host name. Behind a reverse proxy
+that mounts it below a path, e.g. `https://example.org/derekovecs`, the site
+wide `/robots.txt` has to say it instead:
+
+```
+User-agent: *
+Disallow: /derekovecs/*?
+Disallow: /derekovecs?
+```
+
+That is a request, not a barrier - it is obeyed by the crawlers that care, and
+the `403` above is what stops the rest.
+
## Web Service API
In addition to the web user interface, derekovecs also provides a web api which is however still very unsystematic and **not stable**. To figure out the meaning of still undocumented result components, have a look at the table head mouse-overs in the GUI or at the source code around [here](https://korap.ids-mannheim.de/gerrit/plugins/gitiles/ids-kl/derekovecs/+/refs/heads/master/templates/index.html.ep#684).
diff --git a/example.conf b/example.conf
index 34d6297..e999b3d 100644
--- a/example.conf
+++ b/example.conf
@@ -21,6 +21,14 @@
profiles_max_keys => 200 # similar profiles (JSON)
},
+ # Crawlers that collect training data for language models are answered with
+ # 403 before anything is computed. The value is a regular expression that is
+ # matched against the User-Agent header; the built in list is in
+ # script/derekovecs-server. An empty string switches the blocking off.
+ # robots => {
+ # block_user_agents => 'GPTBot|ClaudeBot|Meta-ExternalAgent|CCBot'
+ # },
+
w2v => {
vecs => "example-models/wpd19_10000/wpd19_10000.vecs",
# compare_to => "https://corpora.ids-mannheim.de/openlab/derekovecs", # compare results to this derekovecs instance
diff --git a/script/derekovecs-server b/script/derekovecs-server
index d460345..bc66b01 100755
--- a/script/derekovecs-server
+++ b/script/derekovecs-server
@@ -1,6 +1,6 @@
#!/usr/bin/env perl
-our $VERSION = '0.96';
+our $VERSION = '0.97';
use IDS::DeReKoVecs::Read;
use Mojolicious::Lite;
@@ -61,6 +61,78 @@
our $opt_p = 5676;
our $opt_C;
+# Crawlers that collect training data for language models walk the
+# parameterised URLs of this server, and those are the most expensive thing it
+# does: every word is a URL of its own, every result links to the next words,
+# so the crawl space is the whole vocabulary and nothing of it can be served
+# from a cache. They are turned away before anything is computed.
+#
+# The list is a regular expression and can be replaced in the configuration
+# file, robots => {block_user_agents => '...'}; an empty value switches the
+# blocking off. Ordinary search engines are deliberately not in it - they are
+# kept out of the parameterised URLs by robots.txt and X-Robots-Tag instead,
+# and the entry page stays indexable.
+my $DEFAULT_BLOCKED_AGENTS = join('|',
+ 'AI2Bot', 'Amazonbot', 'anthropic-ai', 'Bytespider', 'CCBot', 'ChatGPT-User',
+ 'Claude-SearchBot', 'Claude-User', 'Claude-Web', 'ClaudeBot', 'cohere-ai',
+ 'cohere-training-data-crawler', 'Diffbot', 'DuckAssistBot', 'FacebookBot',
+ 'facebookexternalhit', 'FriendlyCrawler', 'GPTBot', 'ImagesiftBot',
+ 'Kangaroo Bot', 'Meta-ExternalAgent', 'Meta-ExternalFetcher', 'meta-webindexer',
+ 'OAI-SearchBot', 'omgili', 'PanguBot', 'Perplexity-User', 'PerplexityBot',
+ 'Scrapy', 'SemrushBot-OCOB', 'Timpibot', 'TikTokSpider', 'YouBot'
+);
+
+my $blocked_agents = app->config->{robots}->{block_user_agents} // $DEFAULT_BLOCKED_AGENTS;
+my $blocked_agents_re = length($blocked_agents) ? qr/(?i:$blocked_agents)/ : undef;
+
+# robots.txt for the ones that read it. A reverse proxy that mounts derekovecs
+# below a path has to serve the site wide /robots.txt itself, crawlers only
+# ask for it at the root of the host - this one is for installations that own
+# their host name, and does no harm anywhere else.
+my $ROBOTS_TXT = join("\n",
+ '# Every URL with parameters is a result computed for one word, and the',
+ '# vocabulary is large enough that following them is an endless crawl.',
+ 'User-agent: *',
+ 'Disallow: /*?',
+ 'Crawl-delay: 10',
+ '',
+ '# Collecting training data for language models is not what this service is',
+ '# here for. Please use the corpus itself, https://www.ids-mannheim.de/dereko',
+ (map { "User-agent: $_" }
+ (split(/\|/, $DEFAULT_BLOCKED_AGENTS),
+ # tokens that are only understood in robots.txt, there is no crawler of
+ # that name to match against
+ 'Google-Extended', 'Applebot-Extended')),
+ 'Disallow: /',
+ ''
+);
+
+hook before_dispatch => sub {
+ my $c = shift;
+
+ if ($c->req->url->path->to_string =~ m@(?:^|/)robots\.txt$@) {
+ return $c->render(data => $ROBOTS_TXT, format => 'txt');
+ }
+
+ # Only the entry page is worth indexing, everything else is computed.
+ if (length($c->req->url->query->to_string)) {
+ $c->res->headers->header('X-Robots-Tag' => 'noindex, nofollow');
+ $c->stash(noindex => 1);
+ }
+
+ my $ua = $c->req->headers->user_agent;
+ if ($blocked_agents_re && defined($ua) && $ua =~ $blocked_agents_re) {
+ $c->app->log->info('Blocked crawler ' . $c->remote_addr . ' "' . $ua . '" ' . $c->req->url);
+ $c->res->headers->header('X-Robots-Tag' => 'noindex, nofollow');
+ return $c->render(
+ text => "This service computes an answer for every request and is not a source of training data.\n"
+ . "The corpus behind it is documented at https://www.ids-mannheim.de/dereko\n",
+ status => 403,
+ format => 'txt'
+ );
+ }
+};
+
# Neighbourhood results are the biggest objects the server keeps around (one
# hash plus one vector array per neighbour), so this cache is deliberately
# small. It is per worker, i.e. the total footprint is
diff --git a/templates/index.html.ep b/templates/index.html.ep
index 873c0a7..646ddf5 100644
--- a/templates/index.html.ep
+++ b/templates/index.html.ep
@@ -3,6 +3,11 @@
<head>
<% my $plain_title = $title; $plain_title=~s/<[^>]+>//g; %>
<title><%= $plain_title %>:<%= $word %> · IDS word vector analysis</title>
+%# Result pages are computed for one word each and there is one for every
+%# word of the vocabulary, so they are not something to index or to follow.
+% if (stash 'noindex') {
+ <meta name="robots" content="noindex, nofollow">
+% }
<link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
<link href="https://fonts.googleapis.com/css?family=Lato|Roboto+Condensed" rel="stylesheet">
<script src="https://code.jquery.com/jquery-latest.min.js"></script>