Turn away the crawlers that collect training data
Every URL with parameters is a result computed for one word, there is one
for every word of the vocabulary, and none of it can be answered from a
cache, so a crawler that follows them keeps the machine busy for nothing.
The instance on corpora.ids-mannheim.de answers about 21000 such requests
a day.
Requests whose User-Agent matches a list of such crawlers now get 403 in
before_dispatch, i.e. before any computation. The list can be replaced in
the configuration file, robots => {block_user_agents => '...'}, and an
empty value switches the blocking off. Ordinary search engines are not on
it.
For the crawlers that do read robots.txt, every response to a request
with parameters carries X-Robots-Tag: noindex, nofollow, result pages
carry the matching meta element, and robots.txt is served at every path -
which only helps where derekovecs owns its host name, a proxy that mounts
it below a path still has to serve the site wide one itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Id68487ef716459e0ea9f045060edc34edf85065e
diff --git a/templates/index.html.ep b/templates/index.html.ep
index 873c0a7..646ddf5 100644
--- a/templates/index.html.ep
+++ b/templates/index.html.ep
@@ -3,6 +3,11 @@
<head>
<% my $plain_title = $title; $plain_title=~s/<[^>]+>//g; %>
<title><%= $plain_title %>:<%= $word %> · IDS word vector analysis</title>
+%# Result pages are computed for one word each and there is one for every
+%# word of the vocabulary, so they are not something to index or to follow.
+% if (stash 'noindex') {
+ <meta name="robots" content="noindex, nofollow">
+% }
<link rel="stylesheet" href="//code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
<link href="https://fonts.googleapis.com/css?family=Lato|Roboto+Condensed" rel="stylesheet">
<script src="https://code.jquery.com/jquery-latest.min.js"></script>