| commit | 88c26b1aaf7fb4e6aa88efa4bdc74de3a65f8fd5 | [log] [tgz] |
|---|---|---|
| author | Akron <nils@diewald-online.de> | Mon Sep 07 12:44:18 2020 +0200 |
| committer | Akron <nils@diewald-online.de> | Thu Sep 10 14:21:57 2020 +0200 |
| tree | c9d82d3793631911730b2cd893a11773b5bc2d4e | |
| parent | 45308ceba8cb6d6188ee12656b8bbcdc740a66cb [diff] |
Exclude search results from robot indices (closes #120) Change-Id: If6387bb7585afbba80743cad9cfcf8779517c6b3
diff --git a/dev/robots.txt b/dev/robots.txt new file mode 100644 index 0000000..88e4d02 --- /dev/null +++ b/dev/robots.txt
@@ -0,0 +1,10 @@ +# robots.txt only applies, when KorAP is accessible +# from the domain root. + +User-agent: * +Disallow: /api/ +Disallow: /css/ +Disallow: /font/ +Disallow: /js/ +Disallow: /settings/ +Disallow: /corpus/