Visualizes paradigmatic and syntagmatic relations between words based on wang2vec / structured skip-n-gram (Ling et al. 2015) word embeddings (Mikolov et al. 2013) and word embedding networks.
A query can be a vector expression: König - Mann + Frau is answered with the neighbours of the position that arithmetic yields, and with the contexts that position predicts. See the word parameter.
DeReKoVecs (Fankhauser / Kupietz 2017, 2019, 2022; Kupietz et al. 2018) serves as part of the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the Collocation Database CCDB (Keibel / Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations between words based on their use in very large corpora such as the German Reference Corpus DeReKo (Kupietz et al. 2010).
Makefile.PL refuses to build against an older one: Log-Dice follows Rychlý's definition from 1.8.0 on, and the interface would show the previous values under the current explanations.cpanm https://github.com/Akron/Mojolicious-Plugin-Localize.git cpanm --installdeps . perl Makefile.PL make make install
A detailed and known to work installation procedure can also be found in the GitLab CI pipeline script.
Please note the IDS::DeReKoVecs::Read is not stable and not recommended to be used, yet.
You can build you own models with dereko2vec.
docker run -v ./example-models:/example-models:z -e MOJO_CONFIG=/example-models/example-docker.conf -p 3000:3000 idscorpuslinguistics/derekovecs
docker compose up
MOJO_CONFIG=$(pwd)/example.conf morbo script/derekovecs-server
MOJO_CONFIG=$(pwd)/example.conf hypnotoad script/derekovecs-server
The web user interface will than be available for example at http://localhost:3000
Apart from the model itself, which is memory mapped and therefore shared between all workers, each worker keeps its own result caches. Their size is limited and can be configured in the cache section of the configuration file (see example.conf):
| Option | Default | Caches |
|---|---|---|
max_keys | 50 | paradigmatic and syntagmatic neighbourhood results |
collocators_max_keys | 200 | classic (count-based) collocator profiles |
profiles_max_keys | 200 | precomputed similar profiles |
The worst case memory footprint of the caches is workers * max_keys * size of one result, and a neighbourhood result contains one vector per neighbour, so max_keys should be kept small. Setting a limit to 0 disables the respective cache, as does starting the server with -C.
Every URL of this server with parameters is a result computed for one word, there is one for every word of the vocabulary, and none of it can be answered from a cache. Crawlers that collect training data for language models walk that space, which is expensive and pointless - the corpus they are after is published as a corpus.
The server therefore answers such a crawler with 403 before it computes anything, sends X-Robots-Tag: noindex, nofollow and a matching <meta> tag for every request with parameters, so that only the entry page is indexed, and serves a robots.txt of its own. Ordinary search engines are not blocked.
The list of crawlers is a regular expression matched against the User-Agent header and can be replaced in the configuration file:
robots => { block_user_agents => 'GPTBot|ClaudeBot|Meta-ExternalAgent|CCBot' },
An empty string switches the blocking off. The built in list is in script/derekovecs-server; it contains facebookexternalhit, which Meta uses for collecting data as well as for the link previews of Facebook, Instagram and WhatsApp - remove it if those matter.
Crawlers only ever ask for /robots.txt at the root of a host, so the built in one is only read where derekovecs owns its host name. Behind a reverse proxy that mounts it below a path, e.g. https://example.org/derekovecs, the site wide /robots.txt has to say it instead:
User-agent: * Disallow: /derekovecs/*? Disallow: /derekovecs?
That is a request, not a barrier - it is obeyed by the crawlers that care, and the 403 above is what stops the rest.
The server logs one line per request. The per request diagnostics of the vector code - one line per window position, one per vocabulary lookup, the whole JSON of a similar profile - are off by default; a single neighbourhood request writes a few hundred of them, which is a million lines a day on a busy instance. Set DEREKOVECS_DEBUG=1 in the environment to get them back:
DEREKOVECS_DEBUG=1 MOJO_CONFIG=$(pwd)/example.conf morbo script/derekovecs-server
Startup messages, the memory mapping of the model and error messages are not affected by it.
The perl side logs one line per request. What it used to say in addition - which word is being looked up, which result came from a cache - is logged at debug level and can be turned on with MOJO_LOG_LEVEL=debug. Requests that are turned away as crawlers are logged with their user agent, so that the list can be checked against what actually arrives.
In addition to the web user interface, derekovecs also provides a web api which is however still very unsystematic and not stable. To figure out the meaning of still undocumented result components, have a look at the table head mouse-overs in the GUI or at the source code around here, where the table heads carry them.
| Command | Parameters | Description |
|---|---|---|
| / | word, n, dedupe, cutoff, json=1 | get paradigmatic and syntagmatic neighbours, from word embeddings |
| getCollocationAssociation | w, c | get association scores for specific node collocate pairs |
| getSimilarity | w1, w2 | get cosine similarity of w1 and w2 |
| getVersion | get version of derekovecs | |
| getModelName | get name of model (inferred from the file name) | |
| getVocabSize | get vocabulary size of model |
word parameterBlank separated word forms are added into one query vector, so the search is around the centre of the alternatives, and | separates several independent searches. + and - do arithmetic in the vector space: word=König - Mann + Frau returns the neighbours of the position vec(König) - vec(Mann) + vec(Frau). A sign is an operator only at the beginning of a token, which leaves hyphenated words such as Nord-Süd-Dialog searchable; note that a literal + has to be percent encoded as %2B in a URL, since a + in a query string is a blank and would simply be another separator.
The predictive collocators answer for an expression as well: their score is sigmoid(q . syn1neg[collocate, position]), linear in the query vector q before the sigmoid, so collocators and getPosWiseW2VCollocators return the contexts that the computed position predicts. The positive terms are averaged rather than summed so that the values stay in the range of a single word form. The count based collocators of getClassicCollocators are looked up per node in the co-occurrence database and remain restricted to one word form.
Operands that are not in the vocabulary are left out of the query vector and returned in unknown; operands is the number of operands the query vector was built from.
/?word=…&json=1{ "word" : "Grund", // the query as it was parsed "operands" : 1, // operands the query vector was built from, // for the first "|" separated part "unknown" : "", // blank separated query tokens that are not // in the vocabulary and were left out "list" : [ // one array per "|" separated part [ { "word" : "Grund", "rank" : 492, // frequency rank in the vocabulary "dist" : 0.99999976, // cosine similarity to the query vector "vector" : [] // the length normalized embedding } ] ], "collocators" : [ // predictive collocates of the query vector, // null for a model without a .net file { "word" : "dafür", "rank" : 243, // frequency rank in the vocabulary "dot" : 4.006, // max(z): highest score before the sigmoid, // q . syn1neg[collocate, position] "max" : 0.98129, // max(a): the same maximum after it. Read from // a table quantizing z in steps of 1/83 and // capping it at 6, so collocates can share a // max(a) but not a max(z) "average" : 0.09813, // mean activation over the auto focus positions "prob" : 3.0353e-4, // activation sum over the auto focus positions, // normalized by their total activation "cprob" : 3.0353e-4, // co-norm of the column normalized activations // over the auto focus positions "overall" : 3.0365e-5,// activation sum over the whole window, // normalized by the total window sum "pos" : 16, // binary encoded auto focus positions "heat" : [] // per position maximum activation, left to // right, the node in the middle } ] }
| Command | Parameters | Description |
|---|---|---|
| getClassicCollocators | w | get count based collocates of word w |
{ "N" : 55650540526, // number of tokens in corpus "collocates" : [ // array of collocates { "afwin" : 64, // binary encoded auto-focus window // (see Perkuhn et al. 2012: E8-15): // 64 = 2^6 ≙ 00010 node 00000 // (Aus [gutem] Grund) "delta" : 0, // rank delta compared to collocation in a background // corpus (currently unused) "dice" : 0.00198886, // dice score "f" : 113490, // abs. frequency of collocation "f2" : 10965575, // abs. frequency of collocate "ld" : 5.02616, // log-dice score (Rychlý 2008) for whole window "ldaf" : 7.39257, // log-dice score for auto focus window "lfmd" : 36.0655, // log-frequency biased mutual dependency ≙ pmi³ // (Dalle 1994; Thanopoulos et al. 2002) "llr" : 204906, // log-likelihood (Dunning 1993; Evert 2004) "ln_count" : 36, // frequency of collocate as left neighbour of node "ln_pmi" : -5.81926, // pmi as left neighbour "md" : 19.2733, // mutual dependency ≙ pmi² // (Dalle 1994; Thanopoulos et al. 2002) "npmi" : 0.111633, // normalized pmi (Bouma 2009) "pmi" : 2.4811, // pointwise mutual information "rn_count" : 386, // frequency of collocate as right neighbour of node "rn_pmi" : -2.39672, // pmi as right neighbour "win" : 1023, // binary encoded positions at which the collocate // appears at least once 1023 = 2^10-1 ≙ 11111 node 11111 // (unmarked scores refer to this) "word" : "Aus" // collocate }, // ... ] }
| Command | Parameters | Description |
|---|---|---|
| /getPosWiseW2VCollocators | w(,max=200,cutoff=750000,threshold=0.2,format=json) | get top max predictive collocates position-wise of the query w, which may be a vector expression |
GET 'http://localhost:3000/?word=Grund&n=10&dedupe=0&sort=0&cutoff=1000000&json=1' | json_pp |less
GET 'http://localhost:3000/?word=K%C3%B6nig%20-%20Mann%20%2B%20Frau&n=10&json=1' | json_pp |less
curl -L http://localhost:3000/getClassicCollocators?w=Grund
GET 'http://localhost:3000/getCollocationAssociation?w=Grund&c=diesem'
GET 'http://localhost:3000/getPosWiseW2VCollocators?w=Test'
docker build -t idscorpuslinguistics/derekovecs .
The released image is built in three steps, as in the CI pipeline. mint keeps what the server touches while it is running, so it needs a model to probe: the example models stay in the first image and are removed again in the last step.
# 1. build with the example models still in place sed -e 's/RUN rm -rf example-models//' Dockerfile > Dockerfile.large docker build -f Dockerfile.large -t idscorpuslinguistics/derekovecs:large . # 2. run the server, probe it over HTTP and drop everything it did not touch docker run --rm -v /var/run/docker.sock:/var/run/docker.sock mintoolkit/mint \ --crt-api-version 1.46 build --http-probe=true --http-probe-start-wait 60 \ --include-workdir=true --include-path=/usr/local \ --env MOJO_CONFIG=/derekovecs/example-models/example-docker.conf \ --tag idscorpuslinguistics/derekovecs:tmp \ idscorpuslinguistics/derekovecs:large # 3. remove the example models and the toolchain mint kept docker build -f Dockerfile.remove-example-data -t idscorpuslinguistics/derekovecs .
mint takes the image from about 715 MB to 290 MB, and dropping the example models afterwards leaves about 109 MB.
docker run -v ./example-models:/example-models:z -e MOJO_CONFIG=/example-models/example-docker.conf -p 3000:3000 idscorpuslinguistics/derekovecs
See rderekovecs.
See Changelog
Author: Marc Kupietz
Contributors: Peter Fankhauser, Rainer Perkuhn, Tim Feldmüller
Copyright (c) 2016-2026, Leibniz Institute for the German Language, Mannheim, Germany
DeReKoVecs is published under the Apache 2.0 License.
If you are using DeReKoVecs (results) for a scientific publication, please cite at least Fankhauser / Kupietz (2022).
Belica, Cyril (2011): Semantische Nähe als Ähnlichkeit von Kookkurrenzprofilen. In: Andrea Abel, Renata Zanin, Hrsg., Korpora in Lehre und Forschung, S. 155-178. Bozen-Bolzano University Press. Freie Universität Bozen-Bolzano.
Bouma, Gerlof (2009): Normalized (pointwise) mutual information in collocation extraction. In Proceedings of GSCL
Daille, B. (1994): Approche mixte pour l’extraction automatique de terminologie: statistiques lexicales et filtres linguistiques. PhD thesis, Université Paris 7.
Fankhauser, Peter / Kupietz, Marc (2022): Count-Based and Predictive Language Models for Exploring DeReKo. In: Proceedings of the LREC 2022 Workshop on Challenges in the Management of Large Corpora (CMLC-10 2022). Paris/Marseille: ELRA. pp. 27-31.
Fankhauser, Peter / Kupietz, Marc (2017): Visualizing Language Change in a Corpus of Contemporary German. In: Proceedings of the 9th International Corpus Linguistics Conference. Birmingham: University of Birmingham.
Fankhauser, Peter/Kupietz, Marc (2019): Analyzing domain specific word embeddings for a large corpus of contemporary German. International Corpus Linguistics Conference, Cardiff, Wales, UK, July 22-26, 2019. 2019. 6 S.
Keibel, H. / Belica, C. (2007): CCDB: A Corpus-Linguistic Research and Development Workbench. In: Proceedings of the 4th Corpus Linguistics Conference (CL 2007). Birmingham: University of Birmingham.
Kupietz, M. / Belica, C. / Keibel, H., Witt, A. (2010): The German Reference Corpus DeReKo: A primordial sample for linguistic research. In: Calzolari, N. et al. (eds.): Proceedings of the seventh conference on International Language Resources and Evaluation (LREC 2010). Paris: ELRA, 1848-1854.
Kupietz, M. / Lüngen, H. / Kamocki, P./ Witt, A. (2018): German Reference Corpus DeReKo: New Developments – New Opportunities. In: Calzolari, N. et al (eds): Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki: ELRA, 4353-4360
Levy, Omer / Goldberg, Yoav (2014): Neural Word Embedding as Implicit Matrix Factorization. In: Advances in Neural Information Processing Systems 27 (NIPS 2014), 2177–2185.
Ling, Wang / Dyer, C. / Black, A. / Trancoso, I. (2015): Two/too simple adaptations of word2vec for syntax problems. In Proc. of NAACL.
Mikolov, T. / Sutskever, I. / Chen, K. / Corrado, G. S. / Dean, J.(2013): Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS (Advances in Neural Information Processing Systems) 2013, 3111–3119.
Perkuhn, Rainer / Keibel, Holger / Kupietz, Marc (2012): Korpuslinguistik. Paderborn: Fink, 2012. Addendum
Rychlý, Pavel (2008): A lexicographer-friendly association score. In Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN, 6–9, 2008
Thanopoulos, A. / Fakotakis, N. / Kokkinakis, G. (2002): Comparative evaluation of collocation extraction metrics. In: Proc. of LREC 2002: 620–625.