| commit | 565274ef29a94438e63dffac72ac74c24a534c40 | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Sep 06 13:17:17 2026 +0200 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Sep 06 13:17:17 2026 +0200 |
| tree | bab44688a0ecfbc487dc94cf7e3f85d65604e5fa | |
| parent | 5b8aaf7556e4c5f677d51601c3f1c15e90557516 [diff] |
Search the embedding space with vector expressions "König - Mann + Frau" now looks for the neighbours of vec(König) - vec(Mann) + vec(Frau) instead of the neighbours of a single word. A '+' or '-' is an operator only at the beginning of a token, so hyphenated words like "Nord-Süd-Dialog" stay searchable, and blank separated words keep entering the query with a '+', as before. _get_neighbours() already had the branch for it, but nothing ever filled wl->sep: the wordlist came from malloc() and the signs were read out of uninitialised memory, so a multi word query subtracted operands at random. It is filled now, and its indexing corrected - sep[b] is the sign of operand b, not sep[b-1]. The predictive collocators answer for the whole query as well. getCollocators() read the input weights of wl->wordi[0], the first operand, and nothing said so: "Haus Auto" answered with the collocators of Haus, "Auto Haus" with those of Auto. The score of a candidate is sigmoid(q . syn1neg[collocate, position]), linear in q before the sigmoid, so q can be the signed combination the paradigmatic side searches around rather than the vector of one word. The terms are averaged rather than summed. The sigmoid is informative over a narrow range only - the strongest collocates of a single word reach 0.994 to 0.997 on dereko-2026-ii, against the 0.9975 that MAX_EXP allows - so a plain sum would push them into saturation. Dividing by the number of terms keeps every query in the range MIN_RESP and the auto focus are calibrated for, and leaves one word exactly as it was, divisor one. A balanced analogy is one term as well, +1 -1 +1. Those tails now saturate instead of being dropped. Activations outside ±MAX_EXP used to skip the collocate entirely, removing the strongest ones from the list and from the position and target sums. The count based collocators cannot follow: they are looked up per node in the co-occurrence database. They used to be fetched for the first paradigmatic neighbour, i.e. for the word nearest to the query vector, which for an expression is not what was asked for. The interface says so in place of that table now, and get_neighbours() reports the number of operands so it can tell the two cases apart. Operands outside the vocabulary are named above the result instead of being dropped in silence. The vocabulary lookup only recognised a miss at the very end of the vocabulary, so in a merged model a missing word resolved to whatever sits at the boundary between the two vocabularies; it returns -1 in both halves now. Two more things the tokeniser uncovered. The best array was sized for 10 * max(N, 200) entries but sorted over N * para_threads, which fits only as long as the syntagmatic threads take half of them - it is sized for num_threads slices now. And its unfilled slots kept wordi == 0 from the memset, which the result loop emitted as vocab[-1 * max_w] once a thread found fewer candidates than were asked for; slots start at -1 and the loop stops there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Change-Id: Ia91727eb669d73fc055adf652500464b8a743795
Visualizes paradigmatic and syntagmatic relations between words based on wang2vec / structured skip-n-gram (Ling et al. 2015) word embeddings (Mikolov et al. 2013) and word embedding networks.
DeReKoVecs (Fankhauser / Kupietz 2017, 2019, 2022; Kupietz et al. 2018) serves as part of the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the Collocation Database CCDB (Keibel / Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations between words based on their use in very large corpora such as the German Reference Corpus DeReKo (Kupietz et al. 2010).
cpanm https://github.com/Akron/Mojolicious-Plugin-Localize.git cpanm --installdeps . perl Makefile.PL make make install
A detailed and known to work installation procedure can also be found in the GitLab CI pipeline script.
Please note the IDS::DeReKoVecs::Read is not stable and not recommended to be used, yet.
You can build you own models with dereko2vec.
docker run -v ./example-models:/example-models:z -e MOJO_CONFIG=/example-models/example-docker.conf -p 3000:3000 idscorpuslinguistics/derekovecs
docker compose up
MOJO_CONFIG=$(pwd)/example.conf morbo script/derekovecs-server
MOJO_CONFIG=$(pwd)/example.conf hypnotoad script/derekovecs-server
The web user interface will than be available for example at http://localhost:3000
Apart from the model itself, which is memory mapped and therefore shared between all workers, each worker keeps its own result caches. Their size is limited and can be configured in the cache section of the configuration file (see example.conf):
| Option | Default | Caches |
|---|---|---|
max_keys | 50 | paradigmatic and syntagmatic neighbourhood results |
collocators_max_keys | 200 | classic (count-based) collocator profiles |
profiles_max_keys | 200 | precomputed similar profiles |
The worst case memory footprint of the caches is workers * max_keys * size of one result, and a neighbourhood result contains one vector per neighbour, so max_keys should be kept small. Setting a limit to 0 disables the respective cache, as does starting the server with -C.
Every URL of this server with parameters is a result computed for one word, there is one for every word of the vocabulary, and none of it can be answered from a cache. Crawlers that collect training data for language models walk that space, which is expensive and pointless - the corpus they are after is published as a corpus.
The server therefore answers such a crawler with 403 before it computes anything, sends X-Robots-Tag: noindex, nofollow and a matching <meta> tag for every request with parameters, so that only the entry page is indexed, and serves a robots.txt of its own. Ordinary search engines are not blocked.
The list of crawlers is a regular expression matched against the User-Agent header and can be replaced in the configuration file:
robots => { block_user_agents => 'GPTBot|ClaudeBot|Meta-ExternalAgent|CCBot' },
An empty string switches the blocking off. The built in list is in script/derekovecs-server; it contains facebookexternalhit, which Meta uses for collecting data as well as for the link previews of Facebook, Instagram and WhatsApp - remove it if those matter.
Crawlers only ever ask for /robots.txt at the root of a host, so the built in one is only read where derekovecs owns its host name. Behind a reverse proxy that mounts it below a path, e.g. https://example.org/derekovecs, the site wide /robots.txt has to say it instead:
User-agent: * Disallow: /derekovecs/*? Disallow: /derekovecs?
That is a request, not a barrier - it is obeyed by the crawlers that care, and the 403 above is what stops the rest.
The server logs one line per request. The per request diagnostics of the vector code - one line per window position, one per vocabulary lookup, the whole JSON of a similar profile - are off by default; a single neighbourhood request writes a few hundred of them, which is a million lines a day on a busy instance. Set DEREKOVECS_DEBUG=1 in the environment to get them back:
DEREKOVECS_DEBUG=1 MOJO_CONFIG=$(pwd)/example.conf morbo script/derekovecs-server
Startup messages, the memory mapping of the model and error messages are not affected by it.
The perl side logs one line per request. What it used to say in addition - which word is being looked up, which result came from a cache - is logged at debug level and can be turned on with MOJO_LOG_LEVEL=debug. Requests that are turned away as crawlers are logged with their user agent, so that the list can be checked against what actually arrives.
In addition to the web user interface, derekovecs also provides a web api which is however still very unsystematic and not stable. To figure out the meaning of still undocumented result components, have a look at the table head mouse-overs in the GUI or at the source code around here.
| Command | Parameters | Description |
|---|---|---|
| / | word, n, dedupe, cutoff, json=1 | get paradigmatic and syntagmatic neighbours, from word embeddings |
| getCollocationAssociation | w, c | get association scores for specific node collocate pairs |
| getSimilarity | w1, w2 | get cosine similarity of w1 and w2 |
| getVersion | get version of derekovecs | |
| getModelName | get name of model (inferred from the file name) | |
| getVocabSize | get vocabulary size of model |
word parameterBlank separated word forms are added into one query vector, so the search is around the centre of the alternatives, and | separates several independent searches. + and - do arithmetic in the vector space: word=König - Mann + Frau returns the neighbours of the position vec(König) - vec(Mann) + vec(Frau). A sign is an operator only at the beginning of a token, which leaves hyphenated words such as Nord-Süd-Dialog searchable; note that a literal + has to be percent encoded as %2B in a URL, since a + in a query string is a blank and would simply be another separator.
The predictive collocators answer for an expression as well: their score is sigmoid(q . syn1neg[collocate, position]), linear in the query vector q before the sigmoid, so collocators and getPosWiseW2VCollocators return the contexts that the computed position predicts. The positive terms are averaged rather than summed so that the values stay in the range of a single word form. The count based collocators of getClassicCollocators are looked up per node in the co-occurrence database and remain restricted to one word form.
Operands that are not in the vocabulary are left out of the query vector and returned in unknown; operands is the number of operands the query vector was built from.
| Command | Parameters | Description |
|---|---|---|
| getClassicCollocators | w | get count based collocates of word w |
{ "N" : 55650540526, // number of tokens in corpus "collocates" : [ // array of collocates { "afwin" : 64, // binary encoded auto-focus window // (see Perkuhn et al. 2012: E8-15): // 64 = 2^6 ≙ 00010 node 00000 // (Aus [gutem] Grund) "delta" : 0, // rank delta compared to collocation in a background // corpus (currently unused) "dice" : 0.00198886, // dice score "f" : 113490, // abs. frequency of collocation "f2" : 10965575, // abs. frequency of collocate "ld" : 5.02616, // log-dice score (Rychlý 2008) for whole window "ldaf" : 7.39257, // log-dice score for auto focus window "lfmd" : 36.0655, // log-frequency biased mutual dependency ≙ pmi³ // (Dalle 1994; Thanopoulos et al. 2002) "llr" : 204906, // log-likelihood (Dunning 1993; Evert 2004) "ln_count" : 36, // frequency of collocate as left neighbour of node "ln_pmi" : -5.81926, // pmi as left neighbour "md" : 19.2733, // mutual dependency ≙ pmi² // (Dalle 1994; Thanopoulos et al. 2002) "npmi" : 0.111633, // normalized pmi (Bouma 2009) "pmi" : 2.4811, // pointwise mutual information "rn_count" : 386, // frequency of collocate as right neighbour of node "rn_pmi" : -2.39672, // pmi as right neighbour "win" : 1023, // binary encoded positions at which the collocate // appears at least once 1023 = 2^10-1 ≙ 11111 node 11111 // (unmarked scores refer to this) "word" : "Aus" // collocate }, // ... ] }
| Command | Parameters | Description |
|---|---|---|
| /getPosWiseW2VCollocators | w(,max=200,format=json) | get top maxpredictive collocates position-wise of word w |
GET 'http://localhost:3000/?word=Grund&n=10&dedupe=0&sort=0&cutoff=1000000&json=1' | json_pp |less
GET 'http://localhost:3000/?word=K%C3%B6nig%20-%20Mann%20%2B%20Frau&n=10&json=1' | json_pp |less
curl -L http://localhost:3000/getClassicCollocators?w=Grund
GET 'http://localhost:3000/getCollocationAssociation?w=Grund&c=diesem'
GET 'http://localhost:3000/getPosWiseW2VCollocators?w=Test'
docker build -t idscorpuslinguistics/derekovecs .
slim build --include-path /usr/local/share/perl5 --mount ./example-models:/example-models:z --env MOJO_CONFIG=/example-models/example-docker.conf idscorpuslinguistics/derekovecs
Will build an image ids-kl/derekovecs.slim reduced to ~25% of the original size.
docker run -v ./example-models:/example-models:z -e MOJO_CONFIG=/example-models/example-docker.conf -p 3000:3000 idscorpuslinguistics/derekovecs
See rderekovecs.
See Changelog
Author: Marc Kupietz
Contributors: Peter Fankhauser, Rainer Perkuhn, Tim Feldmüller
Copyright (c) 2016-2024, Leibniz Institute for the German Language, Mannheim, Germany
DeReKoVecs is published under the Apache 2.0 License.
If you are using DeReKoVecs (results) for a scientific publication, please cite at least Fankhauser / Kupietz (2022).
Belica, Cyril (2011): Semantische Nähe als Ähnlichkeit von Kookkurrenzprofilen. In: Andrea Abel, Renata Zanin, Hrsg., Korpora in Lehre und Forschung, S. 155-178. Bozen-Bolzano University Press. Freie Universität Bozen-Bolzano.
Bouma, Gerlof (2009): Normalized (pointwise) mutual information in collocation extraction. In Proceedings of GSCL
Daille, B. (1994): Approche mixte pour l’extraction automatique de terminologie: statistiques lexicales et filtres linguistiques. PhD thesis, Université Paris 7.
Fankhauser, Peter / Kupietz, Marc (2022): Count-Based and Predictive Language Models for Exploring DeReKo. In: Proceedings of the LREC 2022 Workshop on Challenges in the Management of Large Corpora (CMLC-10 2022). Paris/Marseille: ELRA. pp. 27-31.
Fankhauser, Peter / Kupietz, Marc (2017): Visualizing Language Change in a Corpus of Contemporary German. In: Proceedings of the 9th International Corpus Linguistics Conference. Birmingham: University of Birmingham.
Fankhauser, Peter/Kupietz, Marc (2019): Analyzing domain specific word embeddings for a large corpus of contemporary German. International Corpus Linguistics Conference, Cardiff, Wales, UK, July 22-26, 2019. 2019. 6 S.
Keibel, H. / Belica, C. (2007): CCDB: A Corpus-Linguistic Research and Development Workbench. In: Proceedings of the 4th Corpus Linguistics Conference (CL 2007). Birmingham: University of Birmingham.
Kupietz, M. / Belica, C. / Keibel, H., Witt, A. (2010): The German Reference Corpus DeReKo: A primordial sample for linguistic research. In: Calzolari, N. et al. (eds.): Proceedings of the seventh conference on International Language Resources and Evaluation (LREC 2010). Paris: ELRA, 1848-1854.
Kupietz, M. / Lüngen, H. / Kamocki, P./ Witt, A. (2018): German Reference Corpus DeReKo: New Developments – New Opportunities. In: Calzolari, N. et al (eds): Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki: ELRA, 4353-4360
Ling, Wang / Dyer, C. / Black, A. / Trancoso, I. (2015): Two/too simple adaptations of word2vec for syntax problems. In Proc. of NAACL.
Mikolov, T. / Sutskever, I. / Chen, K. / Corrado, G. S. / Dean, J.(2013): Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS (Advances in Neural Information Processing Systems) 2013, 3111–3119.
Perkuhn, Rainer / Keibel, Holger / Kupietz, Marc (2012): Korpuslinguistik. Paderborn: Fink, 2012. Addendum
Rychlý, Pavel (2008): A lexicographer-friendly association score. In Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN, 6–9, 2008
Thanopoulos, A. / Fakotakis, N. / Kokkinakis, G. (2002): Comparative evaluation of collocation extraction metrics. In: Proc. of LREC 2002: 620–625.