Gitiles
Code Review Sign In
korap.ids-mannheim.de / ids-kl / derekovecs / 211fff4f5f9622419eadf2713d47b3ca8160acc1
commit211fff4f5f9622419eadf2713d47b3ca8160acc1[log] [tgz]
authorMarc Kupietz <kupietz@ids-mannheim.de>Fri Jul 31 10:31:10 2026 +0900
committerMarc Kupietz <kupietz@ids-mannheim.de>Fri Jul 31 10:31:10 2026 +0900
tree213185958a06d46c98b202816beb9ec28a917855
parent606a90995f826685fc3fdc6af9cafc9e59abcf8a [diff]
CI: modernize the docker slimming

Follows what Kalamar does. slimtoolkit became mintoolkit, and the invocation
had to change anyway: the tool has no --in-container option anymore, so the
job would have stopped at an unknown flag. The download of the slim binary
goes away with it, it was never used, the containerized tool was called
instead.

FF_NETWORK_PER_BUILD and the dns option of the dind service let the mint
container reach the container it probes, as in Kalamar.

The probe waited 10 seconds for the server, which is not enough: the image
compiles its Inline C at startup and then loads the model, and only answers
after 21 seconds, measured with the image of this commit. It waits 60 now.

The image tarball carries the version in its name, as in Kalamar, and
upload-docker was adapted accordingly. It only runs for v tags now and has to
be started by hand. Before it also ran for every master commit, where
CI_COMMIT_TAG is empty and it would have pushed a tag-less image.

The .dockerignore did not catch .gitlab-ci-local, it was written with a dot
instead of a hyphen. Everybody who runs gitlab-ci-local and then builds the
image locally packed the whole CI scratch directory into it, 549 MB in my
case, including job logs, which contain the CI variables. Release tarballs are
ignored now, too. This does not affect images built in CI, whose checkout is
clean.

Verified locally by running the whole chain: the large image builds, mint
minifies it from 1.3 GB to 905 MB with the http probe answering 200, and the
image from Dockerfile.remove-example-data serves neighbours, collocators and
getVecsByRanks correctly with the compilers removed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I0014c8d2e4f4cee667b0d1c5016f037c9aa44743
2 files changed
tree: 213185958a06d46c98b202816beb9ec28a917855
  1. ci/
  2. css/
  3. example-models/
  4. img/
  5. inc/
  6. js/
  7. lib/
  8. script/
  9. t/
  10. templates/
  11. .dockerignore
  12. .gitignore
  13. .gitlab-ci.yml
  14. Changelog.md
  15. CMakeLists.txt
  16. compose.yaml
  17. derekovecs-server.dict
  18. Dockerfile
  19. Dockerfile.remove-example-data
  20. example.conf
  21. LICENSE
  22. Makefile.PL
  23. README.md
README.md

DeReKoVecs (server and web app)

Visualizes paradigmatic and syntagmatic relations between words based on wang2vec / structured skip-n-gram (Ling et al. 2015) word embeddings (Mikolov et al. 2013) and word embedding networks.

DeReKoVecs (Fankhauser / Kupietz 2017, 2019, 2022; Kupietz et al. 2018) serves as part of the new open lab of the Corpus Linguistics group at IDS Mannheim. Similar to the Collocation Database CCDB (Keibel / Belica 2007, Belica 2011), DeReKoVecs serves for investigating and comparing of measurements, dimension reduction procedures, visualizations etc., to track down detailed paradigmatic and syntagmatic relations between words based on their use in very large corpora such as the German Reference Corpus DeReKo (Kupietz et al. 2010).

Installation from source

Dependencies

  • libcollocatordb >= v1.5.0 (which builds against the rocksdb of the distribution, no rocksdb 5.11 from source needed)

Build and install

cpanm https://github.com/Akron/Mojolicious-Plugin-Localize.git
cpanm --installdeps .

perl Makefile.PL
make
make install

A detailed and known to work installation procedure can also be found in the GitLab CI pipeline script.

Please note the IDS::DeReKoVecs::Read is not stable and not recommended to be used, yet.

Build your own models

You can build you own models with dereko2vec.

Run

From prebuilt docker image

docker run -v ./example-models:/example-models:z -e MOJO_CONFIG=/example-models/example-docker.conf -p 3000:3000 idscorpuslinguistics/derekovecs

From prebuilt docker image with docker compose

docker compose up

From source in debug mode

MOJO_CONFIG=$(pwd)/example.conf morbo script/derekovecs-server

From source in production mode

MOJO_CONFIG=$(pwd)/example.conf hypnotoad script/derekovecs-server

The web user interface will than be available for example at http://localhost:3000

Memory usage and caching

Apart from the model itself, which is memory mapped and therefore shared between all workers, each worker keeps its own result caches. Their size is limited and can be configured in the cache section of the configuration file (see example.conf):

OptionDefaultCaches
max_keys50paradigmatic and syntagmatic neighbourhood results
collocators_max_keys200classic (count-based) collocator profiles
profiles_max_keys200precomputed similar profiles

The worst case memory footprint of the caches is workers * max_keys * size of one result, and a neighbourhood result contains one vector per neighbour, so max_keys should be kept small. Setting a limit to 0 disables the respective cache, as does starting the server with -C.

Web Service API

In addition to the web user interface, derekovecs also provides a web api which is however still very unsystematic and not stable. To figure out the meaning of still undocumented result components, have a look at the table head mouse-overs in the GUI or at the source code around here.

CommandParametersDescription
/word, n, dedupe, cutoff, json=1get paradigmatic and syntagmatic neighbours, from word embeddings
getCollocationAssociationw, cget association scores for specific node collocate pairs
getSimilarityw1, w2get cosine similarity of w1 and w2
getVersionget version of derekovecs
getModelNameget name of model (inferred from the file name)
getVocabSizeget vocabulary size of model

Get classical (count-based) collocates

CommandParametersDescription
getClassicCollocatorswget count based collocates of word w

Example Result (node: Grund)

{
   "N" : 55650540526,           // number of tokens in corpus
   "collocates" : [             // array of collocates
      {
         "afwin" : 64,          // binary encoded auto-focus window
                                // (see Perkuhn et al. 2012: E8-15):
                                // 64 = 2^6 ≙ 00010 node 00000
                                // (Aus [gutem] Grund)
         "delta" : 0,           // rank delta compared to collocation in a background
                                // corpus (currently unused)
         "dice" : 0.00198886,   // dice score
         "f" : 113490,          // abs. frequency of collocation
         "f2" : 10965575,       // abs. frequency of collocate
         "ld" : 5.02616,        // log-dice score (Rychlý 2008) for whole window
         "ldaf" : 7.39257,      // log-dice score for auto focus window
         "lfmd" : 36.0655,      // log-frequency biased mutual dependency ≙ pmi³
                                // (Dalle 1994; Thanopoulos et al. 2002)
         "llr" : 204906,        // log-likelihood (Dunning 1993; Evert 2004)
         "ln_count" : 36,       // frequency of collocate as left neighbour of node
         "ln_pmi" : -5.81926,   // pmi as left neighbour
         "md" : 19.2733,        // mutual dependency ≙ pmi²
                                // (Dalle 1994; Thanopoulos et al. 2002)
         "npmi" : 0.111633,     // normalized pmi (Bouma 2009)
         "pmi" : 2.4811,        // pointwise mutual information
         "rn_count" : 386,      // frequency of collocate as right neighbour of node
         "rn_pmi" : -2.39672,   // pmi as right neighbour
         "win" : 1023,          // binary encoded positions at which the collocate
                                // appears at least once 1023 = 2^10-1 ≙ 11111 node 11111
                                // (unmarked scores refer to this)
         "word" : "Aus"         // collocate
      },
      // ...
   ]
}

Get top predictive collocates position-wise

CommandParametersDescription
/getPosWiseW2VCollocatorsw(,max=200,format=json)get top maxpredictive collocates position-wise of word w

Examples

GET 'http://localhost:3000/?word=Grund&n=10&dedupe=0&sort=0&cutoff=1000000&json=1' | json_pp |less

curl -L http://localhost:3000/getClassicCollocators?w=Grund

GET 'http://localhost:3000/getCollocationAssociation?w=Grund&c=diesem'

GET 'http://localhost:3000/getPosWiseW2VCollocators?w=Test'

(Build and) run using docker / podman

Optional: Build docker image from source

docker build -t idscorpuslinguistics/derekovecs .

Optional: Slim down image using Slim(toolkit)

slim build --include-path /usr/local/share/perl5 --mount ./example-models:/example-models:z --env MOJO_CONFIG=/example-models/example-docker.conf idscorpuslinguistics/derekovecs

Will build an image ids-kl/derekovecs.slim reduced to ~25% of the original size.

Run docker image

docker run -v ./example-models:/example-models:z -e MOJO_CONFIG=/example-models/example-docker.conf -p 3000:3000 idscorpuslinguistics/derekovecs

Client library for R

See rderekovecs.

News

See Changelog

Development and License

Author: Marc Kupietz

Contributors: Peter Fankhauser, Rainer Perkuhn, Tim Feldmüller

Copyright (c) 2016-2024, Leibniz Institute for the German Language, Mannheim, Germany

DeReKoVecs is published under the Apache 2.0 License.

How to cite

If you are using DeReKoVecs (results) for a scientific publication, please cite at least Fankhauser / Kupietz (2022).

References

Belica, Cyril (2011): Semantische Nähe als Ähnlichkeit von Kookkurrenzprofilen. In: Andrea Abel, Renata Zanin, Hrsg., Korpora in Lehre und Forschung, S. 155-178. Bozen-Bolzano University Press. Freie Universität Bozen-Bolzano.

Bouma, Gerlof (2009): Normalized (pointwise) mutual information in collocation extraction. In Proceedings of GSCL

Daille, B. (1994): Approche mixte pour l’extraction automatique de terminologie: statistiques lexicales et filtres linguistiques. PhD thesis, Université Paris 7.

Fankhauser, Peter / Kupietz, Marc (2022): Count-Based and Predictive Language Models for Exploring DeReKo. In: Proceedings of the LREC 2022 Workshop on Challenges in the Management of Large Corpora (CMLC-10 2022). Paris/Marseille: ELRA. pp. 27-31.

Fankhauser, Peter / Kupietz, Marc (2017): Visualizing Language Change in a Corpus of Contemporary German. In: Proceedings of the 9th International Corpus Linguistics Conference. Birmingham: University of Birmingham.

Fankhauser, Peter/Kupietz, Marc (2019): Analyzing domain specific word embeddings for a large corpus of contemporary German. International Corpus Linguistics Conference, Cardiff, Wales, UK, July 22-26, 2019. 2019. 6 S.

Keibel, H. / Belica, C. (2007): CCDB: A Corpus-Linguistic Research and Development Workbench. In: Proceedings of the 4th Corpus Linguistics Conference (CL 2007). Birmingham: University of Birmingham.

Kupietz, M. / Belica, C. / Keibel, H., Witt, A. (2010): The German Reference Corpus DeReKo: A primordial sample for linguistic research. In: Calzolari, N. et al. (eds.): Proceedings of the seventh conference on International Language Resources and Evaluation (LREC 2010). Paris: ELRA, 1848-1854.

Kupietz, M. / Lüngen, H. / Kamocki, P./ Witt, A. (2018): German Reference Corpus DeReKo: New Developments – New Opportunities. In: Calzolari, N. et al (eds): Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki: ELRA, 4353-4360

Ling, Wang / Dyer, C. / Black, A. / Trancoso, I. (2015): Two/too simple adaptations of word2vec for syntax problems. In Proc. of NAACL.

Mikolov, T. / Sutskever, I. / Chen, K. / Corrado, G. S. / Dean, J.(2013): Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS (Advances in Neural Information Processing Systems) 2013, 3111–3119.

Perkuhn, Rainer / Keibel, Holger / Kupietz, Marc (2012): Korpuslinguistik. Paderborn: Fink, 2012. Addendum

Rychlý, Pavel (2008): A lexicographer-friendly association score. In Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN, 6–9, 2008

Thanopoulos, A. / Fakotakis, N. / Kokkinakis, G. (2002): Comparative evaluation of collocation extraction metrics. In: Proc. of LREC 2002: 620–625.