| commit | c2f2fb455f18a75189434d274a3464ebfcf16d7c | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Aug 02 15:37:25 2026 +0200 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Aug 02 15:37:25 2026 +0200 |
| tree | 5e4b72fca510ff795d1d8353b10f478173caf90a | |
| parent | 5814193051abed5cf037f6b605c489190fa0173f [diff] |
Say how to link the collocation database statically The speed comes from linking rocksdb and collocatordb statically, not from a completely static binary, which additionally needs static system libraries that Rocky Linux, RHEL and Fedora do not ship. Pointing the build at the two static libraries is enough and makes indexing about 22% faster. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Change-Id: I41d6f9a532670b821e1938d98f1fc10afc497bb0
Fork of wang2vec with extensions for re-training and count based models, support for tokens with frequencies > 2³² and a more accurate ETA prognosis.
librocksdb-dev on Debian and Ubuntu or rocksdb-devel on Fedora and Rocky Linuxcd dereko2vec mkdir build cd build cmake .. make && ctest --extra-verbose && sudo make install
This installs dereko2vec and vecs2mmap. The build directory has to be build inside the sources, the test looks for the binary relative to it.
Linking rocksdb and collocatordb statically makes counting collocations about 22% faster, measured with 2 million increments against rocksdb 7.8.3. On an indexing run of two weeks that is about three days. Point the build at the static libraries, the compression libraries stay shared:
cmake -DCOLLOCATORDB=/usr/local/lib64/libcollocatordb_static.a \ -DROCKSDB=/path/to/librocksdb.a ..
This does not need static versions of zlib, snappy, lz4 and zstd, which Rocky Linux, RHEL and Fedora do not ship. Only -DSTATIC_DEREKO2VEC=ON, which builds a completely static binary, needs those, and is rarely worth the trouble.
Debian and Ubuntu have librocksdb.a in librocksdb-dev. Fedora, Rocky Linux and RHEL do not, the collocatordb README says how to build one there without shadowing the headers of the package.
The command to build word embeddings is exactly the same as in the original version, except that we added type 5 for setting up a purely count based collocation database.
The -type argument is a integer that defines the architecture to use. These are the possible parameters:
0 - cbow
1 - skipngram
2 - cwindow (see below)
3 - structured skipngram(see below)
4 - collobert's senna context window model (still experimental)
5 - build a collocation count database instead of word embeddings
./dereko2vec -train input_file -output embedding_file -type 0 -size 50 -window 5 -negative 10 -nce 0 -hs 0 -sample 1e-4 -threads 1 -binary 1 -iter 5 -cap 0
The KorAP-XML-CoNLL-U tool can be used to generate input files for dereko2vec from KorAP-XML ZIPs using its tokenization and setence boundary information, for example:
korapxml2conllu --word2vec wpd19.zip > wpd19.w2vinput
dereko2vec -train new.traindata -output new.vecs -save-net new.net -type 3 -size 200 -window 5 -negative 10 -threads 44 -binary 1 -iter 100 -read-vocab old.vocab -read-net old.net
dereko2vec -train new.traindata -output new.rocksdb -type 5 -window 5 -threads 8 -binary 1 -iter 1 -read-vocab old.vocab -sample 0 -min-count 0 dereko2vec -train new.traindata -output .temp.rocksdb -type 5 -window 5 -threads 8 -binary 1 -iter 1 -save-vocab new_focus.vocab -sample 0 -min-count 0 rm -rf .temp.rocksdb python scripts/merge_vocabs.py old.vocab new_focus.vocab new.vocab
@InProceedings{Ling:2015:naacl, author = {Ling, Wang and Dyer, Chris and Black, Alan and Trancoso, Isabel}, title="Two/Too Simple Adaptations of word2vec for Syntax Problems", booktitle="Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies", year="2015", publisher="Association for Computational Linguistics", location="Denver, Colorado", } @InProceedings{FankhauserKupietz2019, author = {Peter Fankhauser and Marc Kupietz}, title = {Analyzing domain specific word embeddings for a large corpus of contemporary German}, series = {Proceedings of the 10th International Corpus Linguistics Conference}, publisher = {University of Cardiff}, address = {Cardiff}, year = {2019}, note = {\url{https://doi.org/10.14618/ids-pub-9117}} }