Fork of wang2vec with extensions for re-training and count based models, support for tokens with frequencies > 2³² and a more accurate ETA prognosis.
scripts/install.sh
installs the packages of the distribution, builds rocksdb where there is none or none with the static library, then collocatordb, and then dereko2vec with the fast static link, testing and installing everything below /usr/local. Works on Fedora, Rocky Linux 9 and 10, RHEL, Debian, Ubuntu and MacOS. See scripts/install.sh --help for the options, e.g. --shared for linking rocksdb and collocatordb shared, or --prefix for another install location.
Where root rights are hard to come by - e.g. a server where you have only the root password and no sudo - the script falls back to su, which prompts for that password. And with --build-only it needs no root at all: it builds everything, tests it, and leaves the binaries in the build directory, linked against the static collocatordb and rocksdb, so they run from there:
scripts/install.sh --build-only
What the script runs, for doing it by hand or where it does not reach:
on Fedora:
sudo dnf install cmake gcc-c++ git rocksdb-devel snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel
on Rocky Linux 9 and 10 and RHEL: the same without rocksdb-devel, rocksdb is built in the next step:
sudo dnf install cmake gcc-c++ git snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel
on MacOS:
brew install cmake rocksdb snappy zlib bzip2 lz4 zstd
Linking rocksdb statically makes counting collocations about 22% faster - on an indexing run of two weeks that is about three days. Debian, Ubuntu and MacOS ship librocksdb.a in their rocksdb package, Fedora, Rocky Linux and RHEL do not. One script builds rocksdb once, with the shared and the static library, into a prefix of its own:
scripts/build-rocksdb.sh
It builds the version of the rocksdb package of the distribution where there is one, so that the static library matches the headers collocatordb is compiled against, and a pinned recent one where there is none (Rocky Linux, RHEL). The version and the prefix can be changed, see scripts/build-rocksdb.sh --help.
On Rocky Linux and RHEL, which have no rocksdb package, this step is not optional, and programs that use the shared library need the prefix in the search path of the loader, which the script prints at its end.
libcollocatordb >= v1.7.0:
git clone "https://korap.ids-mannheim.de/gerrit/ids-kl/collocatordb" cd collocatordb cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/usr/local cmake --build build -j $(nproc) sudo cmake --install build && sudo ldconfig ctest --test-dir build --extra-verbose cd ..
Where rocksdb was built in step 2 and it is not the one of the distribution - on Rocky Linux and RHEL always, elsewhere when -v named another version - collocatordb has to be built against it, and naming its prefix covers the headers and both libraries at once:
cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/rocksdb -DCMAKE_INSTALL_PREFIX=/usr/local
cmake -S . -B build cmake --build build -j $(nproc) ctest --test-dir build --extra-verbose sudo cmake --install build
This installs dereko2vec and vecs2mmap. The build directory has to be build inside the sources, the test looks for the binary relative to it.
For the fast build with rocksdb and collocatordb linked statically, name the prefix of the rocksdb from step 2. The compression libraries stay shared:
cmake -S . -B build -DSTATIC_ROCKSDB=ON -DCMAKE_PREFIX_PATH=$HOME/rocksdb
On Debian, Ubuntu and MacOS, whose rocksdb package brings the static library, -DSTATIC_ROCKSDB=ON without a prefix does. And where rocksdb was installed into a prefix of its own but is to be linked shared, name it without the option:
cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/rocksdb
It has to be the same rocksdb that collocatordb was built against. dereko2vec itself does not include any rocksdb header, but it links what collocatordb refers to, and symbols of one version do not exist in another. Where the script of step 2 was used for both, this is already taken care of.
A completely static binary, -DSTATIC_DEREKO2VEC=ON, needs static versions of zlib, snappy, lz4 and zstd on top, which Rocky Linux, RHEL and Fedora do not ship, and is rarely worth the trouble.
error while loading shared libraries: libcollocatordb.so.1
means the loader does not search the directory the library was installed to. The installed dereko2vec carries the directory of the collocatordb it was linked against, so this only happens with a binary that was built before that, or when it is the rocksdb below that library which is not found. Either run sudo ldconfig after installing collocatordb, or add the directory:
echo $HOME/rocksdb/lib64 | sudo tee /etc/ld.so.conf.d/rocksdb.conf sudo ldconfig
A statically linked dereko2vec has neither problem.
The command to build word embeddings is exactly the same as in the original version, except that we added type 5 for setting up a purely count based collocation database.
The -type argument is a integer that defines the architecture to use. These are the possible parameters:
0 - cbow
1 - skipngram
2 - cwindow (see below)
3 - structured skipngram(see below)
4 - collobert's senna context window model (still experimental)
5 - build a collocation count database instead of word embeddings
./dereko2vec -train input_file -output embedding_file -type 0 -size 50 -window 5 -negative 10 -nce 0 -hs 0 -sample 1e-4 -threads 1 -binary 1 -iter 5 -cap 0
The KorAP-XML-CoNLL-U tool can be used to generate input files for dereko2vec from KorAP-XML ZIPs using its tokenization and setence boundary information, for example:
korapxml2conllu --word2vec wpd19.zip > wpd19.w2vinput
dereko2vec -train new.traindata -output new.vecs -save-net new.net -type 3 -size 200 -window 5 -negative 10 -threads 44 -binary 1 -iter 100 -read-vocab old.vocab -read-net old.net
dereko2vec -train new.traindata -output new.rocksdb -type 5 -window 5 -threads 8 -binary 1 -iter 1 -read-vocab old.vocab -sample 0 -min-count 0 dereko2vec -train new.traindata -output .temp.rocksdb -type 5 -window 5 -threads 8 -binary 1 -iter 1 -save-vocab new_focus.vocab -sample 0 -min-count 0 rm -rf .temp.rocksdb python scripts/merge_vocabs.py old.vocab new_focus.vocab new.vocab
@InProceedings{Ling:2015:naacl, author = {Ling, Wang and Dyer, Chris and Black, Alan and Trancoso, Isabel}, title="Two/Too Simple Adaptations of word2vec for Syntax Problems", booktitle="Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies", year="2015", publisher="Association for Computational Linguistics", location="Denver, Colorado", } @InProceedings{FankhauserKupietz2019, author = {Peter Fankhauser and Marc Kupietz}, title = {Analyzing domain specific word embeddings for a large corpus of contemporary German}, series = {Proceedings of the 10th International Corpus Linguistics Conference}, publisher = {University of Cardiff}, address = {Cardiff}, year = {2019}, note = {\url{https://doi.org/10.14618/ids-pub-9117}} }