| commit | 89c7b9f06c5c00819c30a92c9511533882468d80 | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Aug 02 07:38:44 2026 +0200 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Aug 02 07:38:44 2026 +0200 |
| tree | 4708f1a863054f175d7b202b69c5a5f5a2513f30 | |
| parent | 8fd499eccf7219809657ac32e41614ca48ca2282 [diff] |
Look for the shared library suffix of the platform Only reached on platforms other than MacOS, which has its own branch, but the hard coded .so has no business there either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Change-Id: Ie092e28542768d1da90312cc0d9900b2f64fc7d2
The rocksdb of the distribution is used, no particular version is required.
on Fedora, Rocky Linux, RHEL
sudo dnf install cmake gcc-c++ rocksdb-devel snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel gflags-devel
on Ubuntu, Debian
sudo apt-get install cmake librocksdb-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev libomp-dev
on MacOS
brew install cmake rocksdb snappy zlib bzip2 lz4 zstd libomp gflags
Do not install another rocksdb below /usr/local next to the packaged one. The compiler looks into /usr/local/include before /usr/include, so its headers would be used while the library of the distribution is linked, which ends in a long list of undefined references. The build compares the version of the headers with the version of the library and stops with an explanation when they do not match.
git clone "https://korap.ids-mannheim.de/gerrit/ids-kl/collocatordb" cd collocatordb mkdir -p build cd build cmake -DCMAKE_INSTALL_PREFIX=/usr/local .. make && sudo make install && sudo ldconfig ctest --extra-verbose
The tests run after the installation, not before: one of them calls the collocatordb_query tool, which is built with the rpath of its install location and does not find the library as long as it is not installed. The build directory has to be build inside the sources, the tests look for the tool and for their data relative to it.
libcollocatordb_static.a is built as well. It is linked against the shared rocksdb unless a static rocksdb is found, which is enough for most purposes. A static rocksdb is only needed to link a program completely statically, as dereko2vec does, which is worth about 10% on collocator lookups.
Debian and Ubuntu ship librocksdb.a in librocksdb-dev, so there is nothing to do. Fedora, Rocky Linux and RHEL do not ship one, so it has to be built. Keep it out of /usr/local, where its headers would shadow those of the package, and point the build at it:
git clone https://github.com/facebook/rocksdb.git -b v$(rpm -q --qf '%{VERSION}' rocksdb) --single-branch cd rocksdb make -j $(nproc) static_lib make install-static INSTALL_PATH=$HOME/rocksdb-static cd ../collocatordb/build cmake -DROCKSDB_STATIC=$HOME/rocksdb-static/lib/librocksdb.a -DCMAKE_INSTALL_PREFIX=/usr/local .. make && sudo make install && sudo ldconfig
typedef struct { uint32_t w2; uint64_t f2; uint64_t raw; double pmi; double npmi; double llr; double lfmd; double md; double md_nws; uint64_t left_raw; uint64_t right_raw; double left_pmi; double right_pmi; double dice; double logdice; double ldaf; int window; int af_window; } COLLOCATOR ; COLLOCATORDB *open_collocatordb(const char *s); COLLOCATORDB *open_collocatordb_for_write(const char *s); void inc_collocator(COLLOCATORDB *db, uint64_t w1, uint64_t w2, int8_t dist); void dump_collocators(COLLOCATORDB *db, uint32_t w1, uint32_t w2, int8_t dist); COLLOCATOR *get_collocators(COLLOCATORDB *db, uint32_t w1); COLLOCATOR *get_collocation_scores(COLLOCATORDB *db, uint32_t w1, uint32_t w2); char *get_collocators_as_json(COLLOCATORDB *db, uint32_t w1); char *get_collocation_scores_as_json(COLLOCATORDB *db, uint32_t w1, uint32_t w2); char *get_word(COLLOCATORDB *db, uint32_t w1); void read_vocab(COLLOCATORDB *db, char *fname); char *get_version(); uint64_t get_word_id(COLLOCATORDB *db, const char *word); uint64_t get_corpus_size(COLLOCATORDB *db); uint64_t get_word_frequency(COLLOCATORDB *db, uint64_t w);
v1.5.0 (2026-07-31)
get_corpus_size(), which returns the total token countget_word_frequency(), which returns the absolute frequency of a word in the corpusv1.4.0 (2024-11-23)
collocatordb_query command line toolget_word_id(), which returns the ID of a wordmd_nws MI² score based on nominal window size (=10) instead of actual window size, for which only positions are counted where the collocate actually occursv1.3.2 (2024-11-15)
get_version(), which returns version stringv1.3.1 (2024-11-14)
Based on RocksDB, CollocatorDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.