| commit | 2f65bcf4181630bd55ddc8123579df3ea0c0afd5 | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Aug 02 14:04:37 2026 +0200 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Sun Aug 02 14:04:37 2026 +0200 |
| tree | 51a901b5519cd2bb84d621b9fcf119dd56e8935b | |
| parent | 1d76ed9ffa72aaffb0608b31651d793ea56dd51c [diff] |
Do not lose increments Indexing threw counts away in two ways, both without a word. rocksdb was asked to write with low_pri and no_slowdown, which means "cancel this write when compaction is behind" and "do not wait, report it instead", and the reported status was discarded in both inc() overloads. Measured with 3 million increments against small write buffers: 2,253,066 arrived, 746,934 were gone, a quarter of them. The writer no longer deprioritises itself and waits when it has to, and merge_one() repeats a cancelled write in any case. A cancelled write was not applied, so repeating it cannot count twice. The second one is the end of a run. The database has no write ahead log, and nothing ever closed it, so everything still in a write buffer was lost when the indexer ended: of 2 million increments in a process that just exited, none were readable afterwards. close_collocatordb() writes them and closes the database, and the destructor does the same. While testing this, two more: - get_collocators() and get_collocation_scores() finish a collocate when the next one starts, so the last one was never returned. Every word was missing a collocate, in the API and in the JSON. - the array both of them return was allocated with c.size() + sizeof c[0] instead of c.size() * sizeof c[0], far too small for more than one entry. Callers reading beyond the first entry read whatever was behind it. It is terminated with an empty entry now, so that they can tell where it ends. The settings are aimed at what indexing does, a long stream of merge operands for the same keys: bigger write buffers, several of them combined before they are written, which collapses the operands early, and compaction that is allowed to keep up. All of it can be set from the environment, to be able to tune a run of several days without recompiling. It also explains why more indexing threads stop helping: rocksdb inserts merge operands one writer at a time. The test that found all of this is part of the suite now. The expectations of the writing test were corrected, they described the missing collocate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Change-Id: I283b073b7f3fd36aa3ce960978e82105b1e44355
The rocksdb of the distribution is used, no particular version is required.
on Fedora, Rocky Linux, RHEL
sudo dnf install cmake gcc-c++ rocksdb-devel snappy-devel zlib-devel bzip2-devel lz4-devel libzstd-devel gflags-devel
on Ubuntu, Debian
sudo apt-get install cmake librocksdb-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev libomp-dev
on MacOS
brew install cmake rocksdb snappy zlib bzip2 lz4 zstd libomp gflags
Do not install another rocksdb below /usr/local next to the packaged one. The compiler looks into /usr/local/include before /usr/include, so its headers would be used while the library of the distribution is linked, which ends in a long list of undefined references. The build compares the version of the headers with the version of the library and stops with an explanation when they do not match.
git clone "https://korap.ids-mannheim.de/gerrit/ids-kl/collocatordb" cd collocatordb mkdir -p build cd build cmake -DCMAKE_INSTALL_PREFIX=/usr/local .. make && sudo make install && sudo ldconfig ctest --extra-verbose
The tests run after the installation, not before: one of them calls the collocatordb_query tool, which is built with the rpath of its install location and does not find the library as long as it is not installed. The build directory has to be build inside the sources, the tests look for the tool and for their data relative to it.
libcollocatordb_static.a is built as well. It is linked against the shared rocksdb unless a static rocksdb is found, which is enough for most purposes. A static rocksdb is only needed to link a program completely statically, as dereko2vec does, which is worth about 10% on collocator lookups.
Debian and Ubuntu ship librocksdb.a in librocksdb-dev, so there is nothing to do. Fedora, Rocky Linux and RHEL do not ship one, so it has to be built. Keep it out of /usr/local, where its headers would shadow those of the package, and point the build at it:
git clone https://github.com/facebook/rocksdb.git -b v$(rpm -q --qf '%{VERSION}' rocksdb) --single-branch cd rocksdb make -j $(nproc) static_lib make install-static INSTALL_PATH=$HOME/rocksdb-static cd ../collocatordb/build cmake -DROCKSDB_STATIC=$HOME/rocksdb-static/lib/librocksdb.a -DCMAKE_INSTALL_PREFIX=/usr/local .. make && sudo make install && sudo ldconfig
typedef struct { uint32_t w2; uint64_t f2; uint64_t raw; double pmi; double npmi; double llr; double lfmd; double md; double md_nws; uint64_t left_raw; uint64_t right_raw; double left_pmi; double right_pmi; double dice; double logdice; double ldaf; int window; int af_window; } COLLOCATOR ; COLLOCATORDB *open_collocatordb(const char *s); COLLOCATORDB *open_collocatordb_for_write(const char *s); void close_collocatordb(COLLOCATORDB *db); void inc_collocator(COLLOCATORDB *db, uint64_t w1, uint64_t w2, int8_t dist); void dump_collocators(COLLOCATORDB *db, uint32_t w1, uint32_t w2, int8_t dist); COLLOCATOR *get_collocators(COLLOCATORDB *db, uint32_t w1); COLLOCATOR *get_collocation_scores(COLLOCATORDB *db, uint32_t w1, uint32_t w2); char *get_collocators_as_json(COLLOCATORDB *db, uint32_t w1); char *get_collocation_scores_as_json(COLLOCATORDB *db, uint32_t w1, uint32_t w2); char *get_word(COLLOCATORDB *db, uint32_t w1); void read_vocab(COLLOCATORDB *db, char *fname); char *get_version(); uint64_t get_word_id(COLLOCATORDB *db, const char *word); uint64_t get_corpus_size(COLLOCATORDB *db); uint64_t get_word_frequency(COLLOCATORDB *db, uint64_t w);
Building a collocation database is one long stream of increments for the same keys. The defaults are chosen for that, and can be adjusted from the environment, so that a run of several days does not have to be recompiled to be tuned:
| variable | default | meaning |
|---|---|---|
COLLOCATORDB_WRITE_BUFFER_MB | 256 | size of one write buffer |
COLLOCATORDB_WRITE_BUFFERS | 8 | how many of them |
COLLOCATORDB_WRITE_BUFFERS_TO_MERGE | 4 | how many are combined before they are written, which collapses the increments of the same key early |
COLLOCATORDB_BACKGROUND_JOBS | cores, at most 32 | threads for flushing and compaction |
COLLOCATORDB_SUBCOMPACTIONS | 4 | how far a single compaction is spread over threads |
COLLOCATORDB_BLOCK_CACHE_MB | 512 | block cache, only relevant for reading |
Increments are inserted one writer at a time, rocksdb does not support concurrent memtable writes for merge operations, so more indexing threads stop helping at some point regardless of these settings.
The database has no write ahead log, so an indexer has to call close_collocatordb() when it is done, otherwise everything that has not been flushed is lost.
v1.6.0 (2026-08-02)
close_collocatordb(), which writes it and closes the databaseget_collocators() and get_collocation_scores() dropping the last collocate of every wordget_collocators() and get_collocation_scores() being far too small, + instead of * in the size. It is terminated with an empty entry nowv1.5.0 (2026-07-31)
get_corpus_size(), which returns the total token countget_word_frequency(), which returns the absolute frequency of a word in the corpusv1.4.0 (2024-11-23)
collocatordb_query command line toolget_word_id(), which returns the ID of a wordmd_nws MI² score based on nominal window size (=10) instead of actual window size, for which only positions are counted where the collocate actually occursv1.3.2 (2024-11-15)
get_version(), which returns version stringv1.3.1 (2024-11-14)
Based on RocksDB, CollocatorDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.