tree 51a901b5519cd2bb84d621b9fcf119dd56e8935b
parent 1d76ed9ffa72aaffb0608b31651d793ea56dd51c
author Marc Kupietz <kupietz@ids-mannheim.de> 1785672277 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1785672277 +0200

Do not lose increments

Indexing threw counts away in two ways, both without a word.

rocksdb was asked to write with low_pri and no_slowdown, which means "cancel
this write when compaction is behind" and "do not wait, report it instead", and
the reported status was discarded in both inc() overloads. Measured with 3
million increments against small write buffers: 2,253,066 arrived, 746,934 were
gone, a quarter of them. The writer no longer deprioritises itself and waits
when it has to, and merge_one() repeats a cancelled write in any case. A
cancelled write was not applied, so repeating it cannot count twice.

The second one is the end of a run. The database has no write ahead log, and
nothing ever closed it, so everything still in a write buffer was lost when the
indexer ended: of 2 million increments in a process that just exited, none were
readable afterwards. close_collocatordb() writes them and closes the database,
and the destructor does the same.

While testing this, two more:

- get_collocators() and get_collocation_scores() finish a collocate when the
  next one starts, so the last one was never returned. Every word was missing
  a collocate, in the API and in the JSON.
- the array both of them return was allocated with c.size() + sizeof c[0]
  instead of c.size() * sizeof c[0], far too small for more than one entry.
  Callers reading beyond the first entry read whatever was behind it. It is
  terminated with an empty entry now, so that they can tell where it ends.

The settings are aimed at what indexing does, a long stream of merge operands
for the same keys: bigger write buffers, several of them combined before they
are written, which collapses the operands early, and compaction that is allowed
to keep up. All of it can be set from the environment, to be able to tune a run
of several days without recompiling. It also explains why more indexing threads
stop helping: rocksdb inserts merge operands one writer at a time.

The test that found all of this is part of the suite now. The expectations of
the writing test were corrected, they described the missing collocate.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I283b073b7f3fd36aa3ce960978e82105b1e44355