tree b7d35939c63bdadaf5370b17f751cfcebb5c773c
parent d885c242e973e29c5604c04c08f07f58c6102cf0
author Marc Kupietz <kupietz@ids-mannheim.de> 1786288404 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1786288404 +0200

Batch merges into one rocksdb write per batch

Instead of one rocksdb write per increment, the increments of the indexing
threads are collected in a WriteBatch and written once per batch. The batch
size can be set with COLLOCATORDB_BATCH_SIZE (default 65536).

The batch is swapped out under the lock and written without it, so the other
threads keep counting while the write is in flight. With the lock held across
the write, an indexer with several threads was slower than with one, because
the threads queued up on the lock and rocksdb never saw concurrent writers
to batch into one.

A read writes out what is still in the batch first, so it sees every
increment that came before it. A cancelled write is repeated as before.

Change-Id: Id534c541a23dfc5b74435c6a88cc001ab6d842b9