Do not lose increments
Indexing threw counts away in two ways, both without a word.
rocksdb was asked to write with low_pri and no_slowdown, which means "cancel
this write when compaction is behind" and "do not wait, report it instead", and
the reported status was discarded in both inc() overloads. Measured with 3
million increments against small write buffers: 2,253,066 arrived, 746,934 were
gone, a quarter of them. The writer no longer deprioritises itself and waits
when it has to, and merge_one() repeats a cancelled write in any case. A
cancelled write was not applied, so repeating it cannot count twice.
The second one is the end of a run. The database has no write ahead log, and
nothing ever closed it, so everything still in a write buffer was lost when the
indexer ended: of 2 million increments in a process that just exited, none were
readable afterwards. close_collocatordb() writes them and closes the database,
and the destructor does the same.
While testing this, two more:
- get_collocators() and get_collocation_scores() finish a collocate when the
next one starts, so the last one was never returned. Every word was missing
a collocate, in the API and in the JSON.
- the array both of them return was allocated with c.size() + sizeof c[0]
instead of c.size() * sizeof c[0], far too small for more than one entry.
Callers reading beyond the first entry read whatever was behind it. It is
terminated with an empty entry now, so that they can tell where it ends.
The settings are aimed at what indexing does, a long stream of merge operands
for the same keys: bigger write buffers, several of them combined before they
are written, which collapses the operands early, and compaction that is allowed
to keep up. All of it can be set from the environment, to be able to tune a run
of several days without recompiling. It also explains why more indexing threads
stop helping: rocksdb inserts merge operands one writer at a time.
The test that found all of this is part of the suite now. The expectations of
the writing test were corrected, they described the missing collocate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I283b073b7f3fd36aa3ce960978e82105b1e44355
diff --git a/tests/basic_test.c b/tests/basic_test.c
index 7f33b59..75eae5b 100644
--- a/tests/basic_test.c
+++ b/tests/basic_test.c
@@ -106,10 +106,77 @@
inc_collocator(cdb, 1, 2, 4); size++;
COLLOCATOR *c = get_collocators(cdb, 0);
TEST_ASSERT(c != NULL);
- TEST_CHECK(c[0].w2 == 1);
- TEST_CHECK(c[0].raw == 2001);
- TEST_CHECK(c[0].left_raw == 200);
- TEST_CHECK(c[0].right_raw == 200);
+ /* Both collocates of word0 have to be there, in whatever order they are
+ sorted into: word1 with 2001 and word2 with 2000. The one that came last
+ used to be dropped, which is why this only asked for the first one. */
+ int n, seen1 = 0, seen2 = 0;
+ for (n = 0; c[n].raw > 0; n++) {
+ if (c[n].w2 == 1) {
+ seen1 = 1;
+ TEST_CHECK(c[n].raw == 2001);
+ TEST_CHECK(c[n].left_raw == 200);
+ TEST_CHECK(c[n].right_raw == 200);
+ } else if (c[n].w2 == 2) {
+ seen2 = 1;
+ TEST_CHECK(c[n].raw == 2000);
+ }
+ }
+ TEST_CHECK(n == 2);
+ TEST_MSG("expected 2 collocates, got %d", n);
+ TEST_CHECK(seen1 && seen2);
+
+ rmrf(rocksdbfn);
+}
+
+/* Every increment has to end up in the database, also when rocksdb would
+ rather cancel the write because compaction is behind, and also when they are
+ still in memory when the process that wrote them is gone. Both used to lose
+ counts silently: a quarter of them under write pressure, and everything that
+ had not been flushed when the indexer ended. */
+void test_no_increment_is_lost() {
+ char tmp_template[] = "/tmp/tmpfileXXXXXX";
+ int fd = mkstemp(tmp_template);
+ if (fd == -1) {
+ perror("mkstemp");
+ exit(EXIT_FAILURE);
+ }
+ close(fd);
+ char *tmp = strdup(tmp_template);
+ char rocksdbfn[1024], vocabfn[1024];
+ const long increments = 200000;
+ const int collocates = 100;
+ long i, total = 0;
+
+ snprintf(rocksdbfn, sizeof(rocksdbfn), "%s.rocksdb", tmp);
+ snprintf(vocabfn, sizeof(vocabfn), "%s.vocab", tmp);
+ FILE *h = fopen(vocabfn, "w");
+ for (i = 0; i <= collocates + 1; i++)
+ fprintf(h, "word%ld 1000\n", i);
+ fclose(h);
+
+ /* small write buffers, so that the writer runs into flushes and stalls */
+ setenv("COLLOCATORDB_WRITE_BUFFER_MB", "1", 1);
+ setenv("COLLOCATORDB_WRITE_BUFFERS", "2", 1);
+
+ COLLOCATORDB *cdb = open_collocatordb_for_write(rocksdbfn);
+ TEST_ASSERT(cdb != NULL);
+ for (i = 0; i < increments; i++)
+ inc_collocator(cdb, 1, 2 + (i % collocates), 1); /* never w2 == w1 */
+ close_collocatordb(cdb); /* has to write what is still in memory */
+
+ unsetenv("COLLOCATORDB_WRITE_BUFFER_MB");
+ unsetenv("COLLOCATORDB_WRITE_BUFFERS");
+
+ cdb = open_collocatordb(tmp);
+ TEST_ASSERT(cdb != NULL);
+ COLLOCATOR *c = get_collocators(cdb, 1);
+ TEST_ASSERT(c != NULL);
+ for (i = 0; i < collocates && c[i].raw > 0; i++)
+ total += c[i].raw;
+ TEST_CHECK(i == collocates);
+ TEST_MSG("only %ld of %d collocates are in the database", i, collocates);
+ TEST_CHECK(total == increments);
+ TEST_MSG("%ld of %ld increments survived", total, increments);
rmrf(rocksdbfn);
}
@@ -160,6 +227,7 @@
{ "collocation analysis", test_collocation_analysis },
{ "collocation analysis as json", test_collocation_analysis_as_json },
{ "writing", test_writing },
+ { "no increment is lost", test_no_increment_is_lost },
{ "version function", test_version_function },
{ "get word id", test_get_word_id },
{ "get corpus size", test_get_corpus_size},