)]}'
{
  "commit": "2f65bcf4181630bd55ddc8123579df3ea0c0afd5",
  "tree": "51a901b5519cd2bb84d621b9fcf119dd56e8935b",
  "parents": [
    "1d76ed9ffa72aaffb0608b31651d793ea56dd51c"
  ],
  "author": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Sun Aug 02 14:04:37 2026 +0200"
  },
  "committer": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Sun Aug 02 14:04:37 2026 +0200"
  },
  "message": "Do not lose increments\n\nIndexing threw counts away in two ways, both without a word.\n\nrocksdb was asked to write with low_pri and no_slowdown, which means \"cancel\nthis write when compaction is behind\" and \"do not wait, report it instead\", and\nthe reported status was discarded in both inc() overloads. Measured with 3\nmillion increments against small write buffers: 2,253,066 arrived, 746,934 were\ngone, a quarter of them. The writer no longer deprioritises itself and waits\nwhen it has to, and merge_one() repeats a cancelled write in any case. A\ncancelled write was not applied, so repeating it cannot count twice.\n\nThe second one is the end of a run. The database has no write ahead log, and\nnothing ever closed it, so everything still in a write buffer was lost when the\nindexer ended: of 2 million increments in a process that just exited, none were\nreadable afterwards. close_collocatordb() writes them and closes the database,\nand the destructor does the same.\n\nWhile testing this, two more:\n\n- get_collocators() and get_collocation_scores() finish a collocate when the\n  next one starts, so the last one was never returned. Every word was missing\n  a collocate, in the API and in the JSON.\n- the array both of them return was allocated with c.size() + sizeof c[0]\n  instead of c.size() * sizeof c[0], far too small for more than one entry.\n  Callers reading beyond the first entry read whatever was behind it. It is\n  terminated with an empty entry now, so that they can tell where it ends.\n\nThe settings are aimed at what indexing does, a long stream of merge operands\nfor the same keys: bigger write buffers, several of them combined before they\nare written, which collapses the operands early, and compaction that is allowed\nto keep up. All of it can be set from the environment, to be able to tune a run\nof several days without recompiling. It also explains why more indexing threads\nstop helping: rocksdb inserts merge operands one writer at a time.\n\nThe test that found all of this is part of the suite now. The expectations of\nthe writing test were corrected, they described the missing collocate.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\nChange-Id: I283b073b7f3fd36aa3ce960978e82105b1e44355\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "771e90e375ff47b34e48ae4ee70a0f60c515334c",
      "old_mode": 33188,
      "old_path": "CMakeLists.txt",
      "new_id": "912a49b28c209bfc8f99333fc30e9ca183925a7c",
      "new_mode": 33188,
      "new_path": "CMakeLists.txt"
    },
    {
      "type": "modify",
      "old_id": "00711f4dd87996a0df3926be0fa3a82076873b0a",
      "old_mode": 33188,
      "old_path": "README.md",
      "new_id": "749ca633ebdf74a02de6156ebe45fcb831c248a2",
      "new_mode": 33188,
      "new_path": "README.md"
    },
    {
      "type": "modify",
      "old_id": "1e0a8ec0c86551503eb8d8f4ac3ccd39c8601cc9",
      "old_mode": 33188,
      "old_path": "src/collocatordb.cc",
      "new_id": "79d038dacbd4a0420d5fa7794ab2aa29369f7b2a",
      "new_mode": 33188,
      "new_path": "src/collocatordb.cc"
    },
    {
      "type": "modify",
      "old_id": "74f2f0ec2d4de8ab6818c7498e27bcd846d3d7af",
      "old_mode": 33188,
      "old_path": "src/collocatordb.h",
      "new_id": "adf764df9beb8d54ce138a3c556e6a051074ff4f",
      "new_mode": 33188,
      "new_path": "src/collocatordb.h"
    },
    {
      "type": "modify",
      "old_id": "7f33b59625d5f1b7b5a8320ea421448d55f4a415",
      "old_mode": 33188,
      "old_path": "tests/basic_test.c",
      "new_id": "75eae5bf159a694796830c92129e2228ae24f681",
      "new_mode": 33188,
      "new_path": "tests/basic_test.c"
    }
  ]
}