)]}' { "log": [ { "commit": "2f65bcf4181630bd55ddc8123579df3ea0c0afd5", "tree": "51a901b5519cd2bb84d621b9fcf119dd56e8935b", "parents": [ "1d76ed9ffa72aaffb0608b31651d793ea56dd51c" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sun Aug 02 14:04:37 2026 +0200" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sun Aug 02 14:04:37 2026 +0200" }, "message": "Do not lose increments\n\nIndexing threw counts away in two ways, both without a word.\n\nrocksdb was asked to write with low_pri and no_slowdown, which means \"cancel\nthis write when compaction is behind\" and \"do not wait, report it instead\", and\nthe reported status was discarded in both inc() overloads. Measured with 3\nmillion increments against small write buffers: 2,253,066 arrived, 746,934 were\ngone, a quarter of them. The writer no longer deprioritises itself and waits\nwhen it has to, and merge_one() repeats a cancelled write in any case. A\ncancelled write was not applied, so repeating it cannot count twice.\n\nThe second one is the end of a run. The database has no write ahead log, and\nnothing ever closed it, so everything still in a write buffer was lost when the\nindexer ended: of 2 million increments in a process that just exited, none were\nreadable afterwards. close_collocatordb() writes them and closes the database,\nand the destructor does the same.\n\nWhile testing this, two more:\n\n- get_collocators() and get_collocation_scores() finish a collocate when the\n next one starts, so the last one was never returned. Every word was missing\n a collocate, in the API and in the JSON.\n- the array both of them return was allocated with c.size() + sizeof c[0]\n instead of c.size() * sizeof c[0], far too small for more than one entry.\n Callers reading beyond the first entry read whatever was behind it. It is\n terminated with an empty entry now, so that they can tell where it ends.\n\nThe settings are aimed at what indexing does, a long stream of merge operands\nfor the same keys: bigger write buffers, several of them combined before they\nare written, which collapses the operands early, and compaction that is allowed\nto keep up. All of it can be set from the environment, to be able to tune a run\nof several days without recompiling. It also explains why more indexing threads\nstop helping: rocksdb inserts merge operands one writer at a time.\n\nThe test that found all of this is part of the suite now. The expectations of\nthe writing test were corrected, they described the missing collocate.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\nChange-Id: I283b073b7f3fd36aa3ce960978e82105b1e44355\n" }, { "commit": "65d4479113439e58d8b53d9a1f2cfb14fcdfab81", "tree": "0520bdabbd4a30f8018a39b8eb0182c70ad0543b", "parents": [ "023d4591630964633a5547c0f086109948e36927" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Jul 31 09:46:34 2026 +0900" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Jul 31 09:46:34 2026 +0900" }, "message": "Support rocksdb 11\n\nrocksdb 11 needs C++20 for its headers and hands the database back from\nDB::Open() and DB::OpenForReadOnly() as a unique_ptr instead of a raw pointer.\nThe newest standard the compiler knows is used now, and the two open calls go\nthrough a macro that adapts to the rocksdb version, so 7, 10 and 11 all build.\n\nVerified with rocksdb 7.8.3 (debian), 10.2.1 (fedora) and 11.0.4 (alpine,\nmusl): no warnings, ctest passes, and reading a database written by rocksdb\n5.11 still gives identical results for 251 word ids.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\nChange-Id: If16e6aa2fbe8478f3260e358dce53b2dc5144d9d\n" }, { "commit": "aa354d8fba290e526c084ff49a8a226aa260f9af", "tree": "2dd3bcb6ed29c405d3f6fcf56adce6daae2bdfab", "parents": [ "b61a72a726f10b555057f4ffd6173ddb32e560e0" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Jul 31 09:17:57 2026 +0900" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Jul 31 09:17:57 2026 +0900" }, "message": "Fix two memory leaks in the collocator lookup\n\nEvery collocator lookup leaked a rocksdb iterator and 80 bytes of window sums:\n\nCollocatorIterator takes the iterator that SeekIterator() gets from\nDB::NewIterator(), but had no destructor, so the compiler generated one that\ndoes not touch the raw pointer. The wrapper was deleted by its unique_ptr, the\niterator it wrapped was not, together with everything it pins.\n\nget_collocators() allocated its fixed size sumWindow array with malloc() and\nnever freed it. It is a fixed size, so it lives on the stack now.\n\nMeasured with 20000 calls of get_collocators_as_json() on the test database:\n3.63 kB per call before, no measurable growth at all afterwards. The 80 bytes\nwere per call regardless of the result size, which is what gave them away.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\nChange-Id: I1b1148ada81bd821784b8e327334fc3bafe29e1b\n" }, { "commit": "b61a72a726f10b555057f4ffd6173ddb32e560e0", "tree": "edd090cd346f2953e43178f092d379d12329b29a", "parents": [ "0fde12e5004192bf6e07a33ddb347164f3aff9f1" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Mon Feb 10 15:48:41 2025 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Mon Feb 10 15:48:41 2025 +0100" }, "message": "Revert \"Increase compaction limits\"\n\nToo good to be true\n\nThis reverts commit 0fde12e5004192bf6e07a33ddb347164f3aff9f1.\n" }, { "commit": "0fde12e5004192bf6e07a33ddb347164f3aff9f1", "tree": "ae580d31d4ae1b537984ef606f9dd02195c6bde0", "parents": [ "c630c15e676b91c53c3cadeb2ee87055077859bb" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Jan 23 12:32:32 2025 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Jan 23 12:36:54 2025 +0100" }, "message": "Increase compaction limits\n\nIncreases write throughput by factor 10 on compute2\n\nCreating the full collocation db for DeReKo-2025-I only takes less then\none day, now.\n\nThanks @benhur1999\n\nChange-Id: I68eb18eab2358432aceacc98acbf930a19e76b90\n" }, { "commit": "c630c15e676b91c53c3cadeb2ee87055077859bb", "tree": "edd090cd346f2953e43178f092d379d12329b29a", "parents": [ "21b964cba850116ee4bfe99950dff28f5c51cf6d" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Jan 23 11:17:47 2025 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Jan 23 12:03:50 2025 +0100" }, "message": "Increase writing performance\n\nIncreases write throughput\n\nfrom 350kb/s to 550kb/s on compute2 and\nfrom 560kb/s to 760kb/s on my office computer\n\nboth writing to ssd.\n\ncompute: \t\t 635kb/s\nkorap-worker-08: 590kb/s\n\nChange-Id: I7c8f6c7777c811bf18b5fffdd12d1bd862107ebb\n" }, { "commit": "21b964cba850116ee4bfe99950dff28f5c51cf6d", "tree": "eb99ec170ac714c0e5c5ec4231f44834536c9f79", "parents": [ "d26b105331ccab37288bfafe80cbb67bc7c99a3d" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Tue Dec 10 17:10:50 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Tue Dec 10 17:10:50 2024 +0100" }, "message": "Add get_word_frequency function\n\nChange-Id: Idef55f5a8f84c4bcba0a1bfdbe563eb74f26857a\n" }, { "commit": "d26b105331ccab37288bfafe80cbb67bc7c99a3d", "tree": "c29b01622de564840770e65297502ace9e28d417", "parents": [ "fa73154586cd642ee019afd58e8a8eb67f9a444a" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Tue Dec 10 16:56:39 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Tue Dec 10 16:56:39 2024 +0100" }, "message": "Add get_corpus_size function\n\nChange-Id: Ie95b0f14bb8784006e9dc788d02042b6b5b9485a\n" }, { "commit": "e889cec5dd3a1b39ede361ac1e7cdd73ccb258f3", "tree": "0a81f43e18838f99d9e9cf9c7ecea7b4b1739b76", "parents": [ "b4ade0f87d230ed6eaeff56afc9c2c0dd9fed033" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Nov 23 12:08:42 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Nov 23 12:08:42 2024 +0100" }, "message": "Add `md_nws` MI² score based on nominal window size (\u003d10)\n\nChange-Id: I5e431a8ba0f863191f378d447621f3a10039d122\n" }, { "commit": "39887082c445671ea721745d5d306f8946420580", "tree": "fe106a29269f150a9c0ff7db6fc30e55c113a668", "parents": [ "0347ad6201d8d84eaff9d1ccd40146910e044739" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 22 18:06:20 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 22 18:21:10 2024 +0100" }, "message": "clean up code clang style\n\nChange-Id: Ib8c2960f276107f7fe5dbf1699ff5b20f1315166\n" }, { "commit": "979580e1c2eb917e8e8010941c50c2a4ff97b36e", "tree": "a712b0e6fa903137f79ca1d072651f74ec266050", "parents": [ "f08631e680931eeafd340369cfee7fb7a5e360dc" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Nov 21 18:05:07 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Nov 21 18:05:07 2024 +0100" }, "message": "Add get_word_id(char* word) method to library\n\nChange-Id: Idf0701248ccdf0ecc1c2265d215fbe582510ea96\n" }, { "commit": "75d35d27d07fc065d72cc8e85aef84f49f0852fd", "tree": "b85007cc2348f5f852dcc7638b30efb17ace0123", "parents": [ "9ac200953ac6e60651fdac0022822e3a27cbb343" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Nov 21 14:31:44 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Nov 21 14:39:43 2024 +0100" }, "message": "Fix include file for c++\n\nChange-Id: Ib8c2960f276107f7fe5dbf1699ff5b20f1315166\n" }, { "commit": "b91d5325b0952eb91df2071d35ff411bc999ee9a", "tree": "3f76a3165df5a815fec707e13c006a09f9befed5", "parents": [ "ce2d005b371bfdeca816c7797f48c509f3753df4" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 15 16:04:32 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 15 16:04:32 2024 +0100" }, "message": "add missing config.h.in\n\nChange-Id: I7f294d3a0ce1acb961c49a8a9bee0f3e8e2824eb\n" }, { "commit": "6208fd77e556fdc9219868002589b79b616e476e", "tree": "5bb582ff22617cd4babee90dc374e7a274bd46a9", "parents": [ "5ffc474b237885bf81026c63a9a89683691fd5d9" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 15 15:46:19 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 15 15:53:31 2024 +0100" }, "message": "add get_version function\n\nChange-Id: I7f294d3a0ce1acb961c49a8a9bee0f3e8e2824eb\n" }, { "commit": "5ffc474b237885bf81026c63a9a89683691fd5d9", "tree": "9d3450127e825f5806ba4066d613e6681a3bf68f", "parents": [ "0a3a09ad479b9aa06a9ac09c57b8896d46ee8f97" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 15 15:45:12 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Fri Nov 15 15:52:11 2024 +0100" }, "message": "code clean up\n\nChange-Id: I6109c795a6734b040a034e4b5ee6a62cdb56310e\n" }, { "commit": "2441f7c09b781efd7393c7209103a2012796ffba", "tree": "17752dd136c76bad4bf55ebde97b7b838d117912", "parents": [ "442292368bedc801988a57d2c60d0369f7066e26" ], "author": { "name": "„feldmueller“", "email": "„feldmueller@posteo.de“", "time": "Thu Nov 14 16:31:30 2024 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Thu Nov 14 17:22:06 2024 +0100" }, "message": "Fix calculation of total token count\n\nChange-Id: Ib8c2960f276107f7fe5dbf1699ff5b20f1315166\n" }, { "commit": "442292368bedc801988a57d2c60d0369f7066e26", "tree": "2e90ddad91d3fc7765de603c5fbad38387b8919d", "parents": [ "13f4f1b6446bc1bf0efe2a972b3b6db536a94810" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Mon Aug 05 15:00:20 2024 +0200" }, "committer": { "name": "„feldmueller“", "email": "„feldmueller@posteo.de“", "time": "Thu Aug 08 10:20:50 2024 +0200" }, "message": "Improve MacOS support\n\nChange-Id: Id54a73c025d01ee37d2373a62506eb52a4f75d35\n" }, { "commit": "6663f11ebcbc2b676eba5cf6060af959cdebc0bb", "tree": "e63c7ec18e42d5a12b26718c3b446331c9054acc", "parents": [ "b4a683c0e64ca17c1a20cff91427b35a1d1ca533" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sun Mar 14 09:20:59 2021 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sun Mar 14 14:44:28 2021 +0100" }, "message": "Add first tests\n\nChange-Id: I7ef95cef912feb2b997e1c7da5eb5a369cbe56b1\n" }, { "commit": "b4a683c0e64ca17c1a20cff91427b35a1d1ca533", "tree": "9326620af9d79ba5e5653425fc9aab39e255ff38", "parents": [ "97f433b24b9051a436c614b8dc6d7c421828f41d" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sun Mar 14 09:19:44 2021 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sun Mar 14 09:19:44 2021 +0100" }, "message": "Extend API\n\nChange-Id: I326b2074645f43f81b68cd39eacfe7e247138b41\n" }, { "commit": "97f433b24b9051a436c614b8dc6d7c421828f41d", "tree": "e47097b805dbe42c9512d34598b47003cc8d25d0", "parents": [ "12af019e9e068e2150128b523f2c7f9bc330ebe0" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Mar 13 18:10:52 2021 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Mar 13 18:10:52 2021 +0100" }, "message": "Rename some identifiers in json result to avoid misunderstandings\n\nChange-Id: Ie5b39f5de7279643668a4b27da258b3aad0a6924\n" }, { "commit": "12af019e9e068e2150128b523f2c7f9bc330ebe0", "tree": "e13b3753f4da597e064568a0c94089f059fee7a4", "parents": [ "7042bd31e4ebc70b927add24848dba4ef8ceba49" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Mar 13 18:05:14 2021 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Mar 13 18:05:14 2021 +0100" }, "message": "Reformat code\n\nChange-Id: Id0ea8ec518278984697b6a3b087c532d1704184b\n" }, { "commit": "c2e43215a6916610288aa0b9c7fd800683ef37e5", "tree": "6567a65ac486b43d20d5116de9d47ecced7d5ac6", "parents": [ "0bffe807bf3e6bc8ce570967e222bb3e45db33db" ], "author": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Mar 13 18:05:14 2021 +0100" }, "committer": { "name": "Marc Kupietz", "email": "kupietz@ids-mannheim.de", "time": "Sat Mar 13 18:05:14 2021 +0100" }, "message": "Add some directory structure\n\nChange-Id: I052849c6aff9fd0f311984de67c35a306735c9a9\n" } ] }