Limit the size of the result caches
The neighbourhood, classic collocator and similar profile caches grew by one
entry per queried word and were never evicted, so a long running worker
eventually exhausted the machine's memory. They are size limited Mojo::Cache
instances now, configurable in the new cache section of the configuration
file.
The neighbourhood cache never hit anyway: the lookup used a comma inside the
hash subscript, i.e. a $; joined multidimensional key, the value was read with
a second key and stored under a third one that did not include
searchBaseVocabFirst at all.
Starting the server with -C now really bypasses the caches. It did not before,
because $opt_C was read in IDS::DeReKoVecs::Read but set in main.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I17b801df54137ea81b660229bf789725de5a2b1b
diff --git a/example.conf b/example.conf
index ddc1d2f..34d6297 100644
--- a/example.conf
+++ b/example.conf
@@ -12,6 +12,15 @@
workers => 0
},
+ # Per worker in-memory caches. The total footprint is workers * max_keys *
+ # size of a cached result, so keep max_keys small - neighbourhood results
+ # contain one vector per neighbour. 0 disables the respective cache.
+ cache => {
+ max_keys => 50, # neighbourhood (paradigmatic/syntagmatic) results
+ collocators_max_keys => 200, # classic collocator profiles (JSON)
+ profiles_max_keys => 200 # similar profiles (JSON)
+ },
+
w2v => {
vecs => "example-models/wpd19_10000/wpd19_10000.vecs",
# compare_to => "https://corpora.ids-mannheim.de/openlab/derekovecs", # compare results to this derekovecs instance