Changelog

[0.98] - 2026-08-21

the per request diagnostics of the vector code are off unless DEREKOVECS_DEBUG is set. Every neighbourhood request wrote a few hundred lines - one per window position, one per vocabulary lookup, the whole JSON of a similar profile - which came to a million lines a day on corpora.ids-mannheim.de and filled the log partition of the machine. Startup and error messages are unaffected
the per request narration of the perl side - which word is being looked up, which result comes from a cache, which static file was fetched - is logged at debug level now. It repeated what the access log line of the same request already says

crawlers that collect training data for language models are answered with 403 before anything is computed. They walk the parameterised URLs, which are a computed result per word of the vocabulary and cannot be cached - the instance on corpora.ids-mannheim.de answers about 21000 of them a day. The list of user agents can be replaced with robots => {block_user_agents => '...'} in the configuration file, an empty value switches the blocking off
every response to a request with parameters carries X-Robots-Tag: noindex, nofollow, and the result pages carry the matching <meta name="robots">, so that search engines index the entry page only
robots.txt is served at every path, for installations that own their host name; behind a proxy that mounts derekovecs below a path, the site wide robots.txt still has to say it

the docker image runs as any user now. It only started as root before, because Inline needs a writable cache directory and the image has no compiler left to build the XS part elsewhere. Deployments that run it as a specific user, e.g. because the models come from an NFS export with root_squash, should point pid_file at an absolute path outside the working directory
fixed the docker image build, which failed since alpine moved to a gcc that defaults to C23, in which the bundled isaac code of Session::Token does not compile anymore
CI: the docker slimming uses mintoolkit, the successor of slimtoolkit, whose invocation had changed; the image tarball carries the version in its name
.dockerignore did not exclude .gitlab-ci-local, so local image builds packed the whole CI scratch directory, including job logs, into the image

requires collocatordb >= 1.5.0, which fixes the leaks in the collocator lookup, 3.6 kB per uncached collocator request, and builds against the rocksdb of the distribution
CI and docker image install the rocksdb package instead of building rocksdb 5.11 from source
Devel::CheckLib, which Makefile.PL needs to find libcollocatordb before it can report the dependencies, is bundled in inc/ now, so that a checkout configures without installing it first

fixed a crash on out of range word ranks: a single request with a w parameter beyond the vocabulary size, or a getVecsByRanks request with such a rank, killed the worker process
fixed w2v.merge, which died at startup with "Undefined subroutine &main::mergeVectors", and the lookup of count based collocators and precomputed profiles for merged models, which used unshifted ranks
fixed getVecsByRanks, which answered every request with "Undefined subroutine &main::getVecs", so the collocator SOM never got its vectors, and which was only reachable below the /derekovecs prefix
fixed memory leaks in the request path: per request the server leaked the target sum array, the thread handles, the target word list and the collocator results of all window threads, plus every string and array returned from C to perl (Inline::C copies those but never frees them)
the result caches are size limited now and configurable via the new cache configuration section; the neighbourhood cache never hit before because lookup and store used different keys
added w2v.compare_to configuration option to configure a derekovecs instance to compare results to (currently only count-based LogDice-delta supported)