Changelog

[1.00] - 2026-09-06

  • the search field understands vector arithmetic: König - Mann + Frau looks for the neighbours of the position vec(König) - vec(Mann) + vec(Frau) instead of the position of a single word. A + or - is an operator only at the beginning of a token, so hyphenated words such as Nord-Süd-Dialog are still found, and blank separated words keep entering the query with a + as before. Operands that are not in the vocabulary are named above the result, rather than silently dropped from the query
  • the predictive collocators answer for the whole query instead of for its first operand. getCollocators() read the input weights of wl->wordi[0], so "Haus Auto" answered with the collocators of Haus and "Auto Haus" with those of Auto, without anything saying so. It builds the same signed combination the paradigmatic side searches around, which the score allows because it is sigmoid(q . syn1neg[collocate, position]) and thus linear in q before the sigmoid. The terms are averaged rather than summed: the sigmoid is informative over a narrow range, and the strongest collocates of one word sit at the top of it already, so a plain sum would push them all into saturation. One word, and a balanced analogy, are one term and keep the previous scale
  • the tails of that score saturate rather than being dropped. Activations outside ±MAX_EXP used to skip the collocate entirely, which lost exactly the strongest ones and left them out of the position and target sums as well
  • the predictive collocator table has a max(z) column, the score before the sigmoid, i.e. the dot product of the query vector with the output weights of the collocate. max(a) is its image under expTable, which buckets it at 1/83 and caps it at MAX_EXP, so it cannot separate the strongest collocates from each other: of 1000 rows for "König - Mann + Frau" only 264 have a distinct max(a), and four sit on the cap. The dot product was computed for every candidate and thrown away at that lookup; it is kept now, and sorting the column uses the unrounded value
  • the count based collocators stay restricted to a single word form, since they are looked up per node in the co-occurrence database. For a query of several operands the interface says so in place of that table, rather than showing the profile of the word nearest to the query vector
  • a word that is not in the vocabulary of a merged model is reported as not found. The lookup only recognised a miss at the end of the whole vocabulary, so in a merged model a missing word used to resolve to whatever sits at the boundary between the two vocabularies

[0.99] - 2026-09-04

  • requires collocatordb 1.8.0, in which Log-Dice is computed as Rychlý (2008) defines it, without the window size factor that used to be applied to the frequency of the node. LD values are higher than before, by up to the binary logarithm of the window size, and comparable to those of other tools such as Sketch Engine. Building against an older collocatordb would show the previous values under the new explanations, which is why the minimum version is enforced
  • LL values change as well: its contingency table used to scale only the row total with the window size, leaving cells that do not add up to one sample. Following Evert (2004), the sample consists of co-occurrence tokens, so the sample size and both marginals scale with the window. Thanks to Tim Feldmüller for spotting this. MI, MI², MI³ and nPMI are unaffected, as they only need the expected co-occurrence frequency, which is unchanged
  • LDaf and the order of the collocates are unchanged. It is the auto focus score, not Log-Dice of the auto focus window, and its denominator grows with the width of the selected window, which is what makes it sensitive to how concentrated a pair is. Its tooltip gives the formula and notes that it is therefore not on the same scale as LD
  • the info tab explains that LL, MI, MI², MI³ and nPMI are computed over the attested positions, i.e. those in which the collocate occurs at least once, rather than over the whole context window, and that their values are therefore not directly comparable to those of tools computing with a fixed window size. The term auto focus is reserved for the selection of positions that yields the highest score, as Belica coined it for the CCDB

[0.98] - 2026-08-21

  • the per request diagnostics of the vector code are off unless DEREKOVECS_DEBUG is set. Every neighbourhood request wrote a few hundred lines - one per window position, one per vocabulary lookup, the whole JSON of a similar profile - which came to a million lines a day on corpora.ids-mannheim.de and filled the log partition of the machine. Startup and error messages are unaffected
  • the per request narration of the perl side - which word is being looked up, which result comes from a cache, which static file was fetched - is logged at debug level now. It repeated what the access log line of the same request already says

[0.97] - 2026-08-21

  • crawlers that collect training data for language models are answered with 403 before anything is computed. They walk the parameterised URLs, which are a computed result per word of the vocabulary and cannot be cached - the instance on corpora.ids-mannheim.de answers about 21000 of them a day. The list of user agents can be replaced with robots => {block_user_agents => '...'} in the configuration file, an empty value switches the blocking off
  • every response to a request with parameters carries X-Robots-Tag: noindex, nofollow, and the result pages carry the matching <meta name="robots">, so that search engines index the entry page only
  • robots.txt is served at every path, for installations that own their host name; behind a proxy that mounts derekovecs below a path, the site wide robots.txt still has to say it

[0.96] - 2026-07-31

  • the docker image runs as any user now. It only started as root before, because Inline needs a writable cache directory and the image has no compiler left to build the XS part elsewhere. Deployments that run it as a specific user, e.g. because the models come from an NFS export with root_squash, should point pid_file at an absolute path outside the working directory
  • fixed the docker image build, which failed since alpine moved to a gcc that defaults to C23, in which the bundled isaac code of Session::Token does not compile anymore
  • CI: the docker slimming uses mintoolkit, the successor of slimtoolkit, whose invocation had changed; the image tarball carries the version in its name
  • .dockerignore did not exclude .gitlab-ci-local, so local image builds packed the whole CI scratch directory, including job logs, into the image

[0.95] - 2026-07-31

  • requires collocatordb >= 1.5.0, which fixes the leaks in the collocator lookup, 3.6 kB per uncached collocator request, and builds against the rocksdb of the distribution
  • CI and docker image install the rocksdb package instead of building rocksdb 5.11 from source
  • Devel::CheckLib, which Makefile.PL needs to find libcollocatordb before it can report the dependencies, is bundled in inc/ now, so that a checkout configures without installing it first

[0.94] - 2026-07-30

  • fixed a crash on out of range word ranks: a single request with a w parameter beyond the vocabulary size, or a getVecsByRanks request with such a rank, killed the worker process
  • fixed w2v.merge, which died at startup with "Undefined subroutine &main::mergeVectors", and the lookup of count based collocators and precomputed profiles for merged models, which used unshifted ranks
  • fixed getVecsByRanks, which answered every request with "Undefined subroutine &main::getVecs", so the collocator SOM never got its vectors, and which was only reachable below the /derekovecs prefix
  • fixed memory leaks in the request path: per request the server leaked the target sum array, the thread handles, the target word list and the collocator results of all window threads, plus every string and array returned from C to perl (Inline::C copies those but never frees them)
  • the result caches are size limited now and configurable via the new cache configuration section; the neighbourhood cache never hit before because lookup and store used different keys
  • added w2v.compare_to configuration option to configure a derekovecs instance to compare results to (currently only count-based LogDice-delta supported)

[0.93.2] - 2024-11-15

  • fixed calculation of total token count by using collocatordb 1.3.2

[0.93.1] - 2024-09-13

  • usage call warning fixed
  • switched to alpine base image for docker
  • slimmed down docker image

[0.93] - 2024-08-28

  • export buttons added to all data tables
  • (downtime) announcements can be fetched from Nextcloud calendar
  • fixed KorAP queries for alternatives
  • fixed distance operator in KorAP queries
  • fixed build with gcc 14.2 and clang 18
  • fixed MacOS build
  • KorAP instance url is now configurable
  • CoRoLa reference added to info tab

[0.92.1] - 2023-11-06

  • fixed json encoding of double quotes in getClassicCollocators
  • shows derekovecs version in page footer
  • changelog added

[0.92] - 2023-11-05

  • new WS function getVocabSize (internally getVocabSize())
  • new WS function getServerVersion
  • new WS function getModelName
  • Announce scheduled downtimes in info tab