tree bab44688a0ecfbc487dc94cf7e3f85d65604e5fa
parent 5b8aaf7556e4c5f677d51601c3f1c15e90557516
author Marc Kupietz <kupietz@ids-mannheim.de> 1788693437 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788693437 +0200

Search the embedding space with vector expressions

"König - Mann + Frau" now looks for the neighbours of
vec(König) - vec(Mann) + vec(Frau) instead of the neighbours of a single
word. A '+' or '-' is an operator only at the beginning of a token, so
hyphenated words like "Nord-Süd-Dialog" stay searchable, and blank
separated words keep entering the query with a '+', as before.

_get_neighbours() already had the branch for it, but nothing ever filled
wl->sep: the wordlist came from malloc() and the signs were read out of
uninitialised memory, so a multi word query subtracted operands at
random. It is filled now, and its indexing corrected - sep[b] is the sign
of operand b, not sep[b-1].

The predictive collocators answer for the whole query as well.
getCollocators() read the input weights of wl->wordi[0], the first
operand, and nothing said so: "Haus Auto" answered with the collocators
of Haus, "Auto Haus" with those of Auto. The score of a candidate is
sigmoid(q . syn1neg[collocate, position]), linear in q before the
sigmoid, so q can be the signed combination the paradigmatic side
searches around rather than the vector of one word.

The terms are averaged rather than summed. The sigmoid is informative
over a narrow range only - the strongest collocates of a single word
reach 0.994 to 0.997 on dereko-2026-ii, against the 0.9975 that MAX_EXP
allows - so a plain sum would push them into saturation. Dividing by the
number of terms keeps every query in the range MIN_RESP and the auto
focus are calibrated for, and leaves one word exactly as it was, divisor
one. A balanced analogy is one term as well, +1 -1 +1.

Those tails now saturate instead of being dropped. Activations outside
±MAX_EXP used to skip the collocate entirely, removing the strongest ones
from the list and from the position and target sums.

The count based collocators cannot follow: they are looked up per node in
the co-occurrence database. They used to be fetched for the first
paradigmatic neighbour, i.e. for the word nearest to the query vector,
which for an expression is not what was asked for. The interface says so
in place of that table now, and get_neighbours() reports the number of
operands so it can tell the two cases apart.

Operands outside the vocabulary are named above the result instead of
being dropped in silence. The vocabulary lookup only recognised a miss at
the very end of the vocabulary, so in a merged model a missing word
resolved to whatever sits at the boundary between the two vocabularies;
it returns -1 in both halves now.

Two more things the tokeniser uncovered. The best array was sized for
10 * max(N, 200) entries but sorted over N * para_threads, which fits
only as long as the syntagmatic threads take half of them - it is sized
for num_threads slices now. And its unfilled slots kept wordi == 0 from
the memset, which the result loop emitted as vocab[-1 * max_w] once a
thread found fewer candidates than were asked for; slots start at -1 and
the loop stops there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia91727eb669d73fc055adf652500464b8a743795