Search the embedding space with vector expressions
"König - Mann + Frau" now looks for the neighbours of
vec(König) - vec(Mann) + vec(Frau) instead of the neighbours of a single
word. A '+' or '-' is an operator only at the beginning of a token, so
hyphenated words like "Nord-Süd-Dialog" stay searchable, and blank
separated words keep entering the query with a '+', as before.
_get_neighbours() already had the branch for it, but nothing ever filled
wl->sep: the wordlist came from malloc() and the signs were read out of
uninitialised memory, so a multi word query subtracted operands at
random. It is filled now, and its indexing corrected - sep[b] is the sign
of operand b, not sep[b-1].
The predictive collocators answer for the whole query as well.
getCollocators() read the input weights of wl->wordi[0], the first
operand, and nothing said so: "Haus Auto" answered with the collocators
of Haus, "Auto Haus" with those of Auto. The score of a candidate is
sigmoid(q . syn1neg[collocate, position]), linear in q before the
sigmoid, so q can be the signed combination the paradigmatic side
searches around rather than the vector of one word.
The terms are averaged rather than summed. The sigmoid is informative
over a narrow range only - the strongest collocates of a single word
reach 0.994 to 0.997 on dereko-2026-ii, against the 0.9975 that MAX_EXP
allows - so a plain sum would push them into saturation. Dividing by the
number of terms keeps every query in the range MIN_RESP and the auto
focus are calibrated for, and leaves one word exactly as it was, divisor
one. A balanced analogy is one term as well, +1 -1 +1.
Those tails now saturate instead of being dropped. Activations outside
±MAX_EXP used to skip the collocate entirely, removing the strongest ones
from the list and from the position and target sums.
The count based collocators cannot follow: they are looked up per node in
the co-occurrence database. They used to be fetched for the first
paradigmatic neighbour, i.e. for the word nearest to the query vector,
which for an expression is not what was asked for. The interface says so
in place of that table now, and get_neighbours() reports the number of
operands so it can tell the two cases apart.
Operands outside the vocabulary are named above the result instead of
being dropped in silence. The vocabulary lookup only recognised a miss at
the very end of the vocabulary, so in a merged model a missing word
resolved to whatever sits at the boundary between the two vocabularies;
it returns -1 in both halves now.
Two more things the tokeniser uncovered. The best array was sized for
10 * max(N, 200) entries but sorted over N * para_threads, which fits
only as long as the syntagmatic threads take half of them - it is sized
for num_threads slices now. And its unfilled slots kept wordi == 0 from
the memset, which the result loop emitted as vocab[-1 * max_w] once a
thread found fewer candidates than were asked for; slots start at -1 and
the loop stops there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia91727eb669d73fc055adf652500464b8a743795
diff --git a/templates/en/about.html.ep b/templates/en/about.html.ep
index 20c0ecc..039c427 100644
--- a/templates/en/about.html.ep
+++ b/templates/en/about.html.ep
@@ -9,6 +9,33 @@
The models used here are based on an extension of word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
and on the other hand on simple co-occurence counts and analysis methods that operate on these.
</p>
+<h3>Queries</h3>
+<p>
+ Several word forms separated by blanks are added into one query vector, i.e. the search is
+ around the centre of the alternatives, while a “|” separates several independent
+ searches. Beyond that, “+” and “-” do arithmetic in the vector space:
+ <span class="mono">König - Mann + Frau</span> searches the neighbours of the position
+ vec(König) - vec(Mann) + vec(Frau) rather than the position of a single word. A sign is an
+ operator only at the beginning of a token, so hyphenated words such as
+ <span class="mono">Nord-Süd-Dialog</span> remain searchable.
+</p>
+<p>
+ The syntagmatic view follows along. The predictive score of a candidate is
+ \(\sigma(q \cdot u)\), which is linear in \(q\) before the sigmoid, so an expression
+ yields the contexts that the computed position predicts – for “König - Mann +
+ Frau” the ones König and Frau predict but Mann does not. The positive terms are
+ averaged rather than summed, so that the values stay in the range a single word form
+ occupies: the sigmoid is informative over a narrow range only, and the strongest collocates
+ of a single word already sit at the top of it. A balanced analogy (+1 -1 +1) is one term and
+ is therefore on the same scale as a single word form.
+</p>
+<p>
+ The count based collocators, in contrast, remain restricted to a single word form: they are
+ looked up per node in the co-occurrence database and cannot be computed from the query
+ vector. Operands that are not in the vocabulary are named above the result and left out of
+ the computation.
+</p>
+
<h3>Effective window size and auto focus</h3>
<p>
The association measures LL, MI, MI², MI³ and nPMI are not computed over the whole context
@@ -40,6 +67,11 @@
</p>
<h3>Changes</h3>
<p>
+ <strong>2026-09-06</strong>
+
+ The search field understands vector arithmetic, see <em>Queries</em> above.
+</p>
+<p>
<strong>2026-09-04</strong>
Log-Dice (LD) is now computed exactly as defined by Rychlý (2008),