Search the embedding space with vector expressions
"König - Mann + Frau" now looks for the neighbours of
vec(König) - vec(Mann) + vec(Frau) instead of the neighbours of a single
word. A '+' or '-' is an operator only at the beginning of a token, so
hyphenated words like "Nord-Süd-Dialog" stay searchable, and blank
separated words keep entering the query with a '+', as before.
_get_neighbours() already had the branch for it, but nothing ever filled
wl->sep: the wordlist came from malloc() and the signs were read out of
uninitialised memory, so a multi word query subtracted operands at
random. It is filled now, and its indexing corrected - sep[b] is the sign
of operand b, not sep[b-1].
The predictive collocators answer for the whole query as well.
getCollocators() read the input weights of wl->wordi[0], the first
operand, and nothing said so: "Haus Auto" answered with the collocators
of Haus, "Auto Haus" with those of Auto. The score of a candidate is
sigmoid(q . syn1neg[collocate, position]), linear in q before the
sigmoid, so q can be the signed combination the paradigmatic side
searches around rather than the vector of one word.
The terms are averaged rather than summed. The sigmoid is informative
over a narrow range only - the strongest collocates of a single word
reach 0.994 to 0.997 on dereko-2026-ii, against the 0.9975 that MAX_EXP
allows - so a plain sum would push them into saturation. Dividing by the
number of terms keeps every query in the range MIN_RESP and the auto
focus are calibrated for, and leaves one word exactly as it was, divisor
one. A balanced analogy is one term as well, +1 -1 +1.
Those tails now saturate instead of being dropped. Activations outside
±MAX_EXP used to skip the collocate entirely, removing the strongest ones
from the list and from the position and target sums.
The count based collocators cannot follow: they are looked up per node in
the co-occurrence database. They used to be fetched for the first
paradigmatic neighbour, i.e. for the word nearest to the query vector,
which for an expression is not what was asked for. The interface says so
in place of that table now, and get_neighbours() reports the number of
operands so it can tell the two cases apart.
Operands outside the vocabulary are named above the result instead of
being dropped in silence. The vocabulary lookup only recognised a miss at
the very end of the vocabulary, so in a merged model a missing word
resolved to whatever sits at the boundary between the two vocabularies;
it returns -1 in both halves now.
Two more things the tokeniser uncovered. The best array was sized for
10 * max(N, 200) entries but sorted over N * para_threads, which fits
only as long as the syntagmatic threads take half of them - it is sized
for num_threads slices now. And its unfilled slots kept wordi == 0 from
the memset, which the result loop emitted as vocab[-1 * max_w] once a
thread found fewer candidates than were asked for; slots start at -1 and
the loop stops there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia91727eb669d73fc055adf652500464b8a743795
diff --git a/Changelog.md b/Changelog.md
index 28cb306..6f18c1e 100644
--- a/Changelog.md
+++ b/Changelog.md
@@ -1,5 +1,36 @@
# Changelog
+## [1.00] - 2026-09-06
+
+- the search field understands vector arithmetic: `König - Mann + Frau` looks
+ for the neighbours of the position vec(König) - vec(Mann) + vec(Frau)
+ instead of the position of a single word. A `+` or `-` is an operator only
+ at the beginning of a token, so hyphenated words such as `Nord-Süd-Dialog`
+ are still found, and blank separated words keep entering the query with a
+ `+` as before. Operands that are not in the vocabulary are named above the
+ result, rather than silently dropped from the query
+- the predictive collocators answer for the whole query instead of for its
+ first operand. getCollocators() read the input weights of `wl->wordi[0]`, so
+ "Haus Auto" answered with the collocators of Haus and "Auto Haus" with those
+ of Auto, without anything saying so. It builds the same signed combination
+ the paradigmatic side searches around, which the score allows because it is
+ `sigmoid(q . syn1neg[collocate, position])` and thus linear in `q` before the
+ sigmoid. The terms are averaged rather than summed: the sigmoid is
+ informative over a narrow range, and the strongest collocates of one word sit
+ at the top of it already, so a plain sum would push them all into saturation.
+ One word, and a balanced analogy, are one term and keep the previous scale
+- the tails of that score saturate rather than being dropped. Activations
+ outside ±MAX_EXP used to skip the collocate entirely, which lost exactly the
+ strongest ones and left them out of the position and target sums as well
+- the count based collocators stay restricted to a single word form, since they
+ are looked up per node in the co-occurrence database. For a query of several
+ operands the interface says so in place of that table, rather than showing
+ the profile of the word nearest to the query vector
+- a word that is not in the vocabulary of a merged model is reported as not
+ found. The lookup only recognised a miss at the end of the whole vocabulary,
+ so in a merged model a missing word used to resolve to whatever sits at the
+ boundary between the two vocabularies
+
## [0.99] - 2026-09-04
- requires collocatordb 1.8.0, in which Log-Dice is computed as Rychlý (2008)
diff --git a/README.md b/README.md
index 54371a5..8f983e3 100644
--- a/README.md
+++ b/README.md
@@ -153,6 +153,30 @@
| getModelName | | get name of model (inferred from the file name) |
| getVocabSize | | get vocabulary size of model |
+#### The `word` parameter
+
+Blank separated word forms are added into one query vector, so the search is
+around the centre of the alternatives, and `|` separates several independent
+searches. `+` and `-` do arithmetic in the vector space:
+`word=König - Mann + Frau` returns the neighbours of the position
+vec(König) - vec(Mann) + vec(Frau). A sign is an operator only at the
+beginning of a token, which leaves hyphenated words such as `Nord-Süd-Dialog`
+searchable; note that a literal `+` has to be percent encoded as `%2B` in a
+URL, since a `+` in a query string is a blank and would simply be another
+separator.
+
+The predictive collocators answer for an expression as well: their score is
+`sigmoid(q . syn1neg[collocate, position])`, linear in the query vector `q`
+before the sigmoid, so `collocators` and `getPosWiseW2VCollocators` return the
+contexts that the computed position predicts. The positive terms are averaged
+rather than summed so that the values stay in the range of a single word form.
+The count based collocators of `getClassicCollocators` are looked up per node
+in the co-occurrence database and remain restricted to one word form.
+
+Operands that are not in the vocabulary are left out of the query vector and
+returned in `unknown`; `operands` is the number of operands the query vector
+was built from.
+
### Get classical (count-based) collocates
| Command | Parameters | Description |
@@ -211,6 +235,10 @@
```
```bash
+GET 'http://localhost:3000/?word=K%C3%B6nig%20-%20Mann%20%2B%20Frau&n=10&json=1' | json_pp |less
+```
+
+```bash
curl -L http://localhost:3000/getClassicCollocators?w=Grund
```
diff --git a/css/derekovecs.css b/css/derekovecs.css
index 82a7eda..3a1c6db 100644
--- a/css/derekovecs.css
+++ b/css/derekovecs.css
@@ -234,6 +234,27 @@
overflow: hidden; /* will contain if #first is longer than #second */
}
+/* Explains, in the place of a table, why that table has nothing to show. */
+.notice {
+ align-self: flex-start;
+ max-width: 40em;
+ padding: 8px 12px;
+ color: #555;
+ border-left: 3px solid #ccc;
+ background-color: #fafafa;
+}
+
+/* Query operands that are not in the vocabulary. The result is computed
+ without them, so this has to be visible next to it. */
+#unknownwords {
+ width: 800px;
+ margin: 0 auto 10px auto;
+ padding: 4px 8px;
+ border-left: 3px solid #c00;
+ background-color: #fff4f4;
+ box-sizing: border-box;
+}
+
#topwrapper {
width: 100%;
display: flex;
diff --git a/derekovecs-server.dict b/derekovecs-server.dict
index d344bdc..71b13c4 100644
--- a/derekovecs-server.dict
+++ b/derekovecs-server.dict
@@ -26,7 +26,9 @@
Options => 'Optionen',
SEARCH => 'SUCHE',
words_to_be_searched => 'Zu suchende Wortform(en)',
- search_description => 'Wenn sie mehrere Wortformen kontrastieren wollen, trennen Sie diese mit Leerzeichen, um um den Mittelpunkt der Alternativen herum nach ähnlichen Wörtern zu suchen und mit "|", um die Nachbarn zu allen angegebenen Wortformen zur erhalten.',
+ search_description => 'Wenn sie mehrere Wortformen kontrastieren wollen, trennen Sie diese mit Leerzeichen, um um den Mittelpunkt der Alternativen herum nach ähnlichen Wörtern zu suchen und mit "|", um die Nachbarn zu allen angegebenen Wortformen zur erhalten. Mit "+" und "-" lässt sich im Vektorraum rechnen: "König - Mann + Frau" sucht die Nachbarn der so bestimmten Position und die Kontexte, die sie vorhersagt.',
+ not_in_vocabulary => 'Nicht im Vokabular und daher nicht berücksichtigt:',
+ ca_single_word_only => 'Zählbasierte Kollokatoren gibt es nur für eine einzelne Wortform. Sie werden pro Knoten in der Kookkurrenzdatenbank nachgeschlagen und lassen sich nicht aus dem Suchvektor berechnen; die prädiktiven Kollokatoren links dagegen schon.',
paradigmatic_tsne => 'Paradigmatisch (t-SNE)',
paradigmatic_som => 'Paradigmatisch (SOM)',
syntagmatic => 'Syntagmatisch',
@@ -59,7 +61,9 @@
Options => 'Options',
SEARCH => 'SEARCH',
words_to_be_searched => 'Word(s) to be searched',
- search_description => 'When looking for multiple words use spaces as separators to search around the average vector and | as separator to get the neighbours for each word.',
+ search_description => 'When looking for multiple words use spaces as separators to search around the average vector and | as separator to get the neighbours for each word. "+" and "-" do arithmetic in the vector space: "König - Mann + Frau" searches the neighbours of the position it computes, and the contexts that position predicts.',
+ not_in_vocabulary => 'Not in the vocabulary and therefore left out:',
+ ca_single_word_only => 'Count based collocators exist for a single word form only. They are looked up per node in the co-occurrence database and cannot be computed from the query vector; the predictive collocators on the left can.',
paradigmatic_tsne => 'Paradigmatic (t-SNE)',
paradigmatic_som => 'Paradigmatic (SOM)',
syntagmatic => 'Syntagmatic',
diff --git a/lib/IDS/DeReKoVecs/derekovecs-server.c b/lib/IDS/DeReKoVecs/derekovecs-server.c
index b1aa341..5a08486 100644
--- a/lib/IDS/DeReKoVecs/derekovecs-server.c
+++ b/lib/IDS/DeReKoVecs/derekovecs-server.c
@@ -12,6 +12,7 @@
#define max_size 2000
#define max_w 50
#define MAX_NEIGHBOURS 1000
+#define MAX_TARGET_WORDS 100
#define MAX_WORDS -1
#define MAX_THREADS 100
#define MAX_CC 50
@@ -61,9 +62,13 @@
} knn;
typedef struct {
- long long wordi[MAX_NEIGHBOURS];
- char sep[MAX_NEIGHBOURS];
- int length;
+ long long wordi[MAX_TARGET_WORDS];
+ /* '+' or '-': the sign with which wordi[i] enters the query vector */
+ char sep[MAX_TARGET_WORDS];
+ /* blank separated tokens of the query that are not in the vocabulary */
+ char oov[max_size];
+ int length; /* number of tokens found in the vocabulary */
+ int subtractions; /* how many of them enter with a '-' */
} wordlist;
typedef struct {
@@ -405,21 +410,50 @@
free(nbs);
}
+/* Predicts the collocates of a query. The score of a collocate is
+ sigmoid(q . syn1neg[target, position]), which is linear in q before the
+ sigmoid, so the query does not have to be a single word: q can be the same
+ signed combination of input vectors that the paradigmatic side searches
+ around, and "König - Mann + Frau" asks for the contexts that König and Frau
+ predict but Mann does not.
+
+ The combination is the mean over the positive terms rather than their sum,
+ because the sigmoid is only informative over a narrow range of q . syn1neg
+ and the strongest collocates of a single word already sit at the top of it.
+ A plain sum would push them past MAX_EXP, where they all saturate to the
+ same value. Dividing by the number of terms keeps every query in the range
+ the threshold and the auto focus below are calibrated for, and leaves the
+ single word case exactly as it was: one term, divisor one. A balanced
+ analogy has one term as well, +1 -1 +1, and lands in the same range. */
void *getCollocators(void *args) {
knnpars *pars = args;
int N = pars->N;
- int cc = pars->wl->wordi[0];
+ wordlist *wl = pars->wl;
knn *nbs = NULL;
long window_layer_size = size * window * 2;
- long a, b, c, d, window_offset, target, max_target = 0, maxmax_target;
- float f, max_f, maxmax_f;
+ long a, b, c, d, op, window_offset, target, max_target = 0, maxmax_target;
+ float f, max_f, maxmax_f, scale;
+ float qvec[max_size];
+ int terms = 0;
float *target_sums = NULL, worstbest, wpos_sum;
collocator *best;
- if (M2 == NULL || cc == -1)
+ if (M2 == NULL || wl == NULL || wl->length < 1)
return NULL;
+ for (a = 0; a < wl->length; a++) terms += (wl->sep[a] == '-' ? -1 : 1);
+ scale = (terms > 1 ? 1.0f / terms : 1.0f);
+ for (c = 0; c < size; c++) qvec[c] = 0;
+ for (a = 0; a < wl->length; a++) {
+ long long off = wl->wordi[a] * size;
+ if (wl->sep[a] == '-')
+ for (c = 0; c < size; c++) qvec[c] -= M2[off + c];
+ else
+ for (c = 0; c < size; c++) qvec[c] += M2[off + c];
+ }
+ for (c = 0; c < size; c++) qvec[c] *= scale;
+
a = posix_memalign((void **)&target_sums, 128, pars->cutoff * sizeof(float));
memset(target_sums, 0, pars->cutoff * sizeof(float));
best = malloc((N > 200 ? N : 200) * sizeof(collocator));
@@ -434,7 +468,7 @@
best[b].activation = worstbest;
}
- d = cc;
+ d = wl->wordi[0];
maxmax_f = -1;
maxmax_target = 0;
@@ -450,17 +484,22 @@
window_offset -= size;
for (target = 0; target < pars->cutoff; target++) {
if (garbage && garbage[target]) continue;
- if (target == d)
+ /* an operand of the query is not a collocate of itself */
+ for (op = 0; op < wl->length && wl->wordi[op] != target; op++)
+ ;
+ if (op < wl->length)
continue;
f = 0;
for (c = 0; c < size; c++)
- f += M2[d * size + c] * syn1neg_window[target * window_layer_size + window_offset + c];
+ f += qvec[c] * syn1neg_window[target * window_layer_size + window_offset + c];
+ /* Saturate rather than drop. Skipping the tails used to lose exactly
+ the collocates the sigmoid can no longer tell apart, i.e. the
+ strongest ones, and left them out of wpos_sum and target_sums too. */
if (f < -MAX_EXP)
- continue;
+ f = -MAX_EXP;
else if (f > MAX_EXP)
- continue;
- else
- f = expTable[(int)((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2))];
+ f = MAX_EXP;
+ f = expTable[(int)((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2))];
wpos_sum += f;
target_sums[target] += f;
@@ -615,41 +654,74 @@
return res;
}
-wordlist *getTargetWords(char *st1, int search_backw) {
- wordlist *wl = malloc(sizeof(wordlist));
- char st[100][max_size];
- long a, b = 0, c = 0, cn = 0;
+/* Returns the position of a word in the vocabulary, or -1. In a merged model
+ the two vocabularies sit next to each other and search_backw selects which
+ of them is looked at. */
+static long long lookupWord(const char *word, int search_backw) {
+ long long b, lower = (merge_words ? merge_words : 0);
+ long long upper = (merge_words ? merge_words : words);
- while (1) {
- st[cn][b] = st1[c];
- b++;
- c++;
- st[cn][b] = 0;
- if (st1[c] == 0) break;
- if (st1[c] == ' ' /*|| st1[c] == '-'*/) {
- b = 0;
- c++;
- }
+ if (search_backw) {
+ for (b = words - 1; b >= lower && strcmp(&vocab[b * max_w], word) != 0; b--)
+ ;
+ if (b < lower) b = -1;
+ } else {
+ for (b = 0; b < upper && strcmp(&vocab[b * max_w], word) != 0; b++)
+ ;
+ if (b >= upper) b = -1;
}
- cn++;
- for (a = 0; a < cn; a++) {
- if (search_backw) {
- for (b = words - 1; b >= (merge_words ? merge_words : 0) && strcmp(&vocab[b * max_w], st[a]) != 0; b--)
- ;
- } else {
- for (b = 0; b < (merge_words ? merge_words : words) && strcmp(&vocab[b * max_w], st[a]) != 0; b++)
- ;
- }
- if (b == words) b = -1;
- wl->wordi[a] = b;
- if (b == -1) {
- DEBUG_EPRINTF("Out of dictionary word!\n");
- cn--;
- } else {
- DEBUG_EPRINTF("Word: \"%s\" Position in vocabulary: %lld\n", &vocab[wl->wordi[a] * max_w], wl->wordi[a]);
- }
+ return b;
+}
+
+/* Splits a query into the operands of a vector expression. Blanks separate
+ words as before, and a leading '+' or '-' decides with which sign a word
+ enters the query vector, so that "König - Mann + Frau" is answered with the
+ neighbours of vec(König) - vec(Mann) + vec(Frau) rather than with those of
+ any single word. A sign is only an operator at the beginning of a token,
+ which leaves hyphenated words such as "Nord-Süd-Dialog" searchable; a
+ free standing sign applies to the word that follows it.
+
+ Tokens that are not in the vocabulary are left out of the list and
+ collected in wl->oov, so that the caller can report them instead of
+ quietly answering a different question. */
+wordlist *getTargetWords(char *st1, int search_backw) {
+ wordlist *wl = calloc(1, sizeof(wordlist));
+ char *copy = strdup(st1), *tok, *saveptr = NULL;
+ size_t oov_len = 0;
+ int sign = '+';
+
+ if (wl == NULL || copy == NULL) {
+ free(wl);
+ free(copy);
+ return NULL;
}
- wl->length = cn;
+
+ for (tok = strtok_r(copy, " \t\r\n", &saveptr); tok != NULL; tok = strtok_r(NULL, " \t\r\n", &saveptr)) {
+ long long b;
+ if (*tok == '+' || *tok == '-') {
+ sign = *tok++;
+ if (*tok == 0) continue; /* " - Mann": the sign belongs to the next token */
+ }
+ if (wl->length >= MAX_TARGET_WORDS) break;
+ /* Counted before the lookup: what the query is asking for does not change
+ because one of its operands happens to be unknown. */
+ if (sign == '-') wl->subtractions++;
+ b = lookupWord(tok, search_backw);
+ if (b < 0) {
+ DEBUG_EPRINTF("Out of dictionary word: \"%s\"\n", tok);
+ if (oov_len + strlen(tok) + 2 <= sizeof(wl->oov)) {
+ if (oov_len > 0) wl->oov[oov_len++] = ' ';
+ strcpy(wl->oov + oov_len, tok);
+ oov_len += strlen(tok);
+ }
+ } else {
+ DEBUG_EPRINTF("Word: \"%s\" Sign: %c Position in vocabulary: %lld\n", &vocab[b * max_w], sign, b);
+ wl->sep[wl->length] = (char)sign;
+ wl->wordi[wl->length++] = b;
+ }
+ sign = '+';
+ }
+ free(copy);
return (wl);
}
@@ -763,19 +835,19 @@
float worstbest = -1;
- for (a = 0; a < N; a++) best[a].activation = 0;
- a = 0;
+ for (a = 0; a < N; a++) {
+ best[a].activation = -1;
+ best[a].wordi = -1;
+ }
bi = wl->wordi;
cn = wl->length;
sep = wl->sep;
- b = bi[0];
- if (b == -1) {
+ if (cn < 1) {
goto end;
}
for (a = 0; a < size; a++) vec[a] = 0;
for (b = 0; b < cn; b++) {
- if (bi[b] == -1) continue;
- if (b > 0 && sep[b - 1] == '-')
+ if (sep[b] == '-')
for (a = 0; a < size; a++) vec[a] -= M[a + bi[b] * size];
else
for (a = 0; a < size; a++) vec[a] += M[a + bi[b] * size];
@@ -783,8 +855,12 @@
len = 0;
for (a = 0; a < size; a++) len += vec[a] * vec[a];
len = sqrt(len);
+ /* An expression whose operands cancel each other out, "Haus - Haus", has no
+ position to search around. */
+ if (len == 0) {
+ goto end;
+ }
for (a = 0; a < size; a++) vec[a] /= len;
- for (a = 0; a < N; a++) best[a].activation = -1;
for (c = from; c < upto; c++) {
if (garbage && garbage[c]) continue;
a = 0;
@@ -841,7 +917,7 @@
cutoff = words;
wl = getTargetWords(word, search_backw);
- if (wl == NULL || wl->length < 1 || wl->wordi[0] < 0 || syn_threads < 1) {
+ if (wl == NULL || wl->length < 1 || syn_threads < 1) {
free(wl);
return newSVpv("", 0);
}
@@ -923,24 +999,57 @@
knnpars pars[MAX_THREADS];
pthread_t *pt = (pthread_t *)malloc((num_threads + 1) * sizeof(pthread_t));
wordlist *wl = NULL;
- int syn_threads = (M2 ? window * 2 : 0);
- int para_threads = (no_similar_profiles ? 0 : num_threads - syn_threads);
+ int syn_threads = 0;
+ int para_threads = 0;
for (a = 0; a < MAX_THREADS; a++) para_nbs[a] = syn_nbs[a] = NULL;
- collocator *best = NULL;
- posix_memalign((void **)&best, 128, 10 * (N >= 200 ? N : 200) * sizeof(collocator));
- memset(best, 0, (N >= 200 ? N : 200) * sizeof(collocator));
-
if (N > MAX_NEIGHBOURS) N = MAX_NEIGHBOURS;
+ if (N < 1) N = 1;
+
+ /* Every paradigmatic thread fills its own slice of N entries, and the
+ syntagmatic part below works on the first MAX_NEIGHBOURS of the same
+ array. How many paradigmatic threads there are is only known further
+ down, so the array is sized for the maximum. */
+ collocator *best = NULL;
+ long best_entries = (long)N * num_threads;
+ if (best_entries < MAX_NEIGHBOURS) best_entries = MAX_NEIGHBOURS;
+ posix_memalign((void **)&best, 128, best_entries * sizeof(collocator));
+ memset(best, 0, best_entries * sizeof(collocator));
if (cutoff < 1 || cutoff > words)
cutoff = words;
wl = getTargetWords(st1, search_backw);
- if (wl == NULL || wl->length < 1)
+ if (wl == NULL)
goto end;
+ /* Tell the caller which words the query vector was built from and which
+ tokens of the query are unknown, so that "König - Mannn + Frau" is not
+ silently answered as "König + Frau". */
+ {
+ SV *added = newSVpv("", 0);
+ SV *unknown = newSVpv(wl->oov, 0);
+ for (a = 0; a < wl->length; a++) {
+ if (wl->sep[a] == '-') continue;
+ if (SvCUR(added) > 0) sv_catpvn(added, " ", 1);
+ sv_catpv(added, &vocab[wl->wordi[a] * max_w]);
+ }
+ if (latin_enc == 0) {
+ SvUTF8_on(added);
+ SvUTF8_on(unknown);
+ }
+ hv_store(result, "added", strlen("added"), added, 0);
+ hv_store(result, "unknown", strlen("unknown"), unknown, 0);
+ hv_store(result, "operands", strlen("operands"), newSViv(wl->length), 0);
+ }
+
+ if (wl->length < 1)
+ goto end;
+
+ syn_threads = (M2 ? window * 2 : 0);
+ para_threads = (no_similar_profiles ? 0 : num_threads - syn_threads);
+
slice = (para_threads > 0 ? cutoff / para_threads : cutoff);
a = posix_memalign((void **)&target_sums, 128, cutoff * sizeof(float));
@@ -964,7 +1073,7 @@
DEBUG_PRINTF("From: %ld, Upto: %ld\n", pars[a].from, pars[a].upto);
pthread_create(&pt[a], NULL, _get_neighbours, (void *)&pars[a]);
}
- if (M2) {
+ if (syn_threads) {
window_sums = new_window_sums();
for (a = 0; a < syn_threads; a++) {
pars[a + para_threads].cutoff = cutoff;
@@ -999,6 +1108,8 @@
for (a = 0, i = 0; i < N && a < N * para_threads; a++) {
int filtered = 0;
long long c = best[a].wordi;
+ if (c < 0) /* the threads found fewer candidates than were asked for */
+ break;
if ((merge_words && dedupe && i > 1) || (!merge_words && dedupe && i > 0)) {
for (j = 0; j < i && !filtered; j++)
if (strcasestr(&vocab[c * max_w], &vocab[chosen[j] * max_w]) ||
@@ -1055,7 +1166,7 @@
float total_activation = 0;
- if (M2) {
+ if (syn_threads) {
DEBUG_PRINTF("Waiting for syn threads to join\n");
DEBUG_FFLUSH();
for (a = 0; a < syn_threads; a++) pthread_join(pt[a + para_threads], (void *)&syn_nbs[a]);
diff --git a/script/derekovecs-server b/script/derekovecs-server
index c69ef28..ea9fbd4 100755
--- a/script/derekovecs-server
+++ b/script/derekovecs-server
@@ -1,6 +1,6 @@
#!/usr/bin/env perl
-our $VERSION = '0.99';
+our $VERSION = '1.00';
use IDS::DeReKoVecs::Read;
use Mojolicious::Lite;
@@ -435,6 +435,14 @@
my $res;
my @lists;
my @collocations;
+ # The words the query vectors were actually built from and the tokens that
+ # are not in the vocabulary, both reported back by get_neighbours().
+ my @added;
+ my @unknown;
+ # How many operands the first query part has. The count based collocators are
+ # looked up per node in the co-occurrence database, so they only exist for a
+ # query that is one word.
+ my $operands = 0;
if(defined($word) && $word !~ /^\s*$/) {
$c->inactivity_timeout(300);
$word =~ s/\s+/ /g;
@@ -463,12 +471,16 @@
$cache->set($key => $res) unless $opt_C;
}
push(@lists, $res->{paradigmatic});
+ push(@added, $res->{added}) if defined $res->{added} && $res->{added} ne '';
+ $operands = $res->{operands} // 0 unless @lists > 1;
+ push(@unknown, $res->{unknown}) if defined $res->{unknown} && $res->{unknown} ne '';
}
}
$word =~ s/ *\| */ | /g;
if($json) {
- return $c->render(json => {word => $word, list => \@lists, collocators=>$res->{syntagmatic}});
+ return $c->render(json => {word => $word, list => \@lists, collocators=>$res->{syntagmatic},
+ unknown => join(" ", @unknown), operands => $operands});
} elsif($csv) {
my $csv_data="";
for (my $i=0; $i <= $no_nbs; $i++) {
@@ -505,6 +517,9 @@
dedupe => $dedupe,
marked => \%marked,
lists => \@lists,
+ added => join(" ", @added),
+ unknown => join(" ", @unknown),
+ operands => $operands,
collocators => $res->{syntagmatic},
version => $VERSION,
korap_url => $KORAP_URL,
diff --git a/t/server-test.t b/t/server-test.t
index 5a216c6..c5f48ce 100644
--- a/t/server-test.t
+++ b/t/server-test.t
@@ -1,6 +1,6 @@
use strict;
use warnings;
-use Test::More tests=>8;
+use Test::More tests=>13;
use Mojo::JSON qw(decode_json encode_json to_json);
use REST::Client;
use Data::Dump qw(dump);
@@ -22,6 +22,33 @@
is( $res->{list}->[0]->[1]->{word}, "Reaktion", "primary paradigmatic neighbour of Grund" );
is( $res->{collocators}->[0]->{word}, "Hitchcock", "primary syntagmatic neighbour of Grund" );
+# "König - Mann + Frau": the neighbours of a position that belongs to no
+# single word, and no syntagmatic answer for it.
+$client->GET('http://localhost:3000/?word=K%C3%B6nig+-+Mann+%2B+Frau&json=1');
+$res = decode_json($client->responseContent());
+is( $res->{list}->[0]->[0]->{word}, "Dareios", "primary neighbour of a vector expression" );
+ok( $res->{collocators} && @{$res->{collocators}} > 0, "a vector expression has syntagmatic neighbours too" );
+
+# The syntagmatic side used to read the input weights of the first operand
+# alone: "Haus Auto" answered with the collocators of "Haus", "Auto Haus" with
+# those of "Auto". It combines them now, so the order no longer matters and
+# neither single word answer comes back unchanged.
+$client->GET('http://localhost:3000/?word=Haus&json=1');
+my $haus = decode_json($client->responseContent());
+$client->GET('http://localhost:3000/?word=Haus+Auto&json=1');
+my $haus_auto = decode_json($client->responseContent());
+$client->GET('http://localhost:3000/?word=Auto+Haus&json=1');
+my $auto_haus = decode_json($client->responseContent());
+is_deeply( [map { $_->{word} } @{$haus_auto->{collocators}}],
+ [map { $_->{word} } @{$auto_haus->{collocators}}],
+ "the syntagmatic side does not depend on the order of the operands" );
+isnt( $haus_auto->{collocators}->[0]->{max}, $haus->{collocators}->[0]->{max},
+ "and combines them rather than answering for the first" );
+
+$client->GET('http://localhost:3000/?word=Grund+Blahfasel&json=1');
+$res = decode_json($client->responseContent());
+is( $res->{unknown}, "Blahfasel", "operands outside the vocabulary are reported" );
+
$client->GET('http://localhost:3000/getClassicCollocators?w=Grund');
#print STDERR dump($res);
$res = decode_json($client->responseContent());
diff --git a/templates/de/about.html.ep b/templates/de/about.html.ep
index c58aebf..e58acd2 100644
--- a/templates/de/about.html.ep
+++ b/templates/de/about.html.ep
@@ -9,6 +9,34 @@
Die hier verwendeten Modelle beruhen zum einen auf einer Erweiterung von word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
und zum anderen auf einfachen Kookkurrenzhäufigkeiten und Analysemethoden, die auf diesen operieren.
</p>
+<h3>Suchanfragen</h3>
+<p>
+ Mehrere durch Leerzeichen getrennte Wortformen werden zu einem Suchvektor addiert, es wird
+ also um den Mittelpunkt der Alternativen herum gesucht; ein „|“ trennt dagegen mehrere
+ voneinander unabhängige Suchen. Mit „+“ und „-“ lässt sich darüber hinaus im Vektorraum
+ rechnen: <span class="mono">König - Mann + Frau</span> sucht die Nachbarn der Position
+ vec(König) - vec(Mann) + vec(Frau) statt der Position eines einzelnen Wortes. Ein Vorzeichen
+ ist nur am Anfang eines Tokens ein Operator, Bindestrichkomposita wie
+ <span class="mono">Nord-Süd-Dialog</span> bleiben also suchbar.
+</p>
+<p>
+ Die syntagmatische Ansicht rechnet mit. Der prädiktive Kollokationswert eines Kandidaten ist
+ \(\sigma(q \cdot u)\), vor der Sigmoidfunktion also linear in \(q\); ein Ausdruck liefert
+ deshalb die Kontexte, die die berechnete Position vorhersagt – bei „König - Mann + Frau“ die,
+ die König und Frau vorhersagen, Mann aber nicht. Gemittelt wird dabei über die positiven
+ Terme statt zu summieren, damit die Werte in demselben Bereich bleiben wie bei einer
+ einzelnen Wortform: Der informative Bereich der Sigmoidfunktion ist schmal, und die
+ stärksten Kollokatoren eines einzelnen Wortes liegen bereits an seinem oberen Rand. Eine
+ ausgeglichene Analogie (+1 -1 +1) hat einen Term und damit denselben Maßstab wie eine
+ einzelne Wortform.
+</p>
+<p>
+ Die zählbasierten Kollokatoren bleiben dagegen auf eine einzelne Wortform beschränkt: Sie
+ werden pro Knoten in der Kookkurrenzdatenbank nachgeschlagen und lassen sich nicht aus dem
+ Suchvektor berechnen. Operanden, die nicht im Vokabular stehen, werden über dem Ergebnis
+ genannt und bei der Berechnung weggelassen.
+</p>
+
<h3>Effektive Fenstergröße und Auto-Focus</h3>
<p>
Die Assoziationsmaße LL, MI, MI², MI³ und nPMI werden nicht über das gesamte Kontextfenster
@@ -42,6 +70,11 @@
</p>
<h3>Änderungen</h3>
<p>
+ <strong>06.09.2026</strong>
+
+ Das Suchfeld versteht jetzt Vektorarithmetik, siehe <em>Suchanfragen</em> oben.
+</p>
+<p>
<strong>04.09.2026</strong>
Log-Dice (LD) wird jetzt genau so berechnet, wie es Rychlý (2008) definiert,
diff --git a/templates/en/about.html.ep b/templates/en/about.html.ep
index 20c0ecc..039c427 100644
--- a/templates/en/about.html.ep
+++ b/templates/en/about.html.ep
@@ -9,6 +9,33 @@
The models used here are based on an extension of word2vec (Mikolov et al. 2013), wang2vec (Ling et al. 2015)
and on the other hand on simple co-occurence counts and analysis methods that operate on these.
</p>
+<h3>Queries</h3>
+<p>
+ Several word forms separated by blanks are added into one query vector, i.e. the search is
+ around the centre of the alternatives, while a “|” separates several independent
+ searches. Beyond that, “+” and “-” do arithmetic in the vector space:
+ <span class="mono">König - Mann + Frau</span> searches the neighbours of the position
+ vec(König) - vec(Mann) + vec(Frau) rather than the position of a single word. A sign is an
+ operator only at the beginning of a token, so hyphenated words such as
+ <span class="mono">Nord-Süd-Dialog</span> remain searchable.
+</p>
+<p>
+ The syntagmatic view follows along. The predictive score of a candidate is
+ \(\sigma(q \cdot u)\), which is linear in \(q\) before the sigmoid, so an expression
+ yields the contexts that the computed position predicts – for “König - Mann +
+ Frau” the ones König and Frau predict but Mann does not. The positive terms are
+ averaged rather than summed, so that the values stay in the range a single word form
+ occupies: the sigmoid is informative over a narrow range only, and the strongest collocates
+ of a single word already sit at the top of it. A balanced analogy (+1 -1 +1) is one term and
+ is therefore on the same scale as a single word form.
+</p>
+<p>
+ The count based collocators, in contrast, remain restricted to a single word form: they are
+ looked up per node in the co-occurrence database and cannot be computed from the query
+ vector. Operands that are not in the vocabulary are named above the result and left out of
+ the computation.
+</p>
+
<h3>Effective window size and auto focus</h3>
<p>
The association measures LL, MI, MI², MI³ and nPMI are not computed over the whole context
@@ -40,6 +67,11 @@
</p>
<h3>Changes</h3>
<p>
+ <strong>2026-09-06</strong>
+
+ The search field understands vector arithmetic, see <em>Queries</em> above.
+</p>
+<p>
<strong>2026-09-04</strong>
Log-Dice (LD) is now computed exactly as defined by Rychlý (2008),
diff --git a/templates/index.html.ep b/templates/index.html.ep
index 9322028..ce38536 100644
--- a/templates/index.html.ep
+++ b/templates/index.html.ep
@@ -2,6 +2,13 @@
<html>
<head>
<% my $plain_title = $title; $plain_title=~s/<[^>]+>//g; %>
+%# Whether the query was answered at all: not found, and an expression whose
+%# operands cancel each other out, both leave the first list empty.
+% my $have_results = ($lists && @$lists > 0 && $lists->[0] && @{$lists->[0]} > 0);
+%# The count based collocators come out of the co-occurrence database, which is
+%# keyed by one node word, so they have no answer for a query of several
+%# operands. The predictive ones do, see getCollocators().
+% my $single_word = ($operands == 1);
<title><%= $plain_title %>:<%= $word %> · IDS word vector analysis</title>
%# Result pages are computed for one word each and there is one for every
%# word of the vocabulary, so they are not something to index or to follow.
@@ -44,8 +51,18 @@
});
var urlParams = new URLSearchParams(window.location.search);
var currentWords = urlParams.get("word");
- var CIIsearchWords = (currentWords && (currentWords.includes(" ") || currentWords.includes("|")) ? '('+currentWords.replace(/[ |]+/g, " oder ")+')' : currentWords);
+ % use Mojo::ByteStream 'b';
+ // The vocabulary entries the query vector was built from, as the server
+ // resolved them. A query can be a vector expression, "König - Mann +
+ // Frau", and the subtracted operands are deliberately not in here: they
+ // are not marked in the maps, and not part of the KorAP queries built
+ // from the result.
+ var searchedWords = <%= b(Mojo::JSON::to_json($added // '')) %>;
+ var targetWords = " " + searchedWords + " ";
+ var CIIsearchWords = (searchedWords.includes(" ") ? '('+searchedWords.replace(/\s+/g, " oder ")+')' : searchedWords);
var collocatorTable = null;
+ var classicCollocatorTable = null;
+ var singleWord = <%= $single_word ? 'true' : 'false' %>;
var plainTitle ="<%= $plain_title %>"
var korapPath="/";
if (plainTitle.match(/-en/)) {
@@ -105,8 +122,8 @@
var collocatorTable_activated = false;
$( "#tabs" ).on( "tabsactivate", function( event, ui ) {
if (localStorage) localStorage['tab'] = ui.newTab.index();
- if(ui.newTab.index() == 3 && !collocatorTable_activated) {
- classicCollocatorTable.columns.adjust();
+ if(ui.newTab.index() == 3 && !collocatorTable_activated && collocatorTable) {
+ if (classicCollocatorTable) classicCollocatorTable.columns.adjust();
collocatorTable.columns.adjust();
collocatorTable_activated = true;
}
@@ -143,10 +160,9 @@
return changeCharColor(str, heat, word);
}
- % use Mojo::ByteStream 'b';
var paraResults = <%= b(Mojo::JSON::to_json($lists)) %>;
var urlprefix = new URLSearchParams(window.location.search);
- if (paraResults.length > 0 && paraResults[0] != null) {
+ if (paraResults.length > 0 && paraResults[0] != null && paraResults[0].length > 0) {
var nvecs = [],
nwords = [],
nranks = [],
@@ -157,7 +173,7 @@
nranks = nranks.concat(paraResults[i].map(function(a){return a.rank;}));
nmarked = nmarked.concat(paraResults[i].map(function(a){return a.marked;}));
}
- showMap({target: " "+urlParams.get('word')+" ", mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: nvecs, ranks: nranks, marked: nmarked} );
+ showMap({target: targetWords, mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: nvecs, ranks: nranks, marked: nmarked} );
var t = $('#firsttable').DataTable({
data: [].concat.apply([], paraResults),
"sScrollY": "780px",
@@ -311,19 +327,24 @@
var filterQuot = /^quot/;
var ccResult;
var baseURL = window.location.pathname.replace(/[/]$/, '')
- classicCollocatorTable = makeClassicCollocatorTable('#classicoloctable', baseURL, paraResults[0][0].rank)
+ // Only for a one word query: the node of a count based profile is a
+ // word, and paraResults[0][0] is the nearest word to the query
+ // vector, which is not what was asked for.
+ if (singleWord) {
+ classicCollocatorTable = makeClassicCollocatorTable('#classicoloctable', baseURL, paraResults[0][0].rank)
- $('#show-details').change(function (e) {
- var columns = classicCollocatorTable.columns(".detail");
- if(this.checked) {
- columns.visible(true);
- $("#ccd").css('width', 'auto');
- } else {
- columns.visible(false);
- $("#ccd").css('width', '680px');
- }
- classicCollocatorTable.columns.adjust().draw();
- } );
+ $('#show-details').change(function (e) {
+ var columns = classicCollocatorTable.columns(".detail");
+ if(this.checked) {
+ columns.visible(true);
+ $("#ccd").css('width', 'auto');
+ } else {
+ columns.visible(false);
+ $("#ccd").css('width', '680px');
+ }
+ classicCollocatorTable.columns.adjust().draw();
+ } );
+ }
$("td.collocator").click(function(){
queryKorAPCII(this.textContent + " /w1:5,s0 " + CIIsearchWords);
@@ -694,7 +715,7 @@
$.post(baseURL+'/getVecsByRanks',
JSON.stringify(nranks),
function(data, status){
- showMap({target: " "+urlParams.get('word')+" ", mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: data, ranks: nranks, marked: Array(100).fill(false)} );
+ showMap({target: targetWords, mergedEnd: <%= $mergedEnd %>, words: nwords, vecs: data, ranks: nranks, marked: Array(100).fill(false)} );
}, 'json');
}
}
@@ -771,6 +792,12 @@
</div>
</div>
</div>
+ %# Operands that are not in the vocabulary are dropped from the query
+ %# vector, which for an expression like "König - Mannn + Frau" would
+ %# otherwise silently answer a different question.
+ % if($unknown ne '' && $have_results) {
+ <div id="unknownwords"><%= loc 'not_in_vocabulary' %> <span class="mono"><%= $unknown %></span></div>
+ % }
<div id="topwrapper">
<div style="visibility: hidden;" id="tabs">
<ul>
@@ -780,7 +807,11 @@
% }
<li><a href="#tabs-1"><%= loc 'paradigmatic_tsne' %></a></li>
<li><a href="#tabs-2"><%= loc 'paradigmatic_som' %></a></li>
- <li><a href="#tabs-3"><%= loc 'syntagmatic' %></a></li>
+ %# No syntagmatic tab for a model without a .net file, i.e. without
+ %# the output weights the predictive collocators are read from.
+ % if($collocators) {
+ <li><a href="#tabs-3"><%= loc 'syntagmatic' %></a></li>
+ % }
% }
<li><a href="#tabs-4">Info</a></li>
</ul>
@@ -807,7 +838,7 @@
</div>
% }
<div id="tabs-1" style="display: flex; padding: 5px; flex-flow: row wrap;">
- % if($lists && (@$lists) > 0 && (@$lists)[0]) {
+ % if($have_results) {
<div id="wrapper">
<div id="first" style="width: 230px; margin-bottom: 15px;">
<table class="display compact nowrap" id="firsttable">
@@ -893,6 +924,7 @@
</div>
% }
</div>
+ % if($collocators) {
<div id="tabs-3" style="display: flex; padding:5px; flex-flow: row wrap;">
<div style="margin-right: 20px; margin-bottom: 10px;" id="secondt">
<table class="display compact nowrap" id="secondtable">
@@ -928,6 +960,9 @@
</tbody>
</table>
</div>
+ % if(!$single_word) {
+ <div id="ccd" class="notice"><%= loc 'ca_single_word_only' %></div>
+ % } else {
<div id="ccd" style="">
<table class="display compact nowrap" id="classicoloctable">
<thead>
@@ -973,11 +1008,13 @@
</tbody>
</table>
</div>
+ % }
<!--
<div style="clear:both" ></div>
<div style="float: right; overflow: hidden" id="extra"><button onClick="showCollocatorSOM()"> </button></div>
-->
</div>
+ % }
<div id="tabs-4" style="display: flex; padding:5px; flex-flow: row wrap;">
<div id="info">
<h3><%== loc 'about' %></h3>