Show the predictive score before the sigmoid
max(a) is sigmoid(z), and sigmoid is monotone, so it ranks the collocates
the way the raw score z does - but only in principle. It is read out of
expTable, whose index is (z + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2),
integer arithmetic that comes to 83: z is bucketed at 1/83 and everything
from MAX_EXP up lands in one value, 0.99741. Of the 1000 rows for
"König - Mann + Frau" only 264 have a distinct max(a), up to fifteen share
one, and Gemahlin, römisch-deutsche, ungekrönte and Co-Fürsten sit
together on the cap.
z was computed for every candidate at every window position and dropped at
that lookup, which is what the unused max_f and maxmax_f in
getCollocators once tracked. It is carried through now, per position in
collocator.raw and as the maximum over positions in collocator.max_raw,
and shown next to max(a) as max(z).
Nothing else changes: the threshold, the ordering inside a position's list
and the auto focus all still work on the activation. The column renders
three decimals but sorts on the unrounded value.
The info page entry says that the arithmetic covers the paradigmatic and
the predictive syntagmatic analysis, and about the latter that it used to
consider only the first word form and has gained the max(z) column. The
saturation change needs no notice of its own: of ten frequent single words
the highest max(z) is 5.981, so none reaches the cap of 6 and only
expressions get past it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: If26bd78b36e247a4250b63059114470c473fb6c5
diff --git a/lib/IDS/DeReKoVecs/derekovecs-server.c b/lib/IDS/DeReKoVecs/derekovecs-server.c
index 5a08486..e72cf4c 100644
--- a/lib/IDS/DeReKoVecs/derekovecs-server.c
+++ b/lib/IDS/DeReKoVecs/derekovecs-server.c
@@ -53,6 +53,12 @@
float probability;
float activation_sum;
float max_activation;
+ /* The score before the sigmoid, q . syn1neg[collocate, position]. activation
+ is expTable[z], quantized in steps of 1/83 and capped at MAX_EXP; z is
+ not. raw is the value at one position, max_raw the maximum over
+ positions. */
+ float raw;
+ float max_raw;
float heat[16];
} collocator;
@@ -410,21 +416,16 @@
free(nbs);
}
-/* Predicts the collocates of a query. The score of a collocate is
- sigmoid(q . syn1neg[target, position]), which is linear in q before the
- sigmoid, so the query does not have to be a single word: q can be the same
- signed combination of input vectors that the paradigmatic side searches
- around, and "König - Mann + Frau" asks for the contexts that König and Frau
- predict but Mann does not.
+/* Predicts the collocates of a query. The score is
+ sigmoid(q . syn1neg[target, position]), linear in q before the sigmoid, so q
+ may be the signed combination of input vectors that the paradigmatic side
+ searches around rather than the vector of a single word.
- The combination is the mean over the positive terms rather than their sum,
- because the sigmoid is only informative over a narrow range of q . syn1neg
- and the strongest collocates of a single word already sit at the top of it.
- A plain sum would push them past MAX_EXP, where they all saturate to the
- same value. Dividing by the number of terms keeps every query in the range
- the threshold and the auto focus below are calibrated for, and leaves the
- single word case exactly as it was: one term, divisor one. A balanced
- analogy has one term as well, +1 -1 +1, and lands in the same range. */
+ q is the mean over the terms, not their sum: the sigmoid is informative over
+ a narrow range only, and a sum would push the strongest collocates past
+ MAX_EXP into saturation. The mean keeps every query in the range MIN_RESP
+ and the auto focus are calibrated for. One word, and a balanced analogy
+ (+1 -1 +1), are one term and keep the previous scale. */
void *getCollocators(void *args) {
knnpars *pars = args;
int N = pars->N;
@@ -433,7 +434,7 @@
knn *nbs = NULL;
long window_layer_size = size * window * 2;
long a, b, c, d, op, window_offset, target, max_target = 0, maxmax_target;
- float f, max_f, maxmax_f, scale;
+ float f, raw, max_f, maxmax_f, scale;
float qvec[max_size];
int terms = 0;
float *target_sums = NULL, worstbest, wpos_sum;
@@ -466,6 +467,7 @@
best[b].wordi = -1;
best[b].probability = 1;
best[b].activation = worstbest;
+ best[b].raw = 0;
}
d = wl->wordi[0];
@@ -492,9 +494,9 @@
f = 0;
for (c = 0; c < size; c++)
f += qvec[c] * syn1neg_window[target * window_layer_size + window_offset + c];
- /* Saturate rather than drop. Skipping the tails used to lose exactly
- the collocates the sigmoid can no longer tell apart, i.e. the
- strongest ones, and left them out of wpos_sum and target_sums too. */
+ raw = f;
+ /* Saturate rather than drop: skipping the tails removed the strongest
+ collocates from the list and from wpos_sum and target_sums. */
if (f < -MAX_EXP)
f = -MAX_EXP;
else if (f > MAX_EXP)
@@ -508,6 +510,7 @@
if (f > best[b].activation) {
memmove(best + b + 1, best + b, (N - b - 1) * sizeof(collocator));
best[b].activation = f;
+ best[b].raw = raw;
best[b].wordi = target;
best[b].position = window - a;
break;
@@ -682,8 +685,7 @@
free standing sign applies to the word that follows it.
Tokens that are not in the vocabulary are left out of the list and
- collected in wl->oov, so that the caller can report them instead of
- quietly answering a different question. */
+ collected in wl->oov, so that the caller can report them. */
wordlist *getTargetWords(char *st1, int search_backw) {
wordlist *wl = calloc(1, sizeof(wordlist));
char *copy = strdup(st1), *tok, *saveptr = NULL;
@@ -703,8 +705,8 @@
if (*tok == 0) continue; /* " - Mann": the sign belongs to the next token */
}
if (wl->length >= MAX_TARGET_WORDS) break;
- /* Counted before the lookup: what the query is asking for does not change
- because one of its operands happens to be unknown. */
+ /* Counted before the lookup: an unknown operand does not change what was
+ asked for. */
if (sign == '-') wl->subtractions++;
b = lookupWord(tok, search_backw);
if (b < 0) {
@@ -1158,6 +1160,8 @@
for (b = 0; b < MAX_NEIGHBOURS; b++) {
best[b].wordi = -1L;
best[b].activation = 0;
+ best[b].raw = 0;
+ best[b].max_raw = 0;
best[b].probability = 0;
best[b].position = 0;
best[b].activation_sum = 0;
@@ -1183,6 +1187,7 @@
best[b].position = -1; // syn_nbs[0]->pos[b];
best[b].activation_sum = target_sums[syn_nbs[0]->best[b].wordi];
best[b].max_activation = 0.0;
+ best[b].max_raw = 0.0;
best[b].average = 0.0;
best[b].probability = 0.0;
best[b].cprobability = syn_nbs[0]->best[b].cprobability;
@@ -1198,6 +1203,7 @@
break;
if (i >= found_index) {
best[found_index].max_activation = 0.0;
+ best[found_index].max_raw = 0.0;
best[found_index].average = 0.0;
best[found_index].probability = 0.0;
memset(best[found_index].heat, 0, sizeof(float) * 16);
@@ -1244,6 +1250,8 @@
word_activation_sum += syn_nbs[a]->best[b].activation;
if (syn_nbs[a]->best[b].activation > best[i].max_activation)
best[i].max_activation = syn_nbs[a]->best[b].activation;
+ if (syn_nbs[a]->best[b].raw > best[i].max_raw)
+ best[i].max_raw = syn_nbs[a]->best[b].raw;
if (syn_nbs[a]->best[b].activation > best[i].heat[wpos])
best[i].heat[wpos] = syn_nbs[a]->best[b].activation;
}
@@ -1352,6 +1360,7 @@
hv_store(hash, "average", strlen("average"), newSVnv(best[a].average), 0);
hv_store(hash, "prob", strlen("prob"), newSVnv(best[a].probability), 0);
hv_store(hash, "cprob", strlen("cprob"), newSVnv(best[a].cprobability_sum), 0);
+ hv_store(hash, "dot", strlen("dot"), newSVnv(best[a].max_raw), 0);
hv_store(hash, "max", strlen("max"), newSVnv(best[a].max_activation), 0); // newSVnv(target_sums[best[a].wordi]), 0);
hv_store(hash, "overall", strlen("overall"), newSVnv(best[a].activation_sum / total_activation), 0); // newSVnv(target_sums[best[a].wordi]), 0);
hv_store(hash, "pos", strlen("pos"), newSVnv(best[a].position), 0);