Scale the whole log-likelihood table with the window size
ca_ll() multiplied only its row total by the window size, leaving cells
that do not add up to one sample: the row was counted in window
positions, the column and the sample size in corpus tokens.
Following Evert (2004), which the function cites, the sample consists of
co-occurrence tokens rather than corpus tokens. With a window of
window_size positions, every occurrence of either word takes part in
that many pairs, so the sample size and both marginals scale with it.
The expected co-occurrence frequency comes out as window_size * f1 * f2
/ n either way, which is why pmi, md, lfmd and npmi never depended on
this, and why it stayed unnoticed: only llr needs the rest of the table.
There the values differ, by 0.1% for a frequent collocate up to well
over 100% for a frequent node in a wide window. Results with a window of
one position are unchanged, the two being identical there.
It also removes a pathology instead of guarding against it: n -
window_size * f1 could turn negative, at which point the score became
NaN. window_size * (n - f1) cannot. In this database the case was out of
reach anyway, needing a word covering more than a tenth of the corpus,
but the same table is used elsewhere with freely chosen window sizes.
Reported-by: Tim Feldmüller <https://github.com/feldmueller>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ic396d586da7f6a68b2c911d871ff58032b554d1f
diff --git a/src/collocatordb.cc b/src/collocatordb.cc
index 6eda410..63029c4 100644
--- a/src/collocatordb.cc
+++ b/src/collocatordb.cc
@@ -179,12 +179,21 @@
// Collocations. PhD dissertation, IMS, University of Stuttgart. Published in
// 2005, URN urn:nbn:de:bsz:93-opus-23714. Free PDF available from
// http://purl.org/stefan.evert/PUB/Evert2004phd.pdf
+// The table classifies co-occurrence tokens, not corpus tokens: with a window of
+// window_size positions, every occurrence of either word takes part in that many
+// pairs, so the sample size and both marginals scale with it. Scaling only the
+// row, as this did before, leaves cells that do not add up to one sample and lets
+// n - window_size * w1 turn negative for a frequent node in a wide window. The
+// expected co-occurrence frequency is window_size * w1 * w2 / n either way, which
+// is why the other measures do not depend on this.
static double ca_ll(uint64_t w1, uint64_t w2, uint64_t w12, uint64_t n,
uint64_t window_size) {
- double r1 = (double)w1 * window_size, r2 = (double)n - r1, c1 = w2,
- c2 = n - c1, o11 = w12, o12 = r1 - o11, o21 = c1 - w12, o22 = r2 - o21,
- e11 = r1 * c1 / n, e12 = r1 * c2 / n, e21 = r2 * c1 / n,
- e22 = r2 * c2 / n;
+ double total = (double)n * window_size,
+ r1 = (double)w1 * window_size, r2 = total - r1,
+ c1 = (double)w2 * window_size, c2 = total - c1,
+ o11 = w12, o12 = r1 - o11, o21 = c1 - o11, o22 = r2 - o21,
+ e11 = r1 * c1 / total, e12 = r1 * c2 / total, e21 = r2 * c1 / total,
+ e22 = r2 * c2 / total;
return (2 * ((o11 > 0 ? o11 * log(o11 / e11) : 0) +
(o12 > 0 ? o12 * log(o12 / e12) : 0) +
(o21 > 0 ? o21 * log(o21 / e21) : 0) +