Scale the whole log-likelihood table with the window size
ca_ll() multiplied only its row total by the window size, leaving cells
that do not add up to one sample: the row was counted in window
positions, the column and the sample size in corpus tokens.
Following Evert (2004), which the function cites, the sample consists of
co-occurrence tokens rather than corpus tokens. With a window of
window_size positions, every occurrence of either word takes part in
that many pairs, so the sample size and both marginals scale with it.
The expected co-occurrence frequency comes out as window_size * f1 * f2
/ n either way, which is why pmi, md, lfmd and npmi never depended on
this, and why it stayed unnoticed: only llr needs the rest of the table.
There the values differ, by 0.1% for a frequent collocate up to well
over 100% for a frequent node in a wide window. Results with a window of
one position are unchanged, the two being identical there.
It also removes a pathology instead of guarding against it: n -
window_size * f1 could turn negative, at which point the score became
NaN. window_size * (n - f1) cannot. In this database the case was out of
reach anyway, needing a word covering more than a tenth of the corpus,
but the same table is used elsewhere with freely chosen window sizes.
Reported-by: Tim Feldmüller <https://github.com/feldmueller>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ic396d586da7f6a68b2c911d871ff58032b554d1f
diff --git a/tests/basic_test.c b/tests/basic_test.c
index d98aac4..f4a003f 100644
--- a/tests/basic_test.c
+++ b/tests/basic_test.c
@@ -40,7 +40,7 @@
COLLOCATORDB* cdb = open_collocatordb(dbpath);
TEST_ASSERT(cdb != NULL);
char *json = get_collocators_as_json(cdb, testword);
- char *needle = "\"word\":\"um\",\"f2\":264,\"f\":5,\"npmi\":-0.0556349,\"pmi\":-0.958074,\"llr\":2.87723,\"lfmd\":3.68578,\"md\":1.36385,\"md_nws\":0.363854,\"dice\":0.00720461,\"ld\":6.88314,\"ln_count\":0,\"rn_count\":1,\"ln_pmi\":-1,\"rn_pmi\":-1,\"ldaf\":4.79935,\"win\":668,\"afwin\":668";
+ char *needle = "\"word\":\"um\",\"f2\":264,\"f\":5,\"npmi\":-0.0556349,\"pmi\":-0.958074,\"llr\":2.80721,\"lfmd\":3.68578,\"md\":1.36385,\"md_nws\":0.363854,\"dice\":0.00720461,\"ld\":6.88314,\"ln_count\":0,\"rn_count\":1,\"ln_pmi\":-1,\"rn_pmi\":-1,\"ldaf\":4.79935,\"win\":668,\"afwin\":668";
TEST_CHECK(strstr(json, needle) > 0);
TEST_MSG("Expected to contain: %s", needle);
}