tree 1796b87796de05133bf29aaa1943d48c589471e6
parent 28f05d1e8639b16845d54ec9379fd116d5e66be3
author Marc Kupietz <kupietz@ids-mannheim.de> 1788537237 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788543758 +0200

Scale the whole log-likelihood table with the window size

ca_ll() multiplied only its row total by the window size, leaving cells
that do not add up to one sample: the row was counted in window
positions, the column and the sample size in corpus tokens.

Following Evert (2004), which the function cites, the sample consists of
co-occurrence tokens rather than corpus tokens. With a window of
window_size positions, every occurrence of either word takes part in
that many pairs, so the sample size and both marginals scale with it.

The expected co-occurrence frequency comes out as window_size * f1 * f2
/ n either way, which is why pmi, md, lfmd and npmi never depended on
this, and why it stayed unnoticed: only llr needs the rest of the table.
There the values differ, by 0.1% for a frequent collocate up to well
over 100% for a frequent node in a wide window. Results with a window of
one position are unchanged, the two being identical there.

It also removes a pathology instead of guarding against it: n -
window_size * f1 could turn negative, at which point the score became
NaN. window_size * (n - f1) cannot. In this database the case was out of
reach anyway, needing a word covering more than a tenth of the corpus,
but the same table is used elsewhere with freely chosen window sizes.

Reported-by: Tim Feldmüller <https://github.com/feldmueller>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ic396d586da7f6a68b2c911d871ff58032b554d1f