tree a8d0c8a98843e5dc112e7f297a3033d64ed2fc20
parent d4a7b306b2ea21b18e65ac211ca358d8d9f501a5
author Marc Kupietz <kupietz@ids-mannheim.de> 1788537118 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788537478 +0200

Scale the whole log-likelihood table with the window size

ca_ll's contingency table multiplied only its row total by the window
size, leaving cells that do not add up to one sample: the row was
counted in window positions while the column and the sample size were
counted in corpus tokens.

Following Evert (2004), which the function cites, the sample consists of
co-occurrence tokens rather than corpus tokens. With a window of
window_size positions, every occurrence of either word takes part in
that many pairs, so the sample size and both marginals scale with it.

The expected co-occurrence frequency comes out as window_size * O1 * O2
/ N either way, which is why pmi, mi2 and mi3 never depended on this and
why it went unnoticed for so long. Only ll needs the rest of the table,
and there the values differ, by 0.1% for a frequent collocate up to well
over 100% for a frequent node in a wide window. The ll column of the
collocationScoreQuery example in the Readme was recomputed accordingly.
The light verb construction table is unaffected, its total window size
being 1, where the two formulations are identical.

It also removes a pathology rather than papering over it: N -
window_size * O1 could turn negative, at which point the score became
NaN through a log of a negative number. window_size * (N - O1) cannot,
so the warning added earlier in this development version is dropped
again - it treated a symptom of exactly this inconsistency.

Reported-by: Tim Feldmüller <https://github.com/feldmueller>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I8b912c5bd870a0c9913d019043b2447f951f1891