tree ed383baf8d9183a70a226e26810c0916f19e4bfe
parent 0fec0bee779435957d92730316376657f0456463
author Marc Kupietz <kupietz@ids-mannheim.de> 1788528592 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788543758 +0200

Score the first collocate of a query with the right window size

true_window_size counts the positions a pair was observed in and is
passed on as the window size of the expected frequency. It started at
one, but the first key of the iteration takes the same branch as any
further key of the same collocate, which increments it. The first
collocate of every query was therefore scored with a window one position
too wide, so that its expected frequency came out too high and its pmi,
npmi, md, lfmd and llr too low.

The test data shows this: for "Aluminium" and "Anwendungstechnologie",
ln_count is 16 and rn_count 0, so all 16 co-occurrences sit in the single
position left of the node, and the aggregate pmi has to be the pmi of
that position. It was 8.4592 while ln_pmi was 9.4592, exactly log2(2)
apart. They now agree, and npmi, md, lfmd and llr change accordingly,
each matching an independent calculation with a window of one.

Counting up from zero fixes it. Note that md_nws is unaffected, as it
deliberately uses the nominal window size, and logDice no longer takes a
window size at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I58c26bcdf273e320442be5abb5728219682ac951