tree bdd5dbd99a1eb1c098ed1d933a8f0b5d32b98a1e
parent 92f779d03ee09c543434432d222d4267fc286784
author Marc Kupietz <kupietz@ids-mannheim.de> 1788543758 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788543758 +0200

Compute logDice as defined by Rychly, and split off the auto focus score

ca_logdice() and ca_dice() multiplied the node frequency by the window
size, while the citation right above ca_logdice() and the tooltip in the
DeReKoVecs web UI both give Rychly's (2008) definition,
14 + log2(2*f12/(f1+f2)).

The window size does not belong there. It is correct in the expected
frequency of the other measures, where f1 * window_size counts the
positions a collocate can occupy, but the Dice coefficient relates the
co-occurrence frequency to how often the two words occur at all, so its
denominator sums marginal frequencies. Adding f1 * window_size, a count
of window positions, to f2, a count of word tokens, mixes units, and it
made the coefficient asymmetric: swapping node and collocate changed the
score, 0.64 against 3.96 for Grund and triftiger from the same counts.

Because f1 * window_size dominated the denominator for a frequent node,
rare collocates were penalised. Among the collocates of "Grund" in a 5+5
window, "triftiger" scored 0.64, below "Berlin" at 2.03, although Berlin
co-occurs with Grund less often than chance predicts. They are now 3.96
and 3.84, and comparable with other tools such as Sketch Engine.

The auto focus score, reported as LDaf, keeps that factor and moves into
its own function, ca_focus_score(). It is Dice-like but not logDice, and
the two are deliberately not on a common scale: the penalty on wide
windows is what makes it sensitive to how concentrated a pair is, which
is what tells actual collocations from words that merely share contexts,
and is why LDaf orders collocates well enough to be the default order.
Its values are unchanged, bit-identical for all 504 collocates of the
five nodes checked against the state before this change, as are the
selected windows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I72bd69d4d2a74127ca87bf20d94f9ffafda6c191