tree 82d0cb70ae7ca471f2dd268d395a199c91ad80ae
parent 1e9f79174b3cb0b5983ab321db0c9e34d0f781ce
author Marc Kupietz <kupietz@ids-mannheim.de> 1788438396 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788438396 +0200

Compute logDice as defined by Rychly, comparable to other tools

logDice was computed as 14 + log2(2 * O / (w * O1 + O2)), multiplying
the node frequency by the window size, while the documentation cites
Rychly (2008), who defines it as 14 + log2(2 * O / (O1 + O2)).

The window size factor does not belong here. It is correct in the
expected frequency E, where w * O1 counts the positions in which a
collocate can occur, and the other scores take the window into account
through E alone. The Dice coefficient, however, relates the
co-occurrence frequency to how often the two words occur at all, so its
denominator sums marginal frequencies. Adding w * O1, a count of window
positions, to O2, a count of word tokens, mixes units, and it makes the
coefficient asymmetric: for Grund/triftiger it gave 0.64, for
triftiger/Grund 3.96, from the same counts.

The practical effect was that a frequent node made w * O1 dominate the
denominator - for Grund and triftiger in a 5+5 window, O2 contributed
0.006% of it - so that rare collocates were penalised. triftiger scored
0.64, below Berlin at 2.03, although Berlin co-occurs with Grund less
often than chance predicts (pmi -1.74). They are now 3.96 and 3.84.

Scores rise by up to log2(w), so up to 3.32 for the default context of
5 left and 5 right, which makes collocationAnalysis() somewhat more
permissive when recursing with the default thresholdScore = "logDice"
and threshold = 2. Results computed with a total window size of 1 are
unaffected, the two formulas being identical there, which is why the
light verb construction table in the Readme stays as it is, while the
collocationScoreQuery example, which uses the default window, was rerun.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I190fd65d0c53517df74c26c007a6440d974f700c