tree 38ea04c1560562af4a4a75ad8754dd9ecd36dcf6
parent bccdf70f626be8d0efb3b63701caf2e789fe8dbf
author Marc Kupietz <kupietz@ids-mannheim.de> 1788439057 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788439057 +0200

Document that logDice measures salience, not surprise

A collocate that co-occurs less often than expected can still reach a
logDice close to that of a strongly attracted one, which is puzzling
until one sees that logDice never compares against an expected
frequency. Since it only puts O into the numerator of the Dice
coefficient, the difference between the score of a pair and the score it
would get if it co-occurred exactly as often as expected is precisely
the pointwise mutual information:

  logDice(O) - logDice(E) = log2(O / E) = pmi

The level a pair starts from thus depends on the marginal frequencies
alone. Among the collocates of Grund in a 5+5 window, the rare
"triftiger" starts at -4.60 and reaches 3.96, exceeding what is expected
by a pmi of 8.57, while the frequent "Berlin" starts at 5.58 and still
reaches 3.84 while co-occurring 1.74 bits less often than expected.

This is a property of the measure rather than a defect: logDice was
designed to be corpus size independent and comparable across corpora,
not to express surprise. But since collocationAnalysis() ranks and
thresholds by it, users should know to drop pairs that are not attested
more often than expected, with dplyr::filter(O > E) or a minimum pmi or
ll. This is now explained in a section of the association score
documentation, referred to from the thresholdScore parameter where the
default takes effect, and summarized in the Readme.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I9059cd064e2a612d9327cc009da0b4856612e41d