Document that logDice measures salience, not surprise
A collocate that co-occurs less often than expected can still reach a
logDice close to that of a strongly attracted one, which is puzzling
until one sees that logDice never compares against an expected
frequency. Since it only puts O into the numerator of the Dice
coefficient, the difference between the score of a pair and the score it
would get if it co-occurred exactly as often as expected is precisely
the pointwise mutual information:
logDice(O) - logDice(E) = log2(O / E) = pmi
The level a pair starts from thus depends on the marginal frequencies
alone. Among the collocates of Grund in a 5+5 window, the rare
"triftiger" starts at -4.60 and reaches 3.96, exceeding what is expected
by a pmi of 8.57, while the frequent "Berlin" starts at 5.58 and still
reaches 3.84 while co-occurring 1.74 bits less often than expected.
This is a property of the measure rather than a defect: logDice was
designed to be corpus size independent and comparable across corpora,
not to express surprise. But since collocationAnalysis() ranks and
thresholds by it, users should know to drop pairs that are not attested
more often than expected, with dplyr::filter(O > E) or a minimum pmi or
ll. This is now explained in a section of the association score
documentation, referred to from the thresholdScore parameter where the
default takes effect, and summarized in the Readme.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I9059cd064e2a612d9327cc009da0b4856612e41d
diff --git a/R/collocationAnalysis.R b/R/collocationAnalysis.R
index 2d68636..34dc550 100644
--- a/R/collocationAnalysis.R
+++ b/R/collocationAnalysis.R
@@ -95,7 +95,12 @@
#' @param maxRecurse apply collocation analysis recursively `maxRecurse` times
#' @param addExamples If TRUE, examples for instances of collocations will be added in a column `example`. This makes a difference in particular if `node` is given as a lemma query.
#' @param thresholdScore association score function (see \code{\link{association-score-functions}}) to use for computing the threshold that is applied for recursive collocation analysis calls (only applied when \code{maxRecurse > 0})
-#' @param threshold minimum value of `thresholdScore` function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0})
+#' @param threshold minimum value of `thresholdScore` function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0}).
+#' Note that the default score, `logDice`, expresses how salient a pair is
+#' rather than how surprising, so that a frequent collocate can pass it while
+#' co-occurring less often than expected. Adding `dplyr::filter(O > E)`, or a
+#' minimum `pmi` or `ll`, removes those. See the "Salience versus surprise"
+#' section of \code{\link{association-score-functions}}.
#' @param localStopwords vector of stopwords that will not be considered as collocates in the current function call, but that will not be passed to recursive calls
#' @param collocateFilterRegex allow only collocates matching the regular expression
#' @param queryMissingScores if TRUE, attempt to retrieve corpus-based association scores for vc/collocate combinations that would otherwise be imputed, by re-querying the KorAP backend without applying the collocate frequency threshold