Document that logDice measures salience, not surprise
A collocate that co-occurs less often than expected can still reach a
logDice close to that of a strongly attracted one, which is puzzling
until one sees that logDice never compares against an expected
frequency. Since it only puts O into the numerator of the Dice
coefficient, the difference between the score of a pair and the score it
would get if it co-occurred exactly as often as expected is precisely
the pointwise mutual information:
logDice(O) - logDice(E) = log2(O / E) = pmi
The level a pair starts from thus depends on the marginal frequencies
alone. Among the collocates of Grund in a 5+5 window, the rare
"triftiger" starts at -4.60 and reaches 3.96, exceeding what is expected
by a pmi of 8.57, while the frequent "Berlin" starts at 5.58 and still
reaches 3.84 while co-occurring 1.74 bits less often than expected.
This is a property of the measure rather than a defect: logDice was
designed to be corpus size independent and comparable across corpora,
not to express surprise. But since collocationAnalysis() ranks and
thresholds by it, users should know to drop pairs that are not attested
more often than expected, with dplyr::filter(O > E) or a minimum pmi or
ll. This is now explained in a section of the association score
documentation, referred to from the thresholdScore parameter where the
default takes effect, and summarized in the Readme.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I9059cd064e2a612d9327cc009da0b4856612e41d
diff --git a/man/association-score-functions.Rd b/man/association-score-functions.Rd
index 78b5ec4..ce83ef9 100644
--- a/man/association-score-functions.Rd
+++ b/man/association-score-functions.Rd
@@ -65,6 +65,39 @@
words occur at all, and \code{O1} and \code{O2} count word tokens, while a window size
factor would count window positions.
}
+\section{Salience versus surprise}{
+
+
+\code{logDice} is the only one of these scores that does not compare the observed
+co-occurrence frequency to an expected one. As it merely puts \code{O} into the
+numerator of the Dice coefficient, the difference between the score of a pair
+and the score that same pair would get if it co-occurred exactly as often as
+expected is precisely the pointwise mutual information:
+
+\code{logDice(O) - logDice(E) = log2(O / E) = pmi}
+
+The level a pair starts from therefore depends on the marginal frequencies
+alone, and it is high whenever both words are frequent. Collocates of \emph{Grund}
+in DeReKo, in a window of five words to each side, illustrate this:
+\itemize{
+\item \emph{triftiger} is rare, so it starts from a logDice of -4.60 and reaches 3.96,
+exceeding what is expected by a \code{pmi} of 8.57
+\item \emph{Berlin} is frequent, so it starts from 5.58 and still reaches 3.84, while
+co-occurring 1.74 bits \emph{less} often than expected
+}
+
+A frequent collocate can thus reach a respectable logDice although the node
+does not attract it at all. logDice measures how salient a pair is, given how
+often its words occur, rather than how surprising it is. That is what it was
+designed for (Rychlý 2008), and it is why its values do not depend on the
+corpus size and are comparable across corpora.
+
+When ranking or thresholding by logDice, as \code{\link[=collocationAnalysis]{collocationAnalysis()}} does by
+default, it is therefore worth discarding pairs that are not attested more
+often than expected, with \code{dplyr::filter(O > E)}, or requiring a minimum
+\code{pmi} or \code{ll}.
+}
+
\examples{
\dontrun{
diff --git a/man/collocationAnalysis-KorAPConnection-method.Rd b/man/collocationAnalysis-KorAPConnection-method.Rd
index 734a2f6..5a0a43d 100644
--- a/man/collocationAnalysis-KorAPConnection-method.Rd
+++ b/man/collocationAnalysis-KorAPConnection-method.Rd
@@ -71,7 +71,12 @@
\item{thresholdScore}{association score function (see \code{\link{association-score-functions}}) to use for computing the threshold that is applied for recursive collocation analysis calls (only applied when \code{maxRecurse > 0})}
-\item{threshold}{minimum value of \code{thresholdScore} function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0})}
+\item{threshold}{minimum value of \code{thresholdScore} function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0}).
+Note that the default score, \code{logDice}, expresses how salient a pair is
+rather than how surprising, so that a frequent collocate can pass it while
+co-occurring less often than expected. Adding \code{dplyr::filter(O > E)}, or a
+minimum \code{pmi} or \code{ll}, removes those. See the "Salience versus surprise"
+section of \code{\link{association-score-functions}}.}
\item{localStopwords}{vector of stopwords that will not be considered as collocates in the current function call, but that will not be passed to recursive calls}