CA: drop collocates occurring less often than expected by default
collocationAnalysis() ranks and thresholds by logDice, which expresses
how salient a pair is rather than how surprising it is. A frequent word
can therefore appear among the top collocates although the node does not
attract it at all: among the collocates of Grund in a 5+5 window,
"Berlin" reaches a logDice of 3.84, close to "triftiger" at 3.96, while
co-occurring 1.74 bits less often than expected.
Filtering these out was documented as a recipe, dplyr::filter(O > E),
but a default that quietly rewards frequent, unattracted collocates is a
trap for exactly those users who do not know the measure well enough to
apply the recipe. The new minObservedExpectedRatio parameter therefore
defaults to 1, keeping only collocates that occur at least as often as
expected, which corresponds to a non-negative pmi. It can be raised to
demand a stronger contrast, or set to 0 for the unfiltered result of
earlier versions, e.g. in order to study repulsion.
The filter is applied wherever minOccur is, so it also governs which
collocates are recursed into, and it is passed on to the per virtual
corpus analyses. Rows without an expected frequency are kept rather than
silently dropped. collocationScoreQuery() is deliberately left alone:
there the pairs to score are given explicitly, so filtering them would
throw away the answer that was asked for.
Being collected from the call frame rather than from a list, the cacheAs
parameter check picked the new parameter up by itself, so a cache file
written with a different ratio is recomputed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I98328a8766847b4c95363211a939403b1912e9b7
diff --git a/NEWS.md b/NEWS.md
index 7cccf7b..e1c4df8 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,5 +1,6 @@
# unpublished dev version 1.3.0.9000
+- **`collocationAnalysis()` now discards collocates that occur less often than expected** by chance. `logDice`, by which it ranks and thresholds, expresses how salient a pair is rather than how surprising, so a frequent word could appear among the top collocates although the node does not attract it at all: for *Grund* in a 5+5 window, *Berlin* reaches a logDice of 3.84, close to *triftiger* at 3.96, while co-occurring 1.74 bits *less* often than expected. The new `minObservedExpectedRatio` parameter defaults to 1 and keeps such pairs out. Raise it to demand a stronger contrast, e.g. 2 for collocates occurring at least twice as often as expected, or set it to 0 for the unfiltered result of earlier versions, e.g. to study repulsion. `collocationScoreQuery()` is unaffected, as there the pairs to score are given explicitly
- **changed `logDice` values**: `logDice()` is now computed as defined by Rychlý (2008), `14 + log2(2 * O / (O1 + O2))`, so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, `14 + log2(2 * O / (w * O1 + O2))`, which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because `w * O1` dominated the denominator for a frequent node, rare collocates were penalised: for *triftiger* as a collocate of *Grund* in a 5+5 window, logDice was 0.64, below *Berlin* at 2.03, although *Berlin* co-occurs with *Grund* less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to `log2(w)`, that is by up to 3.32 for the default context of 5 left and 5 right, so `collocationAnalysis()` with the default `thresholdScore = "logDice"` and `threshold = 2` is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency `E`, which is correct
- `ll()` now returns `NA` with a warning instead of a silent `NaN` where the windows around the node would cover more than the whole corpus (`window_size * O1 >= N`), which its contingency table cannot represent. This needs a node covering more than `1/w` of the corpus, so in DeReKo it is only reachable for the most frequent words combined with a wide window, e.g. *der* from a context of 18 left and 18 right
diff --git a/R/association-scores.R b/R/association-scores.R
index 7807c08..7c5b081 100644
--- a/R/association-scores.R
+++ b/R/association-scores.R
@@ -40,10 +40,13 @@
#' designed for (Rychlý 2008), and it is why its values do not depend on the
#' corpus size and are comparable across corpora.
#'
-#' When ranking or thresholding by logDice, as [collocationAnalysis()] does by
-#' default, it is therefore worth discarding pairs that are not attested more
-#' often than expected, with `dplyr::filter(O > E)`, or requiring a minimum
-#' `pmi` or `ll`.
+#' Since [collocationAnalysis()] ranks and thresholds by logDice, it therefore
+#' drops collocates occurring less often than expected by default. Its
+#' `minObservedExpectedRatio` parameter controls this: raise it to demand a
+#' stronger contrast, or set it to 0 to see the unfiltered ranking, for instance
+#' to study repulsion. [collocationScoreQuery()] does not filter, as there the
+#' pairs to score are given explicitly. For results obtained otherwise,
+#' `dplyr::filter(O > E)` or a minimum `pmi` or `ll` has the same effect.
NULL
#' NULL
diff --git a/R/collocationAnalysis.R b/R/collocationAnalysis.R
index 34dc550..c9b8695 100644
--- a/R/collocationAnalysis.R
+++ b/R/collocationAnalysis.R
@@ -1,6 +1,25 @@
#' @include logging.R
setGeneric("collocationAnalysis", function(kco, ...) standardGeneric("collocationAnalysis"))
+#' Keep only collocates that are attested often enough relative to expectation
+#'
+#' Rows without an expected frequency are kept, as are all rows if the ratio is
+#' 0 or `NULL`, which switches the filter off.
+#'
+#' @param result collocation analysis result
+#' @param minObservedExpectedRatio minimum ratio of observed to expected
+#' co-occurrence frequency
+#' @return `result` without the rows that fall below the ratio
+#' @noRd
+filterByObservedExpectedRatio <- function(result, minObservedExpectedRatio) {
+ if (is.null(minObservedExpectedRatio) || is.na(minObservedExpectedRatio) ||
+ minObservedExpectedRatio <= 0 || nrow(result) == 0 ||
+ !all(c("O", "E") %in% names(result))) {
+ return(result)
+ }
+ result[is.na(result$E) | result$O >= minObservedExpectedRatio * result$E, , drop = FALSE]
+}
+
#' Name of the attribute under which cache files record their analysis parameters
#' @noRd
collocationCacheAttribute <- "RKorAPClient.collocationAnalysis"
@@ -98,9 +117,19 @@
#' @param threshold minimum value of `thresholdScore` function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0}).
#' Note that the default score, `logDice`, expresses how salient a pair is
#' rather than how surprising, so that a frequent collocate can pass it while
-#' co-occurring less often than expected. Adding `dplyr::filter(O > E)`, or a
-#' minimum `pmi` or `ll`, removes those. See the "Salience versus surprise"
-#' section of \code{\link{association-score-functions}}.
+#' co-occurring less often than expected. `minObservedExpectedRatio` keeps
+#' those out. See the "Salience versus surprise" section of
+#' \code{\link{association-score-functions}}.
+#' @param minObservedExpectedRatio minimum ratio of observed to expected co-occurrence
+#' frequency a collocate must reach. Defaults to 1, which keeps only collocates
+#' that occur at least as often as expected by chance, corresponding to a
+#' non-negative `pmi`. Without it, frequent words can end up among the top
+#' collocates by `logDice` although the node does not attract them at all (see
+#' the "Salience versus surprise" section of
+#' \code{\link{association-score-functions}}). Raise it to demand a stronger
+#' contrast, e.g. 2 for collocates occurring at least twice as often as
+#' expected, or set it to 0 to switch the filter off and obtain the unfiltered
+#' result of earlier versions, e.g. in order to study repulsion.
#' @param localStopwords vector of stopwords that will not be considered as collocates in the current function call, but that will not be passed to recursive calls
#' @param collocateFilterRegex allow only collocates matching the regular expression
#' @param queryMissingScores if TRUE, attempt to retrieve corpus-based association scores for vc/collocate combinations that would otherwise be imputed, by re-querying the KorAP backend without applying the collocate frequency threshold
@@ -224,6 +253,7 @@
threshold = 2.0,
localStopwords = c(),
collocateFilterRegex = "^[:alnum:]+-?[:alnum:]*$",
+ minObservedExpectedRatio = 1,
queryMissingScores = FALSE,
missingScoreQuantile = 0.05,
vcLabel = NA_character_,
@@ -308,6 +338,7 @@
node = node,
vc = vc,
minOccur = minOccur,
+ minObservedExpectedRatio = minObservedExpectedRatio,
leftContextSize = leftContextSize,
rightContextSize = rightContextSize,
topCollocatesLimit = topCollocatesLimit,
@@ -402,6 +433,7 @@
...
) |>
filter(O >= minOccur) |>
+ filterByObservedExpectedRatio(minObservedExpectedRatio) |>
dplyr::arrange(dplyr::desc(logDice))
} else {
tibble()
@@ -433,6 +465,7 @@
rightContextSize = rightContextSize,
withinSpan = withinSpan,
maxRecurse = maxRecurse - 1,
+ minObservedExpectedRatio = minObservedExpectedRatio,
stopwords = stopwords,
localStopwords = recurseWith$collocate,
exactFrequencies = exactFrequencies,
@@ -456,6 +489,7 @@
result <- result |>
filter(O >= minOccur) |>
+ filterByObservedExpectedRatio(minObservedExpectedRatio) |>
dplyr::arrange(dplyr::desc(logDice))
}
}
diff --git a/Readme.md b/Readme.md
index d7d2e97..166cb6c 100644
--- a/Readme.md
+++ b/Readme.md
@@ -147,7 +147,7 @@
|Grund |guter | 12902.50| 2713.24| 6.06| 2.25| 19938.70|
|Grund |Berlin | 7865.50| 26212.26| 3.84| -1.74| 17790.28|
-`O` is the observed and `E` the expected co-occurrence frequency. *Triftiger* is by far the most strongly attracted of the three (highest `pmi`), while *Berlin* co-occurs with *Grund* less often than chance would predict, which is what a negative `pmi` expresses. `logDice`, in contrast, does not compare against an expected frequency but relates the co-occurrence frequency to how often the two words occur at all, which is why the frequent *guter Grund* leads there – and why *Berlin*, although it co-occurs with *Grund* less often than expected, still scores nearly as high as *triftiger*. When ranking by `logDice`, as `collocationAnalysis` does by default, `dplyr::filter(O > E)` or a minimum `pmi` discards such pairs.
+`O` is the observed and `E` the expected co-occurrence frequency. *Triftiger* is by far the most strongly attracted of the three (highest `pmi`), while *Berlin* co-occurs with *Grund* less often than chance would predict, which is what a negative `pmi` expresses. `logDice`, in contrast, does not compare against an expected frequency but relates the co-occurrence frequency to how often the two words occur at all, which is why the frequent *guter Grund* leads there – and why *Berlin*, although it co-occurs with *Grund* less often than expected, still scores nearly as high as *triftiger*. `collocationAnalysis` therefore discards collocates occurring less often than expected by default, which its `minObservedExpectedRatio` parameter controls. `collocationScoreQuery` does not filter, since here the pairs to score are asked for explicitly – which is why *Berlin* is shown above.
### Identify *in … setzen* light verb constructions using `collocationAnalysis`
diff --git a/man/association-score-functions.Rd b/man/association-score-functions.Rd
index ce83ef9..6e1afce 100644
--- a/man/association-score-functions.Rd
+++ b/man/association-score-functions.Rd
@@ -92,10 +92,13 @@
designed for (Rychlý 2008), and it is why its values do not depend on the
corpus size and are comparable across corpora.
-When ranking or thresholding by logDice, as \code{\link[=collocationAnalysis]{collocationAnalysis()}} does by
-default, it is therefore worth discarding pairs that are not attested more
-often than expected, with \code{dplyr::filter(O > E)}, or requiring a minimum
-\code{pmi} or \code{ll}.
+Since \code{\link[=collocationAnalysis]{collocationAnalysis()}} ranks and thresholds by logDice, it therefore
+drops collocates occurring less often than expected by default. Its
+\code{minObservedExpectedRatio} parameter controls this: raise it to demand a
+stronger contrast, or set it to 0 to see the unfiltered ranking, for instance
+to study repulsion. \code{\link[=collocationScoreQuery]{collocationScoreQuery()}} does not filter, as there the
+pairs to score are given explicitly. For results obtained otherwise,
+\code{dplyr::filter(O > E)} or a minimum \code{pmi} or \code{ll} has the same effect.
}
\examples{
diff --git a/man/collocationAnalysis-KorAPConnection-method.Rd b/man/collocationAnalysis-KorAPConnection-method.Rd
index 5a0a43d..ac7e7d9 100644
--- a/man/collocationAnalysis-KorAPConnection-method.Rd
+++ b/man/collocationAnalysis-KorAPConnection-method.Rd
@@ -27,6 +27,7 @@
threshold = 2,
localStopwords = c(),
collocateFilterRegex = "^[:alnum:]+-?[:alnum:]*$",
+ minObservedExpectedRatio = 1,
queryMissingScores = FALSE,
missingScoreQuantile = 0.05,
vcLabel = NA_character_,
@@ -74,14 +75,25 @@
\item{threshold}{minimum value of \code{thresholdScore} function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0}).
Note that the default score, \code{logDice}, expresses how salient a pair is
rather than how surprising, so that a frequent collocate can pass it while
-co-occurring less often than expected. Adding \code{dplyr::filter(O > E)}, or a
-minimum \code{pmi} or \code{ll}, removes those. See the "Salience versus surprise"
-section of \code{\link{association-score-functions}}.}
+co-occurring less often than expected. \code{minObservedExpectedRatio} keeps
+those out. See the "Salience versus surprise" section of
+\code{\link{association-score-functions}}.}
\item{localStopwords}{vector of stopwords that will not be considered as collocates in the current function call, but that will not be passed to recursive calls}
\item{collocateFilterRegex}{allow only collocates matching the regular expression}
+\item{minObservedExpectedRatio}{minimum ratio of observed to expected co-occurrence
+frequency a collocate must reach. Defaults to 1, which keeps only collocates
+that occur at least as often as expected by chance, corresponding to a
+non-negative \code{pmi}. Without it, frequent words can end up among the top
+collocates by \code{logDice} although the node does not attract them at all (see
+the "Salience versus surprise" section of
+\code{\link{association-score-functions}}). Raise it to demand a stronger
+contrast, e.g. 2 for collocates occurring at least twice as often as
+expected, or set it to 0 to switch the filter off and obtain the unfiltered
+result of earlier versions, e.g. in order to study repulsion.}
+
\item{queryMissingScores}{if TRUE, attempt to retrieve corpus-based association scores for vc/collocate combinations that would otherwise be imputed, by re-querying the KorAP backend without applying the collocate frequency threshold}
\item{missingScoreQuantile}{lower quantile (evaluated per association measure over the pooled result set) that anchors the adaptive floor used for imputing missing scores between virtual corpora; a robust spread is subtracted from this anchor so the imputed values stay at or below the weakest observed scores. Imputed cells are marked in the \verb{imputed*} columns; see the section on interpreting multi-VC comparisons below}
diff --git a/tests/testthat/test-association-score-functions.R b/tests/testthat/test-association-score-functions.R
index 3df08c7..64f712f 100644
--- a/tests/testthat/test-association-score-functions.R
+++ b/tests/testthat/test-association-score-functions.R
@@ -16,3 +16,26 @@
expect_equal(x[["logDice"]], -Inf)
})
+
+test_that("filterByObservedExpectedRatio keeps only attested enough collocates", {
+ filterBy <- RKorAPClient:::filterByObservedExpectedRatio
+ result <- tibble::tibble(
+ collocate = c("attracted", "asExpected", "repelled"),
+ O = c(100, 10, 1),
+ E = c(10, 10, 10)
+ )
+
+ expect_equal(filterBy(result, 1)$collocate, c("attracted", "asExpected"))
+ expect_equal(filterBy(result, 5)$collocate, "attracted")
+ # 0 and NULL switch the filter off, for studying repulsion for instance
+ expect_equal(nrow(filterBy(result, 0)), 3)
+ expect_equal(nrow(filterBy(result, NULL)), 3)
+
+ # rows without an expected frequency are kept rather than silently dropped
+ withNA <- tibble::tibble(collocate = "unknown", O = 1, E = NA_real_)
+ expect_equal(nrow(filterBy(withNA, 1)), 1)
+
+ # nothing to do without the columns, or without rows
+ expect_equal(nrow(filterBy(tibble::tibble(collocate = "x"), 1)), 1)
+ expect_equal(nrow(filterBy(result[0, ], 1)), 0)
+})