CA: drop collocates occurring less often than expected by default
collocationAnalysis() ranks and thresholds by logDice, which expresses
how salient a pair is rather than how surprising it is. A frequent word
can therefore appear among the top collocates although the node does not
attract it at all: among the collocates of Grund in a 5+5 window,
"Berlin" reaches a logDice of 3.84, close to "triftiger" at 3.96, while
co-occurring 1.74 bits less often than expected.
Filtering these out was documented as a recipe, dplyr::filter(O > E),
but a default that quietly rewards frequent, unattracted collocates is a
trap for exactly those users who do not know the measure well enough to
apply the recipe. The new minObservedExpectedRatio parameter therefore
defaults to 1, keeping only collocates that occur at least as often as
expected, which corresponds to a non-negative pmi. It can be raised to
demand a stronger contrast, or set to 0 for the unfiltered result of
earlier versions, e.g. in order to study repulsion.
The filter is applied wherever minOccur is, so it also governs which
collocates are recursed into, and it is passed on to the per virtual
corpus analyses. Rows without an expected frequency are kept rather than
silently dropped. collocationScoreQuery() is deliberately left alone:
there the pairs to score are given explicitly, so filtering them would
throw away the answer that was asked for.
Being collected from the call frame rather than from a list, the cacheAs
parameter check picked the new parameter up by itself, so a cache file
written with a different ratio is recomputed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I98328a8766847b4c95363211a939403b1912e9b7
diff --git a/R/collocationAnalysis.R b/R/collocationAnalysis.R
index 34dc550..c9b8695 100644
--- a/R/collocationAnalysis.R
+++ b/R/collocationAnalysis.R
@@ -1,6 +1,25 @@
#' @include logging.R
setGeneric("collocationAnalysis", function(kco, ...) standardGeneric("collocationAnalysis"))
+#' Keep only collocates that are attested often enough relative to expectation
+#'
+#' Rows without an expected frequency are kept, as are all rows if the ratio is
+#' 0 or `NULL`, which switches the filter off.
+#'
+#' @param result collocation analysis result
+#' @param minObservedExpectedRatio minimum ratio of observed to expected
+#' co-occurrence frequency
+#' @return `result` without the rows that fall below the ratio
+#' @noRd
+filterByObservedExpectedRatio <- function(result, minObservedExpectedRatio) {
+ if (is.null(minObservedExpectedRatio) || is.na(minObservedExpectedRatio) ||
+ minObservedExpectedRatio <= 0 || nrow(result) == 0 ||
+ !all(c("O", "E") %in% names(result))) {
+ return(result)
+ }
+ result[is.na(result$E) | result$O >= minObservedExpectedRatio * result$E, , drop = FALSE]
+}
+
#' Name of the attribute under which cache files record their analysis parameters
#' @noRd
collocationCacheAttribute <- "RKorAPClient.collocationAnalysis"
@@ -98,9 +117,19 @@
#' @param threshold minimum value of `thresholdScore` function call to apply collocation analysis recursively (only applied when \code{maxRecurse > 0}).
#' Note that the default score, `logDice`, expresses how salient a pair is
#' rather than how surprising, so that a frequent collocate can pass it while
-#' co-occurring less often than expected. Adding `dplyr::filter(O > E)`, or a
-#' minimum `pmi` or `ll`, removes those. See the "Salience versus surprise"
-#' section of \code{\link{association-score-functions}}.
+#' co-occurring less often than expected. `minObservedExpectedRatio` keeps
+#' those out. See the "Salience versus surprise" section of
+#' \code{\link{association-score-functions}}.
+#' @param minObservedExpectedRatio minimum ratio of observed to expected co-occurrence
+#' frequency a collocate must reach. Defaults to 1, which keeps only collocates
+#' that occur at least as often as expected by chance, corresponding to a
+#' non-negative `pmi`. Without it, frequent words can end up among the top
+#' collocates by `logDice` although the node does not attract them at all (see
+#' the "Salience versus surprise" section of
+#' \code{\link{association-score-functions}}). Raise it to demand a stronger
+#' contrast, e.g. 2 for collocates occurring at least twice as often as
+#' expected, or set it to 0 to switch the filter off and obtain the unfiltered
+#' result of earlier versions, e.g. in order to study repulsion.
#' @param localStopwords vector of stopwords that will not be considered as collocates in the current function call, but that will not be passed to recursive calls
#' @param collocateFilterRegex allow only collocates matching the regular expression
#' @param queryMissingScores if TRUE, attempt to retrieve corpus-based association scores for vc/collocate combinations that would otherwise be imputed, by re-querying the KorAP backend without applying the collocate frequency threshold
@@ -224,6 +253,7 @@
threshold = 2.0,
localStopwords = c(),
collocateFilterRegex = "^[:alnum:]+-?[:alnum:]*$",
+ minObservedExpectedRatio = 1,
queryMissingScores = FALSE,
missingScoreQuantile = 0.05,
vcLabel = NA_character_,
@@ -308,6 +338,7 @@
node = node,
vc = vc,
minOccur = minOccur,
+ minObservedExpectedRatio = minObservedExpectedRatio,
leftContextSize = leftContextSize,
rightContextSize = rightContextSize,
topCollocatesLimit = topCollocatesLimit,
@@ -402,6 +433,7 @@
...
) |>
filter(O >= minOccur) |>
+ filterByObservedExpectedRatio(minObservedExpectedRatio) |>
dplyr::arrange(dplyr::desc(logDice))
} else {
tibble()
@@ -433,6 +465,7 @@
rightContextSize = rightContextSize,
withinSpan = withinSpan,
maxRecurse = maxRecurse - 1,
+ minObservedExpectedRatio = minObservedExpectedRatio,
stopwords = stopwords,
localStopwords = recurseWith$collocate,
exactFrequencies = exactFrequencies,
@@ -456,6 +489,7 @@
result <- result |>
filter(O >= minOccur) |>
+ filterByObservedExpectedRatio(minObservedExpectedRatio) |>
dplyr::arrange(dplyr::desc(logDice))
}
}