Scale the whole log-likelihood table with the window size
ca_ll's contingency table multiplied only its row total by the window
size, leaving cells that do not add up to one sample: the row was
counted in window positions while the column and the sample size were
counted in corpus tokens.
Following Evert (2004), which the function cites, the sample consists of
co-occurrence tokens rather than corpus tokens. With a window of
window_size positions, every occurrence of either word takes part in
that many pairs, so the sample size and both marginals scale with it.
The expected co-occurrence frequency comes out as window_size * O1 * O2
/ N either way, which is why pmi, mi2 and mi3 never depended on this and
why it went unnoticed for so long. Only ll needs the rest of the table,
and there the values differ, by 0.1% for a frequent collocate up to well
over 100% for a frequent node in a wide window. The ll column of the
collocationScoreQuery example in the Readme was recomputed accordingly.
The light verb construction table is unaffected, its total window size
being 1, where the two formulations are identical.
It also removes a pathology rather than papering over it: N -
window_size * O1 could turn negative, at which point the score became
NaN through a log of a negative number. window_size * (N - O1) cannot,
so the warning added earlier in this development version is dropped
again - it treated a symptom of exactly this inconsistency.
Reported-by: Tim Feldmüller <https://github.com/feldmueller>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I8b912c5bd870a0c9913d019043b2447f951f1891
diff --git a/NEWS.md b/NEWS.md
index e1c4df8..43a233c 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -3,7 +3,7 @@
- **`collocationAnalysis()` now discards collocates that occur less often than expected** by chance. `logDice`, by which it ranks and thresholds, expresses how salient a pair is rather than how surprising, so a frequent word could appear among the top collocates although the node does not attract it at all: for *Grund* in a 5+5 window, *Berlin* reaches a logDice of 3.84, close to *triftiger* at 3.96, while co-occurring 1.74 bits *less* often than expected. The new `minObservedExpectedRatio` parameter defaults to 1 and keeps such pairs out. Raise it to demand a stronger contrast, e.g. 2 for collocates occurring at least twice as often as expected, or set it to 0 for the unfiltered result of earlier versions, e.g. to study repulsion. `collocationScoreQuery()` is unaffected, as there the pairs to score are given explicitly
- **changed `logDice` values**: `logDice()` is now computed as defined by Rychlý (2008), `14 + log2(2 * O / (O1 + O2))`, so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, `14 + log2(2 * O / (w * O1 + O2))`, which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because `w * O1` dominated the denominator for a frequent node, rare collocates were penalised: for *triftiger* as a collocate of *Grund* in a 5+5 window, logDice was 0.64, below *Berlin* at 2.03, although *Berlin* co-occurs with *Grund* less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to `log2(w)`, that is by up to 3.32 for the default context of 5 left and 5 right, so `collocationAnalysis()` with the default `thresholdScore = "logDice"` and `threshold = 2` is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency `E`, which is correct
-- `ll()` now returns `NA` with a warning instead of a silent `NaN` where the windows around the node would cover more than the whole corpus (`window_size * O1 >= N`), which its contingency table cannot represent. This needs a node covering more than `1/w` of the corpus, so in DeReKo it is only reachable for the most frequent words combined with a wide window, e.g. *der* from a context of 18 left and 18 right
+- **changed `ll()` values**: the contingency table of `ll()` scaled only its row total by the window size, an inconsistency spotted by [Tim Feldmüller](https://github.com/feldmueller), leaving cells that do not add up to one sample. Following Evert (2004), the sample consists of co-occurrence tokens, so the sample size and both marginals scale with the window: an occurrence of either word takes part in `window_size` pairs. The expected co-occurrence frequency is unchanged at `window_size * O1 * O2 / N`, which is why `pmi`, `mi2` and `mi3` are unaffected, but log-likelihood values differ, by 0.1% to well over 100% depending on the frequencies and the window. This also removes the case where `N - window_size * O1` turned negative and the score became `NaN`, since `window_size * (N - O1)` cannot: the warning added earlier in this development version is therefore gone again, having treated a symptom of this
- `collocationAnalysis()` now stores the analysis parameters in its `cacheAs` file and compares them on the next call. If they differ, the cached result is not the one that was asked for, so it is recomputed and the file overwritten, with a warning naming the parameters that differ. This catches the case of a parameter being changed while an old cache file is still lying around. Cache files written by 1.3.0 do not contain the parameters yet and keep being used as they are
diff --git a/R/association-scores.R b/R/association-scores.R
index 7c5b081..21a569b 100644
--- a/R/association-scores.R
+++ b/R/association-scores.R
@@ -153,40 +153,27 @@
#' Free PDF available from <https://purl.org/stefan.evert/PUB/Evert2004phd.pdf>
#'
ll <- function(O1, O2, O, N, E, window_size) {
+ # The contingency table classifies co-occurrence tokens, not corpus tokens:
+ # with a window of `window_size` positions, every occurrence of a word takes
+ # part in that many pairs, so all of the sample size and both marginals scale
+ # with it. Scaling only the row, as this did before, leaves a table whose
+ # cells do not add up to one sample, and lets `N - window_size * O1` turn
+ # negative for a frequent node in a wide window. The expected co-occurrence
+ # frequency is unaffected, being `window_size * O1 * O2 / N` either way, which
+ # is why the other scores do not depend on this.
+ total = as.double(N) * window_size
r1 = as.double(O1) * window_size
-
- # The contingency table classifies all N corpus tokens by whether they are
- # inside a window around the node. That breaks down once the windows, counted
- # as window_size * O1, cover more than the corpus, which happens for a very
- # frequent node combined with a wide window: the table would get a negative
- # cell and the score would silently become NaN.
- exceedsCorpus = !is.na(r1) & r1 >= as.double(N)
- if (any(exceedsCorpus)) {
- warning(
- sprintf(
- paste0(
- "Log-likelihood is not defined where the windows around the node cover the whole corpus ",
- "(window_size * O1 >= N, affecting %d of %d values). Returning NA for these. ",
- "Use a smaller window."
- ),
- sum(exceedsCorpus), length(exceedsCorpus)
- ),
- call. = FALSE
- )
- }
- r1 = dplyr::if_else(exceedsCorpus, NA_real_, r1)
-
- r2 = as.double(N) - r1
- c1 = O2
- c2 = N - c1
+ r2 = total - r1
+ c1 = as.double(O2) * window_size
+ c2 = total - c1
o11 = O
o12 = r1 - o11
- o21 = c1 - O
+ o21 = c1 - o11
o22 = r2 - o21
- e11 = r1 * c1 / N
- e12 = r1 * c2 / N
- e21 = r2 * c1 / N
- e22 = r2 * c2 / N
+ e11 = r1 * c1 / total
+ e12 = r1 * c2 / total
+ e21 = r2 * c1 / total
+ e22 = r2 * c2 / total
2 * ( dplyr::if_else(o11>0, o11 * log(o11/e11), 0)
+ dplyr::if_else(o12>0, o12 * log(o12/e12), 0)
+ dplyr::if_else(o21>0, o21 * log(o21/e21), 0)
diff --git a/Readme.md b/Readme.md
index 166cb6c..d38a7a9 100644
--- a/Readme.md
+++ b/Readme.md
@@ -143,9 +143,9 @@
|node |collocate | O| E| logDice| pmi| ll|
|:-----|:---------|--------:|---------:|-------:|-----:|--------:|
-|Grund |triftiger | 2390.50| 6.31| 3.96| 8.57| 26287.80|
-|Grund |guter | 12902.50| 2713.24| 6.06| 2.25| 19938.70|
-|Grund |Berlin | 7865.50| 26212.26| 3.84| -1.74| 17790.28|
+|Grund |triftiger | 2390.50| 6.31| 3.96| 8.57| 23808.61|
+|Grund |guter | 12902.50| 2713.24| 6.06| 2.25| 19868.49|
+|Grund |Berlin | 7865.50| 26212.26| 3.84| -1.74| 17766.78|
`O` is the observed and `E` the expected co-occurrence frequency. *Triftiger* is by far the most strongly attracted of the three (highest `pmi`), while *Berlin* co-occurs with *Grund* less often than chance would predict, which is what a negative `pmi` expresses. `logDice`, in contrast, does not compare against an expected frequency but relates the co-occurrence frequency to how often the two words occur at all, which is why the frequent *guter Grund* leads there – and why *Berlin*, although it co-occurs with *Grund* less often than expected, still scores nearly as high as *triftiger*. `collocationAnalysis` therefore discards collocates occurring less often than expected by default, which its `minObservedExpectedRatio` parameter controls. `collocationScoreQuery` does not filter, since here the pairs to score are asked for explicitly – which is why *Berlin* is shown above.
@@ -400,6 +400,8 @@
**Authors**: [Marc Kupietz](https://www.ids-mannheim.de/digspra/personal/kupietz/), [Nils Diewald](https://www.ids-mannheim.de/digspra/personal/diewald/)
+**Contributors**: [Tim Feldmüller](https://github.com/feldmueller)
+
Copyright (c) 2026, [Leibniz Institute for the German Language](http://www.ids-mannheim.de/), Mannheim, Germany
This package is developed as part of the [KorAP](http://korap.ids-mannheim.de/)
diff --git a/tests/testthat/test-association-score-functions.R b/tests/testthat/test-association-score-functions.R
index 64f712f..0b14d6e 100644
--- a/tests/testthat/test-association-score-functions.R
+++ b/tests/testthat/test-association-score-functions.R
@@ -1,6 +1,6 @@
test_that("association scores are calculated correctly", {
x <- sapply(defaultAssociationScoreFunctions(), mapply, 4258869, 2165, 32, 21304641202, 4.327907, 10)
- expect_that(x[["ll"]], equals(73.05347, tolerance=0.01))
+ expect_that(x[["ll"]], equals(72.73295, tolerance=0.01))
expect_that(x[["pmi"]], equals(2.886331, tolerance=0.01))
expect_that(x[["mi2"]], equals(7.886331, tolerance=0.01))
expect_that(x[["mi3"]], equals(12.886331, tolerance=0.01))
@@ -9,7 +9,7 @@
expect_that(x[["logDice"]], equals(-2.022772, tolerance=0.01))
x <- sapply(defaultAssociationScoreFunctions(), mapply, 4258869, 2165, 0, 21304641202, 4.327907, 10)
- expect_that(x[["ll"]], equals(8.664477, tolerance=0.01))
+ expect_that(x[["ll"]], equals(8.656744, tolerance=0.01))
expect_equal(x[["pmi"]], -Inf)
expect_equal(x[["mi2"]], -Inf)
expect_equal(x[["mi3"]], -Inf)