Compute logDice as defined by Rychly, comparable to other tools
logDice was computed as 14 + log2(2 * O / (w * O1 + O2)), multiplying
the node frequency by the window size, while the documentation cites
Rychly (2008), who defines it as 14 + log2(2 * O / (O1 + O2)).
The window size factor does not belong here. It is correct in the
expected frequency E, where w * O1 counts the positions in which a
collocate can occur, and the other scores take the window into account
through E alone. The Dice coefficient, however, relates the
co-occurrence frequency to how often the two words occur at all, so its
denominator sums marginal frequencies. Adding w * O1, a count of window
positions, to O2, a count of word tokens, mixes units, and it makes the
coefficient asymmetric: for Grund/triftiger it gave 0.64, for
triftiger/Grund 3.96, from the same counts.
The practical effect was that a frequent node made w * O1 dominate the
denominator - for Grund and triftiger in a 5+5 window, O2 contributed
0.006% of it - so that rare collocates were penalised. triftiger scored
0.64, below Berlin at 2.03, although Berlin co-occurs with Grund less
often than chance predicts (pmi -1.74). They are now 3.96 and 3.84.
Scores rise by up to log2(w), so up to 3.32 for the default context of
5 left and 5 right, which makes collocationAnalysis() somewhat more
permissive when recursing with the default thresholdScore = "logDice"
and threshold = 2. Results computed with a total window size of 1 are
unaffected, the two formulas being identical there, which is why the
light verb construction table in the Readme stays as it is, while the
collocationScoreQuery example, which uses the default window, was rerun.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I190fd65d0c53517df74c26c007a6440d974f700c
diff --git a/NEWS.md b/NEWS.md
index 49b4319..efa0a7e 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,5 +1,7 @@
# unpublished dev version 1.3.0.9000
+- **changed `logDice` values**: `logDice()` is now computed as defined by Rychlý (2008), `14 + log2(2 * O / (O1 + O2))`, so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, `14 + log2(2 * O / (w * O1 + O2))`, which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because `w * O1` dominated the denominator for a frequent node, rare collocates were penalised: for *triftiger* as a collocate of *Grund* in a 5+5 window, logDice was 0.64, below *Berlin* at 2.03, although *Berlin* co-occurs with *Grund* less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to `log2(w)`, that is by up to 3.32 for the default context of 5 left and 5 right, so `collocationAnalysis()` with the default `thresholdScore = "logDice"` and `threshold = 2` is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency `E`, which is correct
+
- `collocationAnalysis()` now stores the analysis parameters in its `cacheAs` file and compares them on the next call. If they differ, the cached result is not the one that was asked for, so it is recomputed and the file overwritten, with a warning naming the parameters that differ. This catches the case of a parameter being changed while an old cache file is still lying around. Cache files written by 1.3.0 do not contain the parameters yet and keep being used as they are
- dropped the `PTXQC` dependency, which was imported for two small string functions (`lcpCount()` and `lcsCount()`, used by `queryStringToLabel()`) but pulled in `rmzqc`, `jsonvalidate` and `V8`, and with them the only dependency requiring a `libv8` installation. The two functions are now implemented in the package itself, 10 to 65 times faster than the originals, and with unchanged results
diff --git a/R/association-scores.R b/R/association-scores.R
index 741e69f..fec7991 100644
--- a/R/association-scores.R
+++ b/R/association-scores.R
@@ -82,6 +82,16 @@
#'
#' @description
#' **logDice**: log-Dice coefficient, a heuristic measure that is popular in lexicography (Rychlý 2008)
+#'
+#' @details
+#' `logDice` is computed as defined by Rychlý (2008), that is from the plain
+#' marginal frequencies of node and collocate, so that its values are
+#' comparable to those of other tools, such as Sketch Engine. Unlike the
+#' expectation based scores, it does not take the window size into account: the
+#' Dice coefficient relates the co-occurrence frequency to how often the two
+#' words occur at all, and `O1` and `O2` count word tokens, while a window size
+#' factor would count window positions.
+#'
#' @export
#'
#' @references
@@ -89,7 +99,7 @@
#'
logDice <- function(O1, O2, O, N, E, window_size) {
- 14 + log2(2 * O / (window_size * O1 + O2))
+ 14 + log2(2 * O / (O1 + O2))
}
@@ -111,6 +121,28 @@
#'
ll <- function(O1, O2, O, N, E, window_size) {
r1 = as.double(O1) * window_size
+
+ # The contingency table classifies all N corpus tokens by whether they are
+ # inside a window around the node. That breaks down once the windows, counted
+ # as window_size * O1, cover more than the corpus, which happens for a very
+ # frequent node combined with a wide window: the table would get a negative
+ # cell and the score would silently become NaN.
+ exceedsCorpus = !is.na(r1) & r1 >= as.double(N)
+ if (any(exceedsCorpus)) {
+ warning(
+ sprintf(
+ paste0(
+ "Log-likelihood is not defined where the windows around the node cover the whole corpus ",
+ "(window_size * O1 >= N, affecting %d of %d values). Returning NA for these. ",
+ "Use a smaller window."
+ ),
+ sum(exceedsCorpus), length(exceedsCorpus)
+ ),
+ call. = FALSE
+ )
+ }
+ r1 = dplyr::if_else(exceedsCorpus, NA_real_, r1)
+
r2 = as.double(N) - r1
c1 = O2
c2 = N - c1
diff --git a/Readme.md b/Readme.md
index c492359..21de106 100644
--- a/Readme.md
+++ b/Readme.md
@@ -143,11 +143,11 @@
|node |collocate | O| E| logDice| pmi| ll|
|:-----|:---------|--------:|---------:|-------:|-----:|--------:|
-|Grund |triftiger | 2390.06| 6.31| 0.64| 8.57| 26288.28|
-|Grund |guter | 12902.28| 2713.05| 3.04| 2.25| 19938.66|
-|Grund |Berlin | 7866.48| 26211.71| 2.03| -1.74| 17789.94|
+|Grund |triftiger | 2390.50| 6.31| 3.96| 8.57| 26287.80|
+|Grund |guter | 12902.50| 2713.24| 6.06| 2.25| 19938.70|
+|Grund |Berlin | 7865.50| 26212.26| 3.84| -1.74| 17790.28|
-`O` is the observed and `E` the expected co-occurrence frequency. *Triftiger* is by far the most strongly attracted of the three (highest `pmi`), while *Berlin* co-occurs with *Grund* less often than chance would predict, which is what a negative `pmi` expresses.
+`O` is the observed and `E` the expected co-occurrence frequency. *Triftiger* is by far the most strongly attracted of the three (highest `pmi`), while *Berlin* co-occurs with *Grund* less often than chance would predict, which is what a negative `pmi` expresses. `logDice`, in contrast, does not compare against an expected frequency but relates the co-occurrence frequency to how often the two words occur at all, which is why the frequent *guter Grund* leads there.
### Identify *in … setzen* light verb constructions using `collocationAnalysis`
diff --git a/man/association-score-functions.Rd b/man/association-score-functions.Rd
index 74a2864..78b5ec4 100644
--- a/man/association-score-functions.Rd
+++ b/man/association-score-functions.Rd
@@ -56,6 +56,15 @@
\strong{ll}: log-likelihood (Dunning 1993) using Stefan Evert's (2004) simplified implementation
}
+\details{
+\code{logDice} is computed as defined by Rychlý (2008), that is from the plain
+marginal frequencies of node and collocate, so that its values are
+comparable to those of other tools, such as Sketch Engine. Unlike the
+expectation based scores, it does not take the window size into account: the
+Dice coefficient relates the co-occurrence frequency to how often the two
+words occur at all, and \code{O1} and \code{O2} count word tokens, while a window size
+factor would count window positions.
+}
\examples{
\dontrun{
diff --git a/tests/testthat/test-association-score-functions.R b/tests/testthat/test-association-score-functions.R
index f7228d3..3df08c7 100644
--- a/tests/testthat/test-association-score-functions.R
+++ b/tests/testthat/test-association-score-functions.R
@@ -4,7 +4,9 @@
expect_that(x[["pmi"]], equals(2.886331, tolerance=0.01))
expect_that(x[["mi2"]], equals(7.886331, tolerance=0.01))
expect_that(x[["mi3"]], equals(12.886331, tolerance=0.01))
- expect_that(x[["logDice"]], equals(-5.34404, tolerance=0.01))
+ # logDice follows Rychly (2008): 14 + log2(2 * O / (O1 + O2)), without a
+ # window size factor, so that values are comparable to other tools
+ expect_that(x[["logDice"]], equals(-2.022772, tolerance=0.01))
x <- sapply(defaultAssociationScoreFunctions(), mapply, 4258869, 2165, 0, 21304641202, 4.327907, 10)
expect_that(x[["ll"]], equals(8.664477, tolerance=0.01))