Guard log-likelihood against windows covering the whole corpus
The contingency table of ll() classifies all N corpus tokens by whether
they lie inside a window around the node, using window_size * O1 as the
row total. That stops being a partition once the windows cover more than
the corpus: N - window_size * O1 turns negative, and the score became
NaN through a log() of a negative number, with only a warning from log()
itself to indicate what had happened.
Such a node has to cover more than 1/w of the corpus, so with the
default window it is out of reach, but in DeReKo "der" (2.90% of all
tokens) gets there from a context of 18 left and 18 right, "die" from
19+19 and "und" from 24+24.
These cases now yield NA with a warning that names the condition and
suggests a smaller window. The check is vectorized, so that only the
affected values become NA.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I5e11396c67bc9103ec73745726d052857536df7f
diff --git a/NEWS.md b/NEWS.md
index efa0a7e..7cccf7b 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -2,6 +2,8 @@
- **changed `logDice` values**: `logDice()` is now computed as defined by Rychlý (2008), `14 + log2(2 * O / (O1 + O2))`, so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, `14 + log2(2 * O / (w * O1 + O2))`, which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because `w * O1` dominated the denominator for a frequent node, rare collocates were penalised: for *triftiger* as a collocate of *Grund* in a 5+5 window, logDice was 0.64, below *Berlin* at 2.03, although *Berlin* co-occurs with *Grund* less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to `log2(w)`, that is by up to 3.32 for the default context of 5 left and 5 right, so `collocationAnalysis()` with the default `thresholdScore = "logDice"` and `threshold = 2` is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency `E`, which is correct
+- `ll()` now returns `NA` with a warning instead of a silent `NaN` where the windows around the node would cover more than the whole corpus (`window_size * O1 >= N`), which its contingency table cannot represent. This needs a node covering more than `1/w` of the corpus, so in DeReKo it is only reachable for the most frequent words combined with a wide window, e.g. *der* from a context of 18 left and 18 right
+
- `collocationAnalysis()` now stores the analysis parameters in its `cacheAs` file and compares them on the next call. If they differ, the cached result is not the one that was asked for, so it is recomputed and the file overwritten, with a warning naming the parameters that differ. This catches the case of a parameter being changed while an old cache file is still lying around. Cache files written by 1.3.0 do not contain the parameters yet and keep being used as they are
- dropped the `PTXQC` dependency, which was imported for two small string functions (`lcpCount()` and `lcsCount()`, used by `queryStringToLabel()`) but pulled in `rmzqc`, `jsonvalidate` and `V8`, and with them the only dependency requiring a `libv8` installation. The two functions are now implemented in the package itself, 10 to 65 times faster than the originals, and with unchanged results