NEWS: put the cacheAs changes into one entry
Six entries described what is one story - cacheAs reaching the other
query functions, recording what produced a result, refusing a file that
does not match, and the three functions around it - and a changelog is
read to find out what changed, not to follow how it was built. They are
now one, and the entry from the development cycle about parameters being
recorded goes into it as well, that being the same thing said earlier.
The vc labels entry says "consistently", which is what the change is
about: collocationAnalysis() did this all along.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I24b6ec706b3e8b8c18ea61aacc96308ee9d21793
diff --git a/NEWS.md b/NEWS.md
index 2d265f8..e82f229 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,24 +1,14 @@
# unpublished dev version 1.3.0.9000
-- **`blessCacheAs()`** vouches for a cache file that an older version wrote, so that it is used again as it is. Files from the 1.3.0.9000 development cycle already hold correctly computed scores, the corrections having landed there, but say nothing about the version that wrote them and would otherwise be recomputed. What actually produced a file is left as it stands and the blessing recorded beside it, so `cacheAsInfo()` keeps saying where the numbers come from. A file recording no parameters has nothing left to compare once blessed, and is then reused for any call naming it
+- **`cacheAs` now covers the other query functions** – `frequencyQuery()`, `corpusStats()`, `collocationScoreQuery()` and `textMetadata()` – where it is less about time than about a result that stays what it was while the corpus grows. A file records what produced it: the parameters, the KorAP instance, its index revision and the package version. It is reused only for the call that wrote it, and one from before 1.4.0 not at all, its scores having been computed differently. `cacheAsInfo()` reads that record back, `blessCacheAs()` vouches for a file that is sound regardless, and `withCachedResults()` takes the files as they are for one expression, `mode = "offline"` refusing to compute anything that is not in one already
-- **`withCachedResults()`** takes cache files as they are while an expression is evaluated, `mode = "offline"` additionally refusing to compute anything that is not in a file already - for when a talk starts in ten minutes and a collocation analysis takes hours. The same is available document-wide through `options(rkorap.cacheAs=)` or the `KORAP_CACHE_AS` environment variable, with the values `"check"` (the default), `"reuse"` and `"offline"`. A file used despite not matching is still reported, so that outdated numbers do not pass unremarked
-
-- **`cacheAsInfo()`** reads back what a `cacheAs` file was produced by: the parameters of the call, the KorAP instance, the index revision its corpus had at the time, and the version of RKorAPClient that wrote it. For a result kept next to a document, that is what says which numbers it rests on
-
-- **`cacheAs` is now offered by `frequencyQuery()`, `corpusStats()`, `collocationScoreQuery()` and `textMetadata()`** as well, not only by `collocationAnalysis()`. It is a different thing from the connection's `cache`, which is a transparent speed-up: a `cacheAs` file belongs to the caller and is what keeps an analysis reproducible, since KorAP corpora grow and the same query returns different numbers next year. That is worth having for the quick functions too, where caching for speed would be pointless
-
-- **`cacheAs` files record the version that wrote them** and are refused, with a warning, when that is older than 1.4.0. Their contents are finished results including the association scores, which this version computes differently, so an old file would silently hand back numbers that would not be arrived at again - something no comparison of parameters can notice. The file is then recomputed and overwritten; pass a different name to keep it. How loud a query is no longer counts as a parameter either: `verbose` does not change what is returned
-
-- **the names of a named `vc` vector are now used as labels** by `frequencyQuery()`, `corpusStats()` and `collocationScoreQuery()`, as `collocationAnalysis()` already did. `frequencyQuery()` ignored them, `corpusStats()` put them into row names, which the first `bind_rows()` drops, and `collocationScoreQuery()` derived a label from the corpus definitions instead, so that `c(before = ..., since = ...)` came out as `"1990 & pubDat…"`. Where a vector carries no names, nothing changes: no `label` column appears that was not there before
+- **the names of a named `vc` vector are now consistently used as labels** by `frequencyQuery()`, `corpusStats()` and `collocationScoreQuery()`, as `collocationAnalysis()` already did. `frequencyQuery()` ignored them, `corpusStats()` put them into row names, which the first `bind_rows()` drops, and `collocationScoreQuery()` derived a label from the corpus definitions instead, so that `c(before = ..., since = ...)` came out as `"1990 & pubDat…"`. Where a vector carries no names, nothing changes: no `label` column appears that was not there before
- **`collocationAnalysis()` now discards collocates that occur less often than expected** by chance. `logDice`, by which it ranks and thresholds, expresses how salient a pair is rather than how surprising, so a frequent word could appear among the top collocates although the node does not attract it at all: for *Grund* in a 5+5 window, *Berlin* reaches a logDice of 3.84, close to *triftiger* at 3.96, while co-occurring 1.74 bits *less* often than expected. The new `minObservedExpectedRatio` parameter defaults to 1 and keeps such pairs out. Raise it to demand a stronger contrast, e.g. 2 for collocates occurring at least twice as often as expected, or set it to 0 for the unfiltered result of earlier versions, e.g. to study repulsion. `collocationScoreQuery()` is unaffected, as there the pairs to score are given explicitly
- **changed `logDice` values**: `logDice()` is now computed as defined by Rychlý (2008), `14 + log2(2 * O / (O1 + O2))`, so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, `14 + log2(2 * O / (w * O1 + O2))`, which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because `w * O1` dominated the denominator for a frequent node, rare collocates were penalised: for *triftiger* as a collocate of *Grund* in a 5+5 window, logDice was 0.64, below *Berlin* at 2.03, although *Berlin* co-occurs with *Grund* less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to `log2(w)`, that is by up to 3.32 for the default context of 5 left and 5 right, so `collocationAnalysis()` with the default `thresholdScore = "logDice"` and `threshold = 2` is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency `E`, which is correct
- **changed `ll()` values**: the contingency table of `ll()` scaled only its row total by the window size, an inconsistency spotted by [Tim Feldmüller](https://github.com/feldmueller), leaving cells that do not add up to one sample. Following Evert (2004), the sample consists of co-occurrence tokens, so the sample size and both marginals scale with the window: an occurrence of either word takes part in `window_size` pairs. The expected co-occurrence frequency is unchanged at `window_size * O1 * O2 / N`, which is why `pmi`, `mi2` and `mi3` are unaffected, but log-likelihood values differ, by 0.1% to well over 100% depending on the frequencies and the window. This also removes the case where `N - window_size * O1` turned negative and the score became `NaN`, since `window_size * (N - O1)` cannot: the warning added earlier in this development version is therefore gone again, having treated a symptom of this
-- `collocationAnalysis()` now stores the analysis parameters in its `cacheAs` file and compares them on the next call. If they differ, the cached result is not the one that was asked for, so it is recomputed and the file overwritten, with a warning naming the parameters that differ. This catches the case of a parameter being changed while an old cache file is still lying around. Cache files written by 1.3.0 do not contain the parameters yet and keep being used as they are
-
- dropped the `PTXQC` dependency, which was imported for two small string functions (`lcpCount()` and `lcsCount()`, used by `queryStringToLabel()`) but pulled in `rmzqc`, `jsonvalidate` and `V8`, and with them the only dependency requiring a `libv8` installation. The two functions are now implemented in the package itself, 10 to 65 times faster than the originals, and with unchanged results
# RKorAPClient 1.3.0