Record the documentation prompting tests, and cite them
They are nowhere in the Readme or the changelog, although they are what
the Readme is now checked by, and Kupietz/Feldmüller/Konieczny/Lang
(2026) names them as an example of a practice that keeps LLM-written
analysis code honest. A reader arriving from there found nothing.
Two sentences at the end of the Readme say what they do and how they are
switched on, the changelog says how far they now reach - ten tasks - and
the paper goes into the references, where it also belongs as a
publication about the package.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: I961cb651ef3485919ec22f0ed23a21ca4fdf262b
diff --git a/NEWS.md b/NEWS.md
index e82f229..af0ae77 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,15 +1,18 @@
# unpublished dev version 1.3.0.9000
-- **`cacheAs` now covers the other query functions** – `frequencyQuery()`, `corpusStats()`, `collocationScoreQuery()` and `textMetadata()` – where it is less about time than about a result that stays what it was while the corpus grows. A file records what produced it: the parameters, the KorAP instance, its index revision and the package version. It is reused only for the call that wrote it, and one from before 1.4.0 not at all, its scores having been computed differently. `cacheAsInfo()` reads that record back, `blessCacheAs()` vouches for a file that is sound regardless, and `withCachedResults()` takes the files as they are for one expression, `mode = "offline"` refusing to compute anything that is not in one already
+- **`cacheAs` now consistently covers all query functions** – `frequencyQuery()`, `corpusStats()`, `collocationScoreQuery()` and `textMetadata()`. The cache file records its relevant production context: parameters, KorAP instance, index revision and package version. `cacheAsInfo()` prints that record, `blessCacheAs()` vouches for a file that is sound regardless, and `withCachedResults()` takes the files as they are for one expression, `mode = "offline"` refusing to compute anything that is not in one already
-- **the names of a named `vc` vector are now consistently used as labels** by `frequencyQuery()`, `corpusStats()` and `collocationScoreQuery()`, as `collocationAnalysis()` already did. `frequencyQuery()` ignored them, `corpusStats()` put them into row names, which the first `bind_rows()` drops, and `collocationScoreQuery()` derived a label from the corpus definitions instead, so that `c(before = ..., since = ...)` came out as `"1990 & pubDat…"`. Where a vector carries no names, nothing changes: no `label` column appears that was not there before
+- **the names of a named `vc` vector are now consistently used as labels** by `frequencyQuery()`, `corpusStats()` and `collocationScoreQuery()`, as `collocationAnalysis()` already did.
+ Where a vector carries no names, nothing changes: no `label` column appears that was not there before
- **`collocationAnalysis()` now discards collocates that occur less often than expected** by chance. `logDice`, by which it ranks and thresholds, expresses how salient a pair is rather than how surprising, so a frequent word could appear among the top collocates although the node does not attract it at all: for *Grund* in a 5+5 window, *Berlin* reaches a logDice of 3.84, close to *triftiger* at 3.96, while co-occurring 1.74 bits *less* often than expected. The new `minObservedExpectedRatio` parameter defaults to 1 and keeps such pairs out. Raise it to demand a stronger contrast, e.g. 2 for collocates occurring at least twice as often as expected, or set it to 0 for the unfiltered result of earlier versions, e.g. to study repulsion. `collocationScoreQuery()` is unaffected, as there the pairs to score are given explicitly
- **changed `logDice` values**: `logDice()` is now computed as defined by Rychlý (2008), `14 + log2(2 * O / (O1 + O2))`, so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, `14 + log2(2 * O / (w * O1 + O2))`, which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because `w * O1` dominated the denominator for a frequent node, rare collocates were penalised: for *triftiger* as a collocate of *Grund* in a 5+5 window, logDice was 0.64, below *Berlin* at 2.03, although *Berlin* co-occurs with *Grund* less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to `log2(w)`, that is by up to 3.32 for the default context of 5 left and 5 right, so `collocationAnalysis()` with the default `thresholdScore = "logDice"` and `threshold = 2` is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency `E`, which is correct
- **changed `ll()` values**: the contingency table of `ll()` scaled only its row total by the window size, an inconsistency spotted by [Tim Feldmüller](https://github.com/feldmueller), leaving cells that do not add up to one sample. Following Evert (2004), the sample consists of co-occurrence tokens, so the sample size and both marginals scale with the window: an occurrence of either word takes part in `window_size` pairs. The expected co-occurrence frequency is unchanged at `window_size * O1 * O2 / N`, which is why `pmi`, `mi2` and `mi3` are unaffected, but log-likelihood values differ, by 0.1% to well over 100% depending on the frequencies and the window. This also removes the case where `N - window_size * O1` turned negative and the score became `NaN`, since `window_size * (N - O1)` cannot: the warning added earlier in this development version is therefore gone again, having treated a symptom of this
-- dropped the `PTXQC` dependency, which was imported for two small string functions (`lcpCount()` and `lcsCount()`, used by `queryStringToLabel()`) but pulled in `rmzqc`, `jsonvalidate` and `V8`, and with them the only dependency requiring a `libv8` installation. The two functions are now implemented in the package itself, 10 to 65 times faster than the originals, and with unchanged results
+- **improved coverage of the doc-prompting tests**, which now range from a frequency query over time to comparing collocates across virtual corpora, keeping a result in a `cacheAs` file and labelling corpora by name. They prompt current LLMs with the Readme and check the code written from it, so that a gap in the documentation shows up as a failing test. This guards the quality of the Readme and improves vibe coding results. The approach is briefly described in [Kupietz et al. (2026)](https://doi.org/10.37307/j.1868-775X.2026.02.08)
+
+- dropped the `PTXQC` dependency, which was imported for two small string functions (`lcpCount()` and `lcsCount()`, used by `queryStringToLabel()`) but pulled in `rmzqc`, `jsonvalidate` and `V8`, and with them the only dependency requiring a `libv8` installation.
# RKorAPClient 1.3.0
diff --git a/Readme.md b/Readme.md
index b7208dd..c8d7acf 100644
--- a/Readme.md
+++ b/Readme.md
@@ -467,6 +467,8 @@
contribution intentionally submitted for inclusion into this software shall –
as this software itself – be under the [BSD-2 License](LICENSE.md).
+Part of the test suite prompts current LLMs with this Readme and checks that the code they write from it uses the package correctly, so that a gap in the documentation shows up as a failing test – see Kupietz et al. (2026) on the approach. The tests are skipped without an API key; `RKORAP_LLM_MODELS` selects the models to prompt.
+
## References
- Diewald, Nils/Barbu Mititelu, Verginica/Kupietz, Marc (2019): The KorAP user interface. Accessing CoRoLa via KorAP. In: On design, creation and use of the Reference Corpus of Contemporary Romanian and its analysis tools. CoRoLa, KorAP, DRuKoLA and EuReCo. Edited by Ruxandra Cosma/Marc Kupietz, 64(3). <https://nbn-resolving.org/urn:nbn:de:bsz:mh39-93866>.
@@ -476,3 +478,5 @@
* Kupietz, Marc / Diewald, Nils / Margaretha, Eliza (2020): [RKorAPClient: An R package for accessing the German Reference Corpus DeReKo via KorAP](http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.867.pdf). In: Calzolari, Nicoletta, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis (eds.): [Proceedings of The 12th Language Resources and Evaluation Conference (LREC 2020)](http://www.lrec-conf.org/proceedings/lrec2020/LREC-2020.pdf). Marseille: European Language Resources Association (ELRA), 7017-7023.
* Kupietz, Marc/Diewald, Nils/Margaretha, Eliza (2022): Building paths to corpus data: A multi-level least effort and maximum return approach. In: Fišer, Darja/Witt, Andreas (eds.): CLARIN. The Infrastructure for Language Resources. Berlin: deGruyter, pp. 163–189. <https://doi.org/10.1515/9783110767377-007>.
+
+* Kupietz, Marc / Feldmüller, Tim / Konieczny, Lars / Lang, Christian (2026): [LLMs als Chance für die Linguistik](https://nbn-resolving.org/urn:nbn:de:bsz:mh39-138499). In: *Deutsche Sprache* 2026(2), 197–204. <https://doi.org/10.37307/j.1868-775X.2026.02.08>.