CI: run the doc prompting tests in their own, non-blocking job
The Readme documentation prompting tests query LLMs, so they are neither
deterministic nor free: a model can miss an expectation on one run and
meet it on the next, and an empty API account fails them regardless of
the state of the package. Letting them decide whether the pipeline is
green means red builds for commits that changed nothing related.
They therefore move into their own job with allow_failure: true, where a
failure is visible as what it is, a hint that the documentation may have
become unclear, without blocking anything. That job runs GLM-5.3-Flash,
which solved all three documentation tasks in every trial run, while
gemini-3.5-flash-lite still missed an expectation every few runs.
Keeping them out of the main job needs more than a testthat filter,
since devtools::check() runs the tests as well, and blanking the API
keys in the job would not work either, because project level variables
set in the GitLab UI take precedence over those in the yml. The model
list is used as the switch instead: RKORAP_LLM_MODELS unset means the
default models, set but empty means none at all, so the main job
generates no LLM tests in either of its two steps.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia1f29f05a0b8401f61307859e97948f4be977565
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 12e8b04..de5db45 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,6 +4,11 @@
build-rkorapclient-on-stable:
extends: .install-rkorapclient
image: rocker/tidyverse
+ variables:
+ # The Readme documentation prompting tests query LLMs and are therefore neither
+ # deterministic nor free. They run in their own job below, so that neither
+ # a model's mood nor an empty API account can turn this one red.
+ RKORAP_LLM_MODELS: ""
cache:
key: rkorapclient-ubuntu-stable
paths:
@@ -17,3 +22,29 @@
- R -q -e 'devtools::test()'
- end_section testing
+# Prompts LLMs with the Readme, to check that it still enables them to use the
+# package. A failure here is a hint that the documentation has become unclear
+# (or that a model had a bad day), not a broken build, hence allow_failure.
+readme-doc-prompting-tests:
+ extends: .install-rkorapclient
+ image: rocker/tidyverse
+ allow_failure: true
+ variables:
+ # GLM-5.3-Flash solves all documentation tasks reliably, while the other
+ # low cost models are not there yet. Add them back here, comma separated,
+ # once they are: "gemini-3.5-flash-lite", "claude-sonnet-5", "gpt-5-nano".
+ # Do not set this as a project variable in the GitLab UI: those take
+ # precedence over the yml and would re-enable the tests in the job above.
+ RKORAP_LLM_MODELS: "hf:zai-org/GLM-5.3-Flash"
+ cache:
+ key: rkorapclient-ubuntu-stable
+ paths:
+ - apt-cache/
+ - ccache/
+ - rlib/
+ script:
+ - start_section docprompting "Prompting LLMs with the Readme"
+ - echo -e "apt-cache\nccache\nrlib" >> .Rbuildignore
+ - R -q -e 'devtools::test(filter = "readme-against-llm")'
+ - end_section docprompting
+
diff --git a/tests/testthat/test-readme-against-llm.R b/tests/testthat/test-readme-against-llm.R
index 8c35df4..886ccf6 100644
--- a/tests/testthat/test-readme-against-llm.R
+++ b/tests/testthat/test-readme-against-llm.R
@@ -10,12 +10,14 @@
)
llmModels <- function() {
- configured <- Sys.getenv("RKORAP_LLM_MODELS", unset = "")
- if (nzchar(configured)) {
- trimws(strsplit(configured, ",", fixed = TRUE)[[1]])
- } else {
- defaultLlmModels
+ configured <- Sys.getenv("RKORAP_LLM_MODELS", unset = NA_character_)
+ if (is.na(configured)) {
+ return(defaultLlmModels)
}
+ # set but empty means no models at all, which is how these tests are kept out
+ # of the pipeline job that runs everything else
+ models <- trimws(strsplit(configured, ",", fixed = TRUE)[[1]])
+ models[nzchar(models)]
}
# Provider and API key environment variable belonging to a model id