CI: run the doc prompting tests in their own, non-blocking job
The Readme documentation prompting tests query LLMs, so they are neither
deterministic nor free: a model can miss an expectation on one run and
meet it on the next, and an empty API account fails them regardless of
the state of the package. Letting them decide whether the pipeline is
green means red builds for commits that changed nothing related.
They therefore move into their own job with allow_failure: true, where a
failure is visible as what it is, a hint that the documentation may have
become unclear, without blocking anything. That job runs GLM-5.3-Flash,
which solved all three documentation tasks in every trial run, while
gemini-3.5-flash-lite still missed an expectation every few runs.
Keeping them out of the main job needs more than a testthat filter,
since devtools::check() runs the tests as well, and blanking the API
keys in the job would not work either, because project level variables
set in the GitLab UI take precedence over those in the yml. The model
list is used as the switch instead: RKORAP_LLM_MODELS unset means the
default models, set but empty means none at all, so the main job
generates no LLM tests in either of its two steps.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia1f29f05a0b8401f61307859e97948f4be977565
diff --git a/tests/testthat/test-readme-against-llm.R b/tests/testthat/test-readme-against-llm.R
index 8c35df4..886ccf6 100644
--- a/tests/testthat/test-readme-against-llm.R
+++ b/tests/testthat/test-readme-against-llm.R
@@ -10,12 +10,14 @@
)
llmModels <- function() {
- configured <- Sys.getenv("RKORAP_LLM_MODELS", unset = "")
- if (nzchar(configured)) {
- trimws(strsplit(configured, ",", fixed = TRUE)[[1]])
- } else {
- defaultLlmModels
+ configured <- Sys.getenv("RKORAP_LLM_MODELS", unset = NA_character_)
+ if (is.na(configured)) {
+ return(defaultLlmModels)
}
+ # set but empty means no models at all, which is how these tests are kept out
+ # of the pipeline job that runs everything else
+ models <- trimws(strsplit(configured, ",", fixed = TRUE)[[1]])
+ models[nzchar(models)]
}
# Provider and API key environment variable belonging to a model id