tree 2f472382e2dbeb87241e9e6577edec0de530a4c1
parent 323a4d399c1b52943d8cd72d8a7c31fe2dcddc57
author Marc Kupietz <kupietz@ids-mannheim.de> 1788430707 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1788436152 +0200

CI: run the doc prompting tests in their own, non-blocking job

The Readme documentation prompting tests query LLMs, so they are neither
deterministic nor free: a model can miss an expectation on one run and
meet it on the next, and an empty API account fails them regardless of
the state of the package. Letting them decide whether the pipeline is
green means red builds for commits that changed nothing related.

They therefore move into their own job with allow_failure: true, where a
failure is visible as what it is, a hint that the documentation may have
become unclear, without blocking anything. That job runs GLM-5.3-Flash,
which solved all three documentation tasks in every trial run, while
gemini-3.5-flash-lite still missed an expectation every few runs.

Keeping them out of the main job needs more than a testthat filter,
since devtools::check() runs the tests as well, and blanking the API
keys in the job would not work either, because project level variables
set in the GitLab UI take precedence over those in the yml. The model
list is used as the switch instead: RKORAP_LLM_MODELS unset means the
default models, set but empty means none at all, so the main job
generates no LLM tests in either of its two steps.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Change-Id: Ia1f29f05a0b8401f61307859e97948f4be977565