)]}'
{
  "commit": "323a4d399c1b52943d8cd72d8a7c31fe2dcddc57",
  "tree": "72193c6a46d2ead305d9765f85019f6f137b6c07",
  "parents": [
    "37f96075e1fdcab348c358be3ce2820eb5ac0a50"
  ],
  "author": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Thu Sep 03 12:04:35 2026 +0200"
  },
  "committer": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Thu Sep 03 13:48:31 2026 +0200"
  },
  "message": "Doc prompting tests: run against several current low cost models\n\nThe Readme documentation tests ran against a single hardcoded model,\ngemini-2.5-pro. Since the tasks are simple, and since what is being\ntested is the Readme rather than the model, cheap and fast models are\nthe better fit: if such a model cannot follow the Readme, that is a\nfinding about the Readme. They are also considerably cheaper to run,\nin money and in energy.\n\nThe model is therefore no longer a constant but a list, run one test per\nmodel and task, and overridable via RKORAP_LLM_MODELS. The defaults are\ngemini-3.5-flash-lite, claude-sonnet-5 and GLM-5.3-Flash, the latter via\nthe OpenAI compatible Synthetic API at https://api.synthetic.new/openai/v1,\nauthorized with SYNTHETIC_API_KEY. tidyllm\u0027s openai provider takes no\ncustom base url, so that endpoint is queried directly with httr2, which\nis a hard dependency anyway.\n\nEach model is skipped individually if its API key is missing, so a\npartial set of keys tests whatever it can. Conditions of the account\nrather than of the documentation - rate limits, failed authorization\nand, newly, exhausted credits - lead to a skip instead of a failure.\n\nAlso fixes a problem that was independent of all this: the file called\nlibrary(tidyllm) at top level, although tidyllm is only suggested. Where\nit is not installed, as when checking with _R_CHECK_DEPENDS_ONLY_, that\nmade the whole file fail instead of skip. tidyllm is now qualified at\nits call sites and skipped on when missing.\n\nCo-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e\nChange-Id: I8be20b040f7549f39996af1d02afcefa69e63ce2\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "b77b93f658bf08dbd171c86188a7bec93434b386",
      "old_mode": 33188,
      "old_path": "tests/testthat/test-readme-against-llm.R",
      "new_id": "8c35df4cdfcb6618ff744e30a119cffd1e9be9fc",
      "new_mode": 33188,
      "new_path": "tests/testthat/test-readme-against-llm.R"
    }
  ]
}