)]}'
{
  "commit": "d3e66d27ba1b381f00c3a4380d3eb815f9f9ea92",
  "tree": "f328348285a7da9ea10b4722904e52836d6f6e3f",
  "parents": [
    "67a18d916d51ffcfd101e09fa25fe32f8a2582a0"
  ],
  "author": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Tue Jun 16 15:08:49 2026 +0200"
  },
  "committer": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Tue Jun 16 15:08:49 2026 +0200"
  },
  "message": "Fix \"pushback value was too large\" crash on gender forms with non-ASCII slash\n\nThe SLASH macro matches four characters (/ ⁄ ∕ ／), but the gender-slash handlers\ngenderSlashSuffixToken() and genderNounSlashToken() located the separator with\nmatched.lastIndexOf(\u0027/\u0027), which only finds the ASCII solidus. For the non-ASCII\nvariants (U+2044, U+2215, U+FF0F) it returned -1, so yypushback(length - (-1))\nexceeded the match length and JFlex threw \"pushback value was too large\" — crashing\nthe tokenizer on real EPUB input (e.g. DNB25).\n\nAdd a lastIndexOfSlash() helper that matches all four SLASH characters and use it in\nboth handlers, so a non-ASCII slash now tokenizes identically to the ASCII one. Adds\na regression test (testGenderSensitiveSlashFormsWithUnicodeSlash); note tokenizePos\ncatches the Error and recovers lossily, so the test asserts the exact tokens rather\nthan merely \"does not throw\".\n\nCo-Authored-By: Claude Opus 4.8 \u003cnoreply@anthropic.com\u003e\nChange-Id: I0b3ab80813b002b9714fb040f4c292318b70f445\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "a193dcee9a4943eb2d2870f03bf6f9f6af21bfae",
      "old_mode": 33188,
      "old_path": "CHANGELOG.md",
      "new_id": "d767074cbadfe2da414c6e444827b39d327c5469",
      "new_mode": 33188,
      "new_path": "CHANGELOG.md"
    },
    {
      "type": "modify",
      "old_id": "58bbf282c04bbe0bcbaac84e37d2dfa95c91546e",
      "old_mode": 33188,
      "old_path": "src/main/jpc/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.jflex",
      "new_id": "afad9ee4eab8b811cd7f7dc43251bfce89f3976f",
      "new_mode": 33188,
      "new_path": "src/main/jpc/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.jflex"
    },
    {
      "type": "modify",
      "old_id": "8ad27c248de7cd21af0331f7fa39041f75fe7ba3",
      "old_mode": 33188,
      "old_path": "src/test/java/de/ids_mannheim/korap/tokenizer/TokenizerTest.java",
      "new_id": "1499f7650d2adf289a29364d0ea36e0a2feaa9d4",
      "new_mode": 33188,
      "new_path": "src/test/java/de/ids_mannheim/korap/tokenizer/TokenizerTest.java"
    }
  ]
}