tree f328348285a7da9ea10b4722904e52836d6f6e3f
parent 67a18d916d51ffcfd101e09fa25fe32f8a2582a0
author Marc Kupietz <kupietz@ids-mannheim.de> 1781615329 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1781615329 +0200

Fix "pushback value was too large" crash on gender forms with non-ASCII slash

The SLASH macro matches four characters (/ ⁄ ∕ ／), but the gender-slash handlers
genderSlashSuffixToken() and genderNounSlashToken() located the separator with
matched.lastIndexOf('/'), which only finds the ASCII solidus. For the non-ASCII
variants (U+2044, U+2215, U+FF0F) it returned -1, so yypushback(length - (-1))
exceeded the match length and JFlex threw "pushback value was too large" — crashing
the tokenizer on real EPUB input (e.g. DNB25).

Add a lastIndexOfSlash() helper that matches all four SLASH characters and use it in
both handlers, so a non-ASCII slash now tokenizes identically to the ASCII one. Adds
a regression test (testGenderSensitiveSlashFormsWithUnicodeSlash); note tokenizePos
catches the Error and recovers lossily, so the test asserts the exact tokens rather
than merely "does not throw".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I0b3ab80813b002b9714fb040f4c292318b70f445