Fix "pushback value was too large" crash on gender forms with non-ASCII slash
The SLASH macro matches four characters (/ ⁄ ∕ ／), but the gender-slash handlers
genderSlashSuffixToken() and genderNounSlashToken() located the separator with
matched.lastIndexOf('/'), which only finds the ASCII solidus. For the non-ASCII
variants (U+2044, U+2215, U+FF0F) it returned -1, so yypushback(length - (-1))
exceeded the match length and JFlex threw "pushback value was too large" — crashing
the tokenizer on real EPUB input (e.g. DNB25).
Add a lastIndexOfSlash() helper that matches all four SLASH characters and use it in
both handlers, so a non-ASCII slash now tokenizes identically to the ASCII one. Adds
a regression test (testGenderSensitiveSlashFormsWithUnicodeSlash); note tokenizePos
catches the Error and recovers lossily, so the test asserts the exact tokens rather
than merely "does not throw".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I0b3ab80813b002b9714fb040f4c292318b70f445
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a193dce..d767074 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,12 @@
# Changelog
+## 2.4.2 [2026-06-16]
+
+* Fixed a crash (`pushback value was too large`) on German gender-sensitive forms
+ separated by a non-ASCII slash — division slash (U+2215 `∕`), fraction slash
+ (U+2044 `⁄`) or fullwidth solidus (U+FF0F `／`) — which could occur on EPUB input
+
## 2.4.1 [2026-04-03]
* Fixed Unicode surrogate pair handling bugs in German gender sensitive forms
diff --git a/src/main/jpc/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.jflex b/src/main/jpc/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.jflex
index 58bbf28..afad9ee 100644
--- a/src/main/jpc/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.jflex
+++ b/src/main/jpc/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.jflex
@@ -405,6 +405,23 @@
}
/**
+ * Find the last index of a SLASH-class separator. MUST stay in sync with the
+ * SLASH macro ([/⁄∕／]): matched.lastIndexOf('/') alone misses the non-ASCII
+ * variants (U+2044, U+2215, U+FF0F); for those it returns -1, which made
+ * yypushback(length - (-1)) overflow and throw "pushback value was too large"
+ * on gender-slash forms (e.g. "Lehrer∕innenx").
+ */
+ private static int lastIndexOfSlash(String s) {
+ for (int i = s.length() - 1; i >= 0; i--) {
+ char c = s.charAt(i);
+ if (c == '/' || c == '⁄' || c == '∕' || c == '／') {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ /**
* Handle gender short suffix with colon separator.
* Pattern: {WORD}:{suffix}{lookahead}
* If lookahead is a letter, return just WORD, pushing back the rest.
@@ -439,7 +456,7 @@
int lastChar = matched.codePointBefore(matched.length());
// Find the slash position
- int slashPos = matched.lastIndexOf('/');
+ int slashPos = lastIndexOfSlash(matched);
if (isLetter(lastChar)) {
// Followed by a letter - not a valid gender form
@@ -521,7 +538,7 @@
int lastChar = matched.codePointBefore(matched.length());
// Find the slash position
- int slashPos = matched.lastIndexOf('/');
+ int slashPos = lastIndexOfSlash(matched);
if (isLetter(lastChar)) {
// Followed by a letter - not a valid gender form
diff --git a/src/test/java/de/ids_mannheim/korap/tokenizer/TokenizerTest.java b/src/test/java/de/ids_mannheim/korap/tokenizer/TokenizerTest.java
index 8ad27c2..1499f76 100644
--- a/src/test/java/de/ids_mannheim/korap/tokenizer/TokenizerTest.java
+++ b/src/test/java/de/ids_mannheim/korap/tokenizer/TokenizerTest.java
@@ -926,6 +926,31 @@
}
@Test
+ public void testGenderSensitiveSlashFormsWithUnicodeSlash() {
+ // Regression: the SLASH macro matches four characters (/ ⁄ ∕ ／), but the
+ // gender-slash handlers located the separator with matched.lastIndexOf('/'),
+ // which returns -1 for the non-ASCII variants. That made yypushback compute
+ // length-(-1) and throw "pushback value was too large" (crash on real epub
+ // input, e.g. DNB25). See genderSlashSuffixToken/genderNounSlashToken.
+ DerekoDfaTokenizer_de tok = new DerekoDfaTokenizer_de();
+
+ // Valid Unicode-slash gender form (non-letter lookahead) is kept as one
+ // token, exactly like its ASCII counterpart:
+ String[] tokens = tok.tokenize("Lehrer∕innen kamen."); // U+2215 DIVISION SLASH
+ assertEquals("Lehrer∕innen", tokens[0]);
+ assertEquals("kamen", tokens[1]);
+ assertEquals(".", tokens[2]);
+ assertEquals(3, tokens.length);
+
+ // Crash cases (genderSlashSuffixToken): a non-ASCII slash + short gender
+ // suffix + letter lookahead must not overflow the pushback. The stem, the
+ // slash and the remainder are all emitted (no character is lost):
+ assertArrayEquals(new String[]{"Auto", "∕", "nn"}, tok.tokenize("Auto∕nn")); // U+2215
+ assertArrayEquals(new String[]{"Gerd", "⁄", "ss"}, tok.tokenize("Gerd⁄ss")); // U+2044
+ assertArrayEquals(new String[]{"Tag", "／", "rr"}, tok.tokenize("Tag／rr")); // U+FF0F
+ }
+
+ @Test
public void testGenderSensitiveParentheticalForms() {
DerekoDfaTokenizer_de tok = new DerekoDfaTokenizer_de();