Fix worker-pool crash on tokens with whitespace surface forms
A token whose FORM is whitespace (e.g. a lone space) collapsed the
tab-separated CoNLL-U line under str.split(), leaving 9 fields instead
of 10 and raising IndexError on the score column in CoNLLUP_Token. The
exception was uncaught and killed the whole streaming process, so
korapxmltool's worker pool saw a broken pipe, re-queued the in-flight
documents, and re-crashed on the same poisoned input until every worker
died -- observed as running threads dropping off one-by-one after tens
of thousands of texts.
CoNLLUP_Token now parses on the tab delimiter (preserving whitespace
FORM columns), falls back to whitespace splitting for space-delimited
input, and pads short lines to the 10 CoNLL-U columns. As
defense-in-depth, a parse/annotation failure for a single document is
now logged and skipped while still echoing its # eot/# eof marker, so
one bad document can no longer take down the run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I99b96a1ce21e23c1ebb3775afec4c3014d4e9253
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 828e25e..b963c52 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,23 @@
Version numbers follow the pattern: `<spaCy-version>-<release-number>`
+## [Unreleased]
+
+### Fixed
+- Worker-pool cascade crash on corpora containing a token whose surface form
+ (FORM) is whitespace (e.g. a lone space). Such a token collapsed the
+ tab-separated CoNLL-U line under `str.split()`, leaving 9 fields instead of
+ 10 and raising `IndexError` on the score column. The exception was uncaught
+ and killed the whole streaming process, so korapxmltool's worker pool saw a
+ broken pipe, re-queued the in-flight documents, and re-crashed on the same
+ poisoned input until every worker died — observed as running threads dropping
+ off one-by-one after tens of thousands of texts. `CoNLLUP_Token` now parses
+ on the tab delimiter (preserving whitespace FORM columns), falls back to
+ whitespace splitting for space-delimited input, and pads short lines to the
+ 10 CoNLL-U columns. As defense-in-depth, a parse/annotation failure for a
+ single document is now logged and skipped while still echoing its
+ `# eot`/`# eof` marker, so one bad document can no longer take down the run.
+
## [3.8.11-2] - 2026-06-10
### Fixed
diff --git a/lib/CoNLL_Annotation.py b/lib/CoNLL_Annotation.py
index 1a8ddf0..06ea564 100644
--- a/lib/CoNLL_Annotation.py
+++ b/lib/CoNLL_Annotation.py
@@ -73,7 +73,20 @@
class CoNLLUP_Token():
def __init__(self, raw_line, word_ix):
- info = raw_line.split()
+ # CoNLL-U is tab-separated with 10 columns. Split on the tab so that a
+ # token whose FORM is whitespace (e.g. a surface form that is a single
+ # space) keeps its column position instead of collapsing it -- a bare
+ # raw_line.split() would drop the empty FORM field, shift every column
+ # left, and raise IndexError on info[9], which crashes the whole
+ # streaming process and cascades into broken pipes across the worker
+ # pool. Fall back to a generic whitespace split for space-delimited
+ # inputs, then pad to 10 columns so a genuinely malformed/short line
+ # degrades gracefully rather than killing the stream.
+ info = raw_line.rstrip("\n").split("\t")
+ if len(info) < 10:
+ info = raw_line.split()
+ if len(info) < 10:
+ info = info + ["_"] * (10 - len(info))
# print(info)
# [ID, FORM, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC]
# [11, Prügel, Prügel, NN, NN, _, _, _, _, 1.000000]
diff --git a/systems/parse_spacy_pipe.py b/systems/parse_spacy_pipe.py
index 0432709..33f47de 100644
--- a/systems/parse_spacy_pipe.py
+++ b/systems/parse_spacy_pipe.py
@@ -367,10 +367,19 @@
# still processed in CHUNK_SIZE-bounded blocks and streamed out.
last_log_time = start
for block_lines, terminator in iter_documents(stdin, CHUNK_SIZE):
- annos, _ = read_conll(iter(block_lines), 0, token_class=token_class, comment_str=args.comment_str, our_foundry="spacy")
- if annos:
- dependency_warnings += annotate(annos, total_processed_sents)
- total_processed_sents += len(annos)
+ # Never let one malformed document take down the whole stream: a parse
+ # or annotation failure here would otherwise crash the process, break
+ # the pipe, and make korapxmltool's worker pool re-queue and re-crash on
+ # the same poisoned input until every worker dies. Log it, skip the
+ # document, and still echo the protocol marker below so the worker pool
+ # releases its in-flight slot.
+ try:
+ annos, _ = read_conll(iter(block_lines), 0, token_class=token_class, comment_str=args.comment_str, our_foundry="spacy")
+ if annos:
+ dependency_warnings += annotate(annos, total_processed_sents)
+ total_processed_sents += len(annos)
+ except Exception as doc_error:
+ logger.error(f"Skipping document after failure ({terminator or 'chunk'}): {str(doc_error)}")
# Echo the protocol marker back so korapxmltool can pair the output with
# the source document and release the in-flight buffer slot, then flush