Fix worker-pool crash on tokens with whitespace surface forms
A token whose FORM is whitespace (e.g. a lone space) collapsed the
tab-separated CoNLL-U line under str.split(), leaving 9 fields instead
of 10 and raising IndexError on the score column in CoNLLUP_Token. The
exception was uncaught and killed the whole streaming process, so
korapxmltool's worker pool saw a broken pipe, re-queued the in-flight
documents, and re-crashed on the same poisoned input until every worker
died -- observed as running threads dropping off one-by-one after tens
of thousands of texts.
CoNLLUP_Token now parses on the tab delimiter (preserving whitespace
FORM columns), falls back to whitespace splitting for space-delimited
input, and pads short lines to the 10 CoNLL-U columns. As
defense-in-depth, a parse/annotation failure for a single document is
now logged and skipped while still echoing its # eot/# eof marker, so
one bad document can no longer take down the run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I99b96a1ce21e23c1ebb3775afec4c3014d4e9253
diff --git a/systems/parse_spacy_pipe.py b/systems/parse_spacy_pipe.py
index 0432709..33f47de 100644
--- a/systems/parse_spacy_pipe.py
+++ b/systems/parse_spacy_pipe.py
@@ -367,10 +367,19 @@
# still processed in CHUNK_SIZE-bounded blocks and streamed out.
last_log_time = start
for block_lines, terminator in iter_documents(stdin, CHUNK_SIZE):
- annos, _ = read_conll(iter(block_lines), 0, token_class=token_class, comment_str=args.comment_str, our_foundry="spacy")
- if annos:
- dependency_warnings += annotate(annos, total_processed_sents)
- total_processed_sents += len(annos)
+ # Never let one malformed document take down the whole stream: a parse
+ # or annotation failure here would otherwise crash the process, break
+ # the pipe, and make korapxmltool's worker pool re-queue and re-crash on
+ # the same poisoned input until every worker dies. Log it, skip the
+ # document, and still echo the protocol marker below so the worker pool
+ # releases its in-flight slot.
+ try:
+ annos, _ = read_conll(iter(block_lines), 0, token_class=token_class, comment_str=args.comment_str, our_foundry="spacy")
+ if annos:
+ dependency_warnings += annotate(annos, total_processed_sents)
+ total_processed_sents += len(annos)
+ except Exception as doc_error:
+ logger.error(f"Skipping document after failure ({terminator or 'chunk'}): {str(doc_error)}")
# Echo the protocol marker back so korapxmltool can pair the output with
# the source document and release the in-flight buffer slot, then flush