Fix worker-pool crash on tokens with whitespace surface forms
A token whose FORM is whitespace (e.g. a lone space) collapsed the
tab-separated CoNLL-U line under str.split(), leaving 9 fields instead
of 10 and raising IndexError on the score column in CoNLLUP_Token. The
exception was uncaught and killed the whole streaming process, so
korapxmltool's worker pool saw a broken pipe, re-queued the in-flight
documents, and re-crashed on the same poisoned input until every worker
died -- observed as running threads dropping off one-by-one after tens
of thousands of texts.
CoNLLUP_Token now parses on the tab delimiter (preserving whitespace
FORM columns), falls back to whitespace splitting for space-delimited
input, and pads short lines to the 10 CoNLL-U columns. As
defense-in-depth, a parse/annotation failure for a single document is
now logged and skipped while still echoing its # eot/# eof marker, so
one bad document can no longer take down the run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I99b96a1ce21e23c1ebb3775afec4c3014d4e9253
diff --git a/lib/CoNLL_Annotation.py b/lib/CoNLL_Annotation.py
index 1a8ddf0..06ea564 100644
--- a/lib/CoNLL_Annotation.py
+++ b/lib/CoNLL_Annotation.py
@@ -73,7 +73,20 @@
class CoNLLUP_Token():
def __init__(self, raw_line, word_ix):
- info = raw_line.split()
+ # CoNLL-U is tab-separated with 10 columns. Split on the tab so that a
+ # token whose FORM is whitespace (e.g. a surface form that is a single
+ # space) keeps its column position instead of collapsing it -- a bare
+ # raw_line.split() would drop the empty FORM field, shift every column
+ # left, and raise IndexError on info[9], which crashes the whole
+ # streaming process and cascades into broken pipes across the worker
+ # pool. Fall back to a generic whitespace split for space-delimited
+ # inputs, then pad to 10 columns so a genuinely malformed/short line
+ # degrades gracefully rather than killing the stream.
+ info = raw_line.rstrip("\n").split("\t")
+ if len(info) < 10:
+ info = raw_line.split()
+ if len(info) < 10:
+ info = info + ["_"] * (10 - len(info))
# print(info)
# [ID, FORM, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC]
# [11, Prügel, Prügel, NN, NN, _, _, _, _, 1.000000]