tree 9d9e67c742cf7f5692a8980b9512488cafee7c38
parent b155ae276929b458d3cab4cf0a708e09144a8720
author Marc Kupietz <kupietz@ids-mannheim.de> 1781894194 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1781894194 +0200

Fix worker-pool crash on tokens with whitespace surface forms

A token whose FORM is whitespace (e.g. a lone space) collapsed the
tab-separated CoNLL-U line under str.split(), leaving 9 fields instead
of 10 and raising IndexError on the score column in CoNLLUP_Token. The
exception was uncaught and killed the whole streaming process, so
korapxmltool's worker pool saw a broken pipe, re-queued the in-flight
documents, and re-crashed on the same poisoned input until every worker
died -- observed as running threads dropping off one-by-one after tens
of thousands of texts.

CoNLLUP_Token now parses on the tab delimiter (preserving whitespace
FORM columns), falls back to whitespace splitting for space-delimited
input, and pads short lines to the 10 CoNLL-U columns. As
defense-in-depth, a parse/annotation failure for a single document is
now logged and skipped while still echoing its # eot/# eof marker, so
one bad document can no longer take down the run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I99b96a1ce21e23c1ebb3775afec4c3014d4e9253