tree b8cced9b969285d1db4dc91cbdd6e7ce594132fd
parent 800d9b7b9cc945dd7d4083bf775be1bbf0cbe49c
author Marc Kupietz <kupietz@ids-mannheim.de> 1781069133 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1781069133 +0200

Fix worker-pool streaming deadlock on large corpora

Annotate and flush per document instead of buffering 20000 sentences with
no flush. parse_spacy_pipe.py now treats korapxmltool's '# eot'/'# eof'
lines as document delimiters (rather than swallowing them as CoNLL-U
comments), emits the matching marker after each document, and flushes
stdout. This lets korapxmltool's AnnotationWorkerPool deliver each result
and release its bounded in-flight buffer slot.

Previously nothing was emitted until the process exited at stdin EOF, so
the in-flight buffer filled and the producer blocked in
bufferedTaskPermits.acquire while the parser blocked reading stdin for a
full chunk - a deadlock with idle CPU and zero progress that only appeared
on corpora with many small documents (small ones fit inside the buffer).

Inputs without protocol markers (a CoNLL-U file piped in directly) are
still read in SPACY_CHUNK_SIZE-bounded blocks and now stream their output.

Verified end-to-end with 'korapxmltool -P spacy' on zca24.zip: progress
streams 0->95 to completion, morpho/dependency output is correct, and no
markers leak into the annotations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I07d18c54e6e3238758b2c4a885ceabd428ec3f82