tree 08db0c3ac0d5776125098bee14080bc92b202458
parent 849affd74ba15691ab4cd90ba46a44e172a1c85f
author Marc Kupietz <kupietz@ids-mannheim.de> 1786110171 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1786111641 +0200

Stabilize large word2vec exports

Bound streaming entry work globally across open ZIPs and release output claims when each ZIP closes, preventing corpus-sized scheduler state and per-ZIP backlog multiplication.

Separate ZIP-reader parallelism from entry-worker parallelism, avoid redundant annotation ZIP processing for surface output, and keep irrelevant late layers from restoring released document state.

Preserve caller ZIP order by default and add --largest-first as an explicit throughput-oriented option. Also make custom CoNLL-U foundry selection independent of concurrent base-entry completion order.

Change-Id: I1ca15de53bf2f500ee748d2478939b99a3ee77e4