)]}'
{
  "commit": "ffad7372df1ad6ffe4e806ae45fb2982e5f43942",
  "tree": "9d9e67c742cf7f5692a8980b9512488cafee7c38",
  "parents": [
    "b155ae276929b458d3cab4cf0a708e09144a8720"
  ],
  "author": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Fri Jun 19 20:36:34 2026 +0200"
  },
  "committer": {
    "name": "Marc Kupietz",
    "email": "kupietz@ids-mannheim.de",
    "time": "Fri Jun 19 20:36:34 2026 +0200"
  },
  "message": "Fix worker-pool crash on tokens with whitespace surface forms\n\nA token whose FORM is whitespace (e.g. a lone space) collapsed the\ntab-separated CoNLL-U line under str.split(), leaving 9 fields instead\nof 10 and raising IndexError on the score column in CoNLLUP_Token. The\nexception was uncaught and killed the whole streaming process, so\nkorapxmltool\u0027s worker pool saw a broken pipe, re-queued the in-flight\ndocuments, and re-crashed on the same poisoned input until every worker\ndied -- observed as running threads dropping off one-by-one after tens\nof thousands of texts.\n\nCoNLLUP_Token now parses on the tab delimiter (preserving whitespace\nFORM columns), falls back to whitespace splitting for space-delimited\ninput, and pads short lines to the 10 CoNLL-U columns. As\ndefense-in-depth, a parse/annotation failure for a single document is\nnow logged and skipped while still echoing its # eot/# eof marker, so\none bad document can no longer take down the run.\n\nCo-Authored-By: Claude Opus 4.8 \u003cnoreply@anthropic.com\u003e\nChange-Id: I99b96a1ce21e23c1ebb3775afec4c3014d4e9253\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "828e25ef49e3b62bfd14f0304923bbd0a311b499",
      "old_mode": 33188,
      "old_path": "CHANGELOG.md",
      "new_id": "b963c5218b9d472ef4604d90c30764c954541560",
      "new_mode": 33188,
      "new_path": "CHANGELOG.md"
    },
    {
      "type": "modify",
      "old_id": "1a8ddf0ae652082529c5a10126b2ebac58ffe976",
      "old_mode": 33188,
      "old_path": "lib/CoNLL_Annotation.py",
      "new_id": "06ea564874025b577f57af93955927f1258b6506",
      "new_mode": 33188,
      "new_path": "lib/CoNLL_Annotation.py"
    },
    {
      "type": "modify",
      "old_id": "043270991a9eafea730b5495442bc508b1493154",
      "old_mode": 33188,
      "old_path": "systems/parse_spacy_pipe.py",
      "new_id": "33f47de371c64dade8f5fcb83946b3a487f8f718",
      "new_mode": 33188,
      "new_path": "systems/parse_spacy_pipe.py"
    }
  ]
}