tree a1fa196b4683bd566b3bcbcaaf2a352803b85acc
parent 546325064a7f73992a70ff5674d2754beb80e65c
author Marc Kupietz <kupietz@ids-mannheim.de> 1781103817 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1781103817 +0200

Support corpora with custom tokenization and annotations

Corpora converted from TEI with their own tokenization and POS/lemma
annotations ship them inside the base ZIP in a custom foundry folder
(e.g. cmc/morpho.xml) and have no base/tokens.xml. Two fixes for this:

- Derive the foundry of annotation entries in base ZIPs from the folder
  name instead of the ZIP file name, so Krill output indexes the
  annotations (cmc/p, cmc/l, ...) instead of dropping them as "base",
  and CoNLL-U reports the correct "# foundry =" comment.
- Register token spans from morpho.xml in the Krill text data when no
  base/tokens.xml provides them, with tokenSource set to e.g.
  "cmc#morpho"; previously the Krill token stream was empty for such
  corpora.

Additionally, when structure.xml has no sentence (s) spans, sentence
segmentation now falls back to other segment-like TEI elements
(posting, l, seg, u, in this order), so integrated parsers and KorAP
sentence queries work on such corpora.

Tested with a new dck_sample.zip resource, a two-text excerpt from the
CC BY licensed Dortmunder Chat-Korpus; in one text the s spans are
removed to exercise the sentence fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Change-Id: Iacf97fb151edd8c8d2add244e71c3d32282531f1