Keep tagger foundry when annotating corpora with custom inline foundries
External/Docker taggers (treetagger, spacy) and --annotate-with commands
route their output through parseAndWriteAnnotatedConllu, which adopted the
"# foundry =" comment found in the tool's CoNLL-U output. For corpora that
ship their own inline annotations in a custom foundry folder (e.g.
gingko/morpho.xml, with no base/tokens.xml), the CoNLL-U sent to the tool
carries "# foundry = gingko" and the tool echoes it back, so the annotation
was written under — and overwrote — the corpus's own foundry, and the output
ZIP was misnamed (mtz13.gingko.zip instead of mtz13.tree_tagger.zip).
The annotation task already carries the tool's foundry; let that win. The
echoed comment is only used as a fallback when the tool is unrecognized and
the foundry is the generic "annotated" placeholder. Integrated taggers
(marmot, opennlp, corenlp) were unaffected as they never went through this
path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I684d15ea7476e096d3fb48abcb1e23cd4bd64e7e
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3166c3e..86bf61d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@
- `tokenSource` in Krill JSON output no longer resolves to a stand-off annotation foundry ([#48](https://github.com/KorAP/korapxmltool/issues/48)): token source identification and token-map updates are now restricted to base archive processing, so stand-off annotation foundries can no longer overwrite the base tokenization spans or steal the `tokenSource`.
- Krill output now drops texts that contain no tokens instead of emitting empty, unindexable documents ([#46](https://github.com/KorAP/korapxmltool/issues/46)). Such texts (e.g. articles with empty `data.xml`/`base/tokens.xml`) are skipped with a per-text warning and a summary count; texts with at least one token are kept. New `m21_empty_sample.zip` regression fixture (one empty text, one single-token text).
- Corpora with custom tokenization and annotations inside the base ZIP (e.g. `cmc/morpho.xml` from TEI conversions, with no `base/tokens.xml`) are now handled correctly: the foundry is derived from the annotation folder name instead of the ZIP file name, so Krill output indexes the annotations (e.g. `cmc/p`, `cmc/l`) instead of silently dropping them, the token stream is no longer empty (`tokenSource` is set to e.g. `cmc#morpho`), and CoNLL-U output reports `# foundry = cmc` instead of `# foundry = base`
+- External/Docker taggers (e.g. `treetagger`, `spacy`) and `--annotate-with` commands no longer adopt the input corpus's own foundry when annotating corpora that ship custom inline annotations (e.g. `gingko/morpho.xml`). Previously the `# foundry =` comment echoed back through the tool overrode the tagger's foundry, so `korapxmltool -T treetagger -t zip mtz13.zip` wrote `mtz13.gingko.zip` and overwrote the corpus's `gingko` annotations instead of producing `mtz13.tree_tagger.zip`. The annotation tool's foundry now takes precedence; the echoed comment is only used as a fallback for unrecognized tools (generic `annotated` foundry).
### Changed
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
index 3cf08eb..69b655b 100644
--- a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
@@ -4851,18 +4851,25 @@
}
}
- // Use extracted foundry from CoNLL-U output if available
- val actualFoundry = if (foundryOverride != null) {
- foundryOverride!!
- } else if (extractedFoundry != null) {
- if (extractedFoundry != foundry) {
- LOGGER.info("Using foundry from CoNLL-U output: $extractedFoundry (was: $foundry)")
+ // Determine the foundry to write the annotation under. The task already
+ // carries the annotation tool's foundry (from -T / -A detection / -F) in
+ // `foundry`, and that must win: a "# foundry =" comment in the tool's
+ // output usually just echoes the INPUT corpus's own foundry back (e.g. a
+ // custom inline foundry like gingko or cmc), and adopting it would name
+ // the output after — and overwrite — the corpus's existing annotations.
+ // Only fall back to the comment when the tool is unrecognized and the
+ // foundry is the generic "annotated" placeholder.
+ val actualFoundry = when {
+ foundryOverride != null -> foundryOverride!!
+ foundry == "annotated" && extractedFoundry != null -> {
+ if (extractedFoundry != foundry) {
+ LOGGER.info("Using foundry from CoNLL-U output: $extractedFoundry (was: $foundry)")
+ }
+ // Update the global externalFoundry variable for consistent naming
+ externalFoundry = extractedFoundry
+ extractedFoundry
}
- // Update the global externalFoundry variable for consistent naming
- externalFoundry = extractedFoundry
- extractedFoundry
- } else {
- foundry
+ else -> foundry
}
try {
diff --git a/app/src/test/kotlin/de/ids_mannheim/korapxmltools/SparseAnnotationExternalTest.kt b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/SparseAnnotationExternalTest.kt
index e853ef2..f3defee 100644
--- a/app/src/test/kotlin/de/ids_mannheim/korapxmltools/SparseAnnotationExternalTest.kt
+++ b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/SparseAnnotationExternalTest.kt
@@ -81,6 +81,51 @@
}
}
+ @Test
+ fun taggerFoundryWinsOverEchoedInputFoundry() {
+ // Regression: when annotating a corpus that ships its own inline foundry
+ // (e.g. cmc/gingko), the tagger's foundry must name the output, not the
+ // "# foundry =" comment echoed back from the input through the tool.
+ val outputDir = createTempDir("conllu_foundry_precedence")
+ try {
+ val outputZip = File(outputDir, "output.zip")
+ val tool = KorapXmlTool()
+ tool.morphoZipOutputStream = ZipArchiveOutputStream(FileOutputStream(outputZip))
+ tool.tokenSeparator = "\n"
+
+ // Task foundry is the tagger's (tree_tagger); the CoNLL-U the tool
+ // produced still carries the input corpus foundry (cmc).
+ val task = AnnotationWorkerPool.AnnotationTask(
+ text = "",
+ docId = "NDY_115.005255",
+ entryPath = "NDY/115/005255|tree_tagger"
+ )
+ val annotatedConllu = """
+ # foundry = cmc
+ # filename = NDY/115/005255/base/tokens.xml
+ # text_id = NDY_115.005255
+ # start_offsets = 0 0 4 11 18 22 27 32 35 41 46 50 56 64
+ # end_offsets = 65 3 10 17 21 26 31 34 40 45 49 55 64 65
+ 7 :) _ _ EMOASC _ _ _ _ _
+
+ """.trimIndent()
+
+ tool.parseAndWriteAnnotatedConllu(annotatedConllu, task)
+ tool.morphoZipOutputStream?.close()
+
+ assertTrue(
+ extractFileFromZip(outputZip, Regex(".*tree_tagger/morpho.xml")) != null,
+ "Annotation must be written under the tagger foundry (tree_tagger)"
+ )
+ assertTrue(
+ extractFileFromZip(outputZip, Regex(".*/cmc/morpho.xml")) == null,
+ "Annotation must NOT be written under the echoed input foundry (cmc)"
+ )
+ } finally {
+ outputDir.deleteRecursively()
+ }
+ }
+
// Helper since kotlin-test doesn't strictly have createTempDir anymore in some versions or usually io.tmp
private fun createTempDir(prefix: String): File {
val f = java.nio.file.Files.createTempDirectory(prefix).toFile()