Add Krill tar merge mode: update existing tars from ZIPs and stand-off metadata
Allow an existing Krill tar as input with -t krill ("merge mode"), updated
from the remaining inputs into a new tar (<name>.updated.krill.tar or -o).
This covers two scenarios: adding/fixing metadata when the source KorAP-XML
ZIPs are no longer available, and replacing single annotation foundries
(e.g. an improved wiki-taxonomy topic-domain classification) without a full
re-export.
- KrillTarMerger streams the input tar; unaffected texts are copied through
byte-identically without being decompressed, affected texts are patched
and recompressed in parallel with bounded memory. The input tar is never
modified.
- KrillJsonPatcher patches documents surgically at the lexical JSON level:
metadata fields replace same-key fields (aware of the legacy/corrected
name pairs textClass/dmozDomain, textDomain/idsColumn) or are appended;
a supplied annotation foundry replaces its stream annotations, sentence
spans/counts and layerInfos/foundries entries entirely. Token offsets are
taken from the existing stream, so the base ZIP is not needed.
- Annotation strings are built by the same code as full generation (helpers
extracted from KrillJsonGenerator, behavior-neutral), so patched texts are
byte-identical to a full re-export with the same inputs.
- creationDate/pubDate are only replaced when a new text-level header is
supplied, so corpus/doc-level headers cannot clobber per-text dates.
- Texts present in the new inputs but missing from the tar are ignored with
a warning; data/tokens/structure entries (base tokenization) are ignored
as well. Tars produced by korapxml2krill (Perl) and LZ4-compressed tars
are supported. The regular Krill export path is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Change-Id: I3850b877eb0fc2eb5bc122ed13a2f60029b03ecf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d455059..650ff91 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,11 @@
# Changelog
+## [Unreleased]
+
+### Added
+
+- Krill tars can now be used as *input* and updated in place of a full re-export ("merge mode"): listing an existing `.krill.tar` among the inputs with `-t krill` streams it to a new tar (`<name>.updated.krill.tar`, or `-o`), updated from the remaining inputs. Two use cases: (1) adding or fixing metadata (stand-off `<standOff>` files, `<xenoData>`, header ZIPs) when the original KorAP-XML ZIPs are no longer available, and (2) replacing or adding single annotation foundries (e.g. an improved wiki-taxonomy topic-domain classification) without touching the rest. Texts unaffected by the new inputs are copied through byte-identically without being decompressed; affected texts are patched surgically at the JSON level and recompressed in parallel, and end up identical to a full re-export with the same inputs. New metadata fields replace same-key fields (including legacy/corrected name pairs such as `textClass`/`dmozDomain`) and are appended otherwise; `creationDate`/`pubDate` are only replaced when a new text-level header is supplied, so corpus-level headers cannot clobber per-text dates. Texts present in the new inputs but missing from the tar are ignored with a warning; base tokenization/structure entries are ignored as well. Tars produced by korapxml2krill (Perl) are supported. The input tar is never modified, and the regular Krill export path is unchanged.
+
## [v4.1.0] - 2026-06-19
### Added
diff --git a/Readme.md b/Readme.md
index 2e43976..74ac4c1 100644
--- a/Readme.md
+++ b/Readme.md
@@ -193,6 +193,44 @@
| docker run --rm -i korap/wiki-taxonomy > rei_sample.wikiDomain.meta.xml
```
+### Updating an existing Krill tar (merge mode)
+
+An existing Krill tar can also be used as *input* and updated with new metadata
+and/or annotation foundries — useful when the original KorAP-XML ZIPs are no longer
+at hand, or when a single annotation foundry (e.g. an improved topic-domain
+classification) must be fixed without re-running the whole conversion. Merge mode is
+switched on simply by listing a `.tar` file among the inputs:
+
+```shell script
+# Update/add a stand-off classification in an existing Krill tar
+./build/bin/korapxmltool -t krill -D out/krill \
+ out/krill/rei_sample.krill.tar rei_sample.wikiDomain.meta.xml
+
+# Replace (or add) the TreeTagger annotations in an existing Krill tar
+./build/bin/korapxmltool -t krill -D out/krill \
+ out/krill/rei_sample.krill.tar app/src/test/resources/rei_sample.tree_tagger.zip
+```
+
+The result is written to a new tar (`<name>.updated.krill.tar`, or the path given
+with `-o`); the input tar is never modified, so an interrupted run cannot damage
+existing data. Properties of the merge:
+
+- Texts not affected by the new inputs are copied through byte-identically, without
+ even being decompressed; affected texts are patched and recompressed in parallel.
+- Every foundry contained in the supplied annotation ZIPs is treated as
+ authoritative: its existing annotations are replaced entirely; other foundries are
+ left untouched. The patched texts are identical to what a full re-export with the
+ same inputs would produce.
+- Metadata fields from stand-off files, `<xenoData>` and header ZIPs replace fields
+ with the same key and are appended otherwise. `creationDate`/`pubDate` are only
+ replaced when a new *text-level* header is supplied, so corpus-level headers cannot
+ clobber per-text dates.
+- Texts present in the new inputs but missing from the tar are ignored with a
+ warning. The base tokenization of the tar cannot be changed:
+ `data.xml`/`tokens.xml`/`structure.xml` entries are ignored as well.
+- Tars produced by korapxml2krill (Perl) are supported; untouched texts and
+ annotations keep their exact original form.
+
## Annotation
### Tagging with integrated MarMoT POS tagger directly to a new KorAP-XML ZIP file
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
index bfdd882..520ad94 100644
--- a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
@@ -4,6 +4,7 @@
import de.ids_mannheim.korapxmltools.AnnotationToolBridgeFactory.Companion.taggerFoundries
import de.ids_mannheim.korapxmltools.formatters.KorapXmlFormatter
import de.ids_mannheim.korapxmltools.formatters.KrillJsonGenerator
+import de.ids_mannheim.korapxmltools.formatters.KrillJsonPatcher
import org.apache.commons.compress.archivers.tar.TarArchiveEntry
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream
import org.apache.commons.compress.archivers.zip.Zip64Mode
@@ -121,7 +122,7 @@
private var targetZipFileName: String? = null
// Locale is now globally forced to ROOT at startup (see main())
- @Parameters(arity = "0..*", description = ["Input files: KorAP-XML ZIP files or CoNLL-U files (.conllu). If omitted, reads from stdin (requires -o for output path)."])
+ @Parameters(arity = "0..*", description = ["Input files: KorAP-XML ZIP files, CoNLL-U files (.conllu), stand-off metadata XML files, or (with -t krill) an existing Krill tar to update from the other inputs. If omitted, reads from stdin (requires -o for output path)."])
var zipFileNames: Array<String>? = null
@Option(
@@ -836,6 +837,26 @@
}
}
+ // An existing Krill tar among the inputs switches on merge mode: the tar is
+ // streamed to a new tar, updated from the remaining inputs (KorAP-XML ZIPs
+ // and/or stand-off metadata files).
+ zipFileNames?.filter { it.endsWith(".tar") }?.takeIf { it.isNotEmpty() }?.let { tarInputs ->
+ if (outputFormat != OutputFormat.KRILL) {
+ throw ParameterException(spec.commandLine(),
+ "Krill tar input is only supported with -f krill")
+ }
+ if (tarInputs.size > 1) {
+ throw ParameterException(spec.commandLine(),
+ "Only one Krill tar can be updated per invocation, got: ${tarInputs.joinToString(", ")}")
+ }
+ krillInputTarPath = tarInputs[0]
+ zipFileNames = zipFileNames!!.filterNot { it in tarInputs }.toTypedArray()
+ if (zipFileNames!!.isEmpty() && standoffMetadata.isEmpty()) {
+ throw ParameterException(spec.commandLine(),
+ "Updating a Krill tar requires at least one KorAP-XML ZIP or stand-off metadata file with new data")
+ }
+ }
+
// For krill format, redirect logging to file before any logging occurs
if (outputFormat == OutputFormat.KRILL) {
// Determine output path for Krill format
@@ -848,6 +869,10 @@
} else {
"$finalOutputPath.tar"
}
+ } else if (krillInputTarPath != null) {
+ // Merge mode: never overwrite the input tar by default
+ val baseName = File(krillInputTarPath!!).name.replace(Regex("(\\.krill)?\\.tar$"), "")
+ File(outputDir, "$baseName.updated.krill.tar").absolutePath
} else {
// Find the base ZIP (one without a foundry suffix)
val baseZip = zipFileNames!!.firstOrNull { zip ->
@@ -904,8 +929,13 @@
// not to the console, so it does not clutter stderr output.
logCallOptionsAndEnvironment()
+ // Krill tar merge mode: update an existing Krill tar from the remaining inputs
+ if (krillInputTarPath != null) {
+ return mergeKrillTar(krillInputTarPath!!, zipFileNames ?: emptyArray())
+ }
+
// CoNLL-U to KorAP XML ZIP conversion mode
- val isConlluInput = zipFileNames == null || zipFileNames!!.isEmpty() ||
+ val isConlluInput = zipFileNames == null || zipFileNames!!.isEmpty() ||
zipFileNames!!.any { it.endsWith(".conllu") }
if (isConlluInput) {
@@ -1245,6 +1275,8 @@
var krillTarOutputStream: TarArchiveOutputStream? = null
var krillOutputFileName: String? = null
private var krillOutputPath: String? = null
+ // Set when an existing Krill tar is given as input (merge mode)
+ private var krillInputTarPath: String? = null
private var textOutputWriter: BufferedWriter? = null
// Fast DocumentBuilderFactory without security features (safe for trusted input)
@@ -2151,44 +2183,7 @@
val headerEntries = allEntries.filter { it.name.contains("header.xml") }
headerEntries.forEach { headerEntry ->
- try {
- val headerBytes = foundryData.zipFile.getInputStream(headerEntry).readBytes()
- val headerDoc = safeDomFactory.newDocumentBuilder().parse(ByteArrayInputStream(headerBytes))
- val headerRoot = headerDoc.documentElement
- headerRoot.normalize()
-
- val entryPath = headerEntry.name
- val pathParts = entryPath.split('/').filter { it.isNotEmpty() && !it.endsWith("header.xml") }
-
- var textSigle = headerRoot.firstText("textSigle")
- var docSigle = headerRoot.firstText("dokumentSigle")
- var corpusSigle = headerRoot.firstText("korpusSigle")
-
- if (textSigle == null && docSigle == null && corpusSigle == null) {
- if (pathParts.size == 1) {
- corpusSigle = pathParts[0]
- } else if (pathParts.size == 2) {
- docSigle = "${pathParts[0]}/${pathParts[1]}"
- } else if (pathParts.size >= 3) {
- textSigle = getTextIdFromPath(entryPath)
- }
- }
-
- val docId = textSigle?.replace('/', '_')
-
- // Call appropriate metadata collection function based on what the header contains
- if (corpusSigle != null) {
- collectCorpusMetadata(corpusSigle, headerRoot)
- }
- if (docSigle != null) {
- collectDocMetadata(docSigle, headerRoot)
- }
- if (docId != null) {
- collectKrillMetadata(docId, headerRoot)
- }
- } catch (e: Exception) {
- LOGGER.warning("Error processing header ${headerEntry.name}: ${e.message}")
- }
+ processKrillHeaderEntry(foundryData.zipFile, headerEntry)
}
}
LOGGER.info("Completed header processing for metadata")
@@ -2225,6 +2220,303 @@
}
/**
+ * Parse a corpus-, document- or text-level header.xml entry and feed the
+ * corpus/doc/text metadata collectors. Shared between the interleaved Krill
+ * flow and Krill tar merge mode.
+ */
+ private fun processKrillHeaderEntry(zipFile: ApacheZipFile, headerEntry: ZipArchiveEntry) {
+ try {
+ val headerBytes = zipFile.getInputStream(headerEntry).readBytes()
+ val headerDoc = safeDomFactory.newDocumentBuilder().parse(ByteArrayInputStream(headerBytes))
+ val headerRoot = headerDoc.documentElement
+ headerRoot.normalize()
+
+ val entryPath = headerEntry.name
+ val pathParts = entryPath.split('/').filter { it.isNotEmpty() && !it.endsWith("header.xml") }
+
+ var textSigle = headerRoot.firstText("textSigle")
+ var docSigle = headerRoot.firstText("dokumentSigle")
+ var corpusSigle = headerRoot.firstText("korpusSigle")
+
+ if (textSigle == null && docSigle == null && corpusSigle == null) {
+ if (pathParts.size == 1) {
+ corpusSigle = pathParts[0]
+ } else if (pathParts.size == 2) {
+ docSigle = "${pathParts[0]}/${pathParts[1]}"
+ } else if (pathParts.size >= 3) {
+ textSigle = getTextIdFromPath(entryPath)
+ }
+ }
+
+ val docId = textSigle?.replace('/', '_')
+
+ // Call appropriate metadata collection function based on what the header contains
+ if (corpusSigle != null) {
+ collectCorpusMetadata(corpusSigle, headerRoot)
+ }
+ if (docSigle != null) {
+ collectDocMetadata(docSigle, headerRoot)
+ }
+ if (docId != null) {
+ collectKrillMetadata(docId, headerRoot)
+ }
+ } catch (e: Exception) {
+ LOGGER.warning("Error processing header ${headerEntry.name}: ${e.message}")
+ }
+ }
+
+ // ------------------------------------------------------------------
+ // Krill tar merge mode
+ // ------------------------------------------------------------------
+
+ /** Text id in the lossy form used for Krill tar entry names ("REI_RBR.00473" -> "REI-RBR-00473"). */
+ private fun normalizeKrillTextId(textId: String): String =
+ textId.replace("_", "-").replace(".", "-")
+
+ /**
+ * Update an existing Krill tar from KorAP-XML ZIPs (annotation foundries and/or
+ * headers) and stand-off metadata files, streaming it into a new tar.
+ *
+ * Texts present in the new inputs but absent from the tar are ignored with a
+ * warning. The base tokenization of the tar texts cannot be changed, so
+ * data.xml/tokens.xml/structure.xml entries are ignored as well.
+ */
+ private fun mergeKrillTar(inputTarPath: String, zips: Array<String>): Int {
+ val inputTar = File(inputTarPath)
+ val outputTar = File(krillOutputPath!!)
+ krillOutputFileName = outputTar.absolutePath
+ if (inputTar.canonicalFile == outputTar.canonicalFile) {
+ LOGGER.severe("Refusing to overwrite the input tar $inputTarPath in place; use -o to choose a different output file")
+ return 1
+ }
+ if (outputTar.exists() && !overwrite) {
+ LOGGER.severe("Output file ${outputTar.path} already exists. Use --force to overwrite.")
+ return 1
+ }
+ outputTar.parentFile?.mkdirs()
+ outputTexts.clear()
+ krillData.clear()
+
+ data class MergeZip(val path: String, val foundry: String, val zipFile: ApacheZipFile)
+
+ val mergeZips = mutableListOf<MergeZip>()
+ // Annotation-layer entries grouped by normalized text id: (zip, docId, entry)
+ val annotationWork = HashMap<String, MutableList<Triple<MergeZip, String, ZipArchiveEntry>>>()
+ var ignoredBaseContentEntries = 0
+ val annotationRegex = Regex(".*(morpho|dependency|sentences|constituency)\\.xml$")
+ val baseContentRegex = Regex(".*(data|tokens|structure)\\.xml$")
+
+ try {
+ zips.forEach { zipPath ->
+ val zipFoundry = getFoundryFromZipFileName(zipPath)
+ val zipFile = try {
+ openZipFile(zipPath)
+ } catch (e: Exception) {
+ LOGGER.severe("Failed to open ZIP $zipPath: ${e.message}")
+ return 1
+ }
+ val mergeZip = MergeZip(zipPath, zipFoundry, zipFile)
+ mergeZips.add(mergeZip)
+ zipFile.entries.asSequence().filter { !it.isDirectory }.forEach { entry ->
+ val name = entry.name
+ // Foundry directory of the entry ("<text>/<foundry>/<file>.xml")
+ val entryDir = name.substringBeforeLast('/', "").substringAfterLast('/')
+ when {
+ name.endsWith("header.xml") -> processKrillHeaderEntry(zipFile, entry)
+ name.matches(annotationRegex) && entryDir != "base" -> {
+ val docId = getTextIdFromPath(name)
+ annotationWork.getOrPut(normalizeKrillTextId(docId)) { mutableListOf() }
+ .add(Triple(mergeZip, docId, entry))
+ }
+ name.matches(baseContentRegex) || name.matches(annotationRegex) ->
+ ignoredBaseContentEntries++
+ }
+ }
+ }
+ if (ignoredBaseContentEntries > 0) {
+ LOGGER.warning(
+ "Merge mode cannot change the base tokenization or structure of texts in the tar: " +
+ "ignoring $ignoredBaseContentEntries data/tokens/structure/base entries"
+ )
+ }
+
+ // Texts with a text-level header in the new inputs (their date fields may be replaced)
+ val textsWithNewHeader = krillData.keys.toSet()
+ val corpusDocMetaProvided = corpusMetadata.isNotEmpty() || docMetadata.isNotEmpty() ||
+ corpusXenoData.isNotEmpty() || docXenoData.isNotEmpty()
+
+ // Normalized text id -> original id, for all texts with per-text metadata updates
+ val textMetaByNorm = HashMap<String, String>()
+ (textsWithNewHeader + standoffMetadata.keys + textXenoData.keys).forEach { docId ->
+ val norm = normalizeKrillTextId(docId)
+ val previous = textMetaByNorm.put(norm, docId)
+ if (previous != null && previous != docId) {
+ LOGGER.warning("Ambiguous text ids $previous and $docId both normalize to tar entry name $norm")
+ }
+ }
+
+ LOGGER.info(
+ "Updating Krill tar $inputTarPath -> ${outputTar.path}: " +
+ "${annotationWork.size} texts with new annotations, " +
+ "${textMetaByNorm.size} texts with new per-text metadata, " +
+ "corpus/doc-level metadata provided: $corpusDocMetaProvided"
+ )
+
+ val mergeProgressBar = if (!quiet) ProgressBarBuilder()
+ .setTaskName(outputTar.name)
+ .setInitialMax(inputTar.length())
+ .setStyle(ProgressBarStyle.COLORFUL_UNICODE_BAR)
+ .setUpdateIntervalMillis(500)
+ .setUnit(" MB", 1L shl 20)
+ .showSpeed()
+ .build() else null
+
+ val patcher = KrillTarMerger.TextPatcher { normId ->
+ val textMetaDocId = textMetaByNorm[normId]
+ val annotationEntries = annotationWork[normId]
+ if (!corpusDocMetaProvided && textMetaDocId == null && annotationEntries == null) {
+ null
+ } else {
+ { json ->
+ patchKrillText(json, normId, textMetaDocId, textsWithNewHeader,
+ annotationEntries?.map { (mz, docId, entry) ->
+ Triple(mz.zipFile to mz.path, mz.foundry to docId, entry)
+ })
+ }
+ }
+ }
+
+ val stats = try {
+ KrillTarMerger(LOGGER, maxThreads).merge(inputTar, outputTar, patcher) { bytesRead ->
+ mergeProgressBar?.stepTo(bytesRead)
+ }
+ } finally {
+ mergeProgressBar?.close()
+ }
+
+ // Texts in the new inputs that the tar does not contain are ignored with a warning
+ val unmatchedNorms = (textMetaByNorm.keys + annotationWork.keys) - stats.seenTextIds
+ if (unmatchedNorms.isNotEmpty()) {
+ val displayIds = unmatchedNorms.map { norm ->
+ textMetaByNorm[norm] ?: annotationWork[norm]?.firstOrNull()?.second ?: norm
+ }.sorted()
+ displayIds.take(20).forEach { docId ->
+ LOGGER.warning("Ignoring text $docId from the new input(s): not present in $inputTarPath")
+ }
+ if (displayIds.size > 20) {
+ LOGGER.warning("... and ${displayIds.size - 20} more texts not present in $inputTarPath")
+ }
+ LOGGER.warning("Ignored ${displayIds.size} text(s) from the new input(s) that are not in the input tar")
+ }
+
+ LOGGER.info(
+ "Krill tar update complete: ${stats.entries} entries " +
+ "(${stats.patched} updated, ${stats.copied} copied unchanged) -> ${outputTar.path}"
+ )
+ return 0
+ } finally {
+ mergeZips.forEach { mz ->
+ try {
+ mz.zipFile.close()
+ } catch (e: Exception) {
+ LOGGER.warning("Failed to close ZIP ${mz.path}: ${e.message}")
+ }
+ }
+ }
+ }
+
+ /**
+ * Patch a single text's Krill JSON: merge new metadata fields (from headers,
+ * stand-off metadata and xenodata) and replace/add annotation foundries.
+ * Runs on merge worker threads; per-text state is keyed by docId so
+ * concurrent texts don't interfere.
+ */
+ private fun patchKrillText(
+ json: String,
+ normId: String,
+ textMetaDocId: String?,
+ textsWithNewHeader: Set<String>,
+ annotationEntries: List<Triple<Pair<ApacheZipFile, String>, Pair<String, String>, ZipArchiveEntry>>?
+ ): String {
+ // Recover the real text id: the entry file name is lossy, the JSON's own
+ // textSigle ("REI/RBR/00473") is authoritative.
+ val sigle = KrillJsonPatcher.extractTextSigle(json)
+ val docId = when {
+ sigle != null && sigle.count { it == '/' } >= 2 ->
+ sigle.replaceFirst("/", "_").replaceFirst("/", ".")
+ sigle != null -> sigle.replace('/', '_')
+ else -> textMetaDocId ?: annotationEntries?.firstOrNull()?.second?.second
+ }
+ if (docId == null) {
+ LOGGER.warning("Cannot determine text id for tar entry $normId; leaving it unchanged")
+ return json
+ }
+
+ var result = json
+
+ // ---- metadata fields ----
+ val stub = KrillJsonGenerator.KrillTextData(textId = docId)
+ (krillData[docId] ?: textMetaDocId?.let { krillData[it] })
+ ?.headerMetadata?.let { stub.headerMetadata.putAll(it) }
+
+ val corpusSigle = docId.substringBefore('_')
+ val docSigle = "$corpusSigle/${docId.substringAfter('_').substringBefore('.')}"
+ val combinedStandoff = mutableListOf<KrillJsonGenerator.StandoffField>()
+ (standoffMetadata[docId] ?: textMetaDocId?.let { standoffMetadata[it] })
+ ?.let { combinedStandoff.addAll(it) }
+ corpusXenoData[corpusSigle]?.let { combinedStandoff.addAll(it) }
+ docXenoData[docSigle]?.let { combinedStandoff.addAll(it) }
+ (textXenoData[docId] ?: textMetaDocId?.let { textXenoData[it] })
+ ?.let { combinedStandoff.addAll(it) }
+ stub.standoffFields = combinedStandoff
+
+ val fieldEntries = KrillJsonGenerator.buildMetadataFieldEntries(
+ stub, corpusMetadata, docMetadata, legacyFieldNames
+ )
+ val hasNewTextHeader = docId in textsWithNewHeader ||
+ (textMetaDocId != null && textMetaDocId in textsWithNewHeader)
+ val fieldPatches = fieldEntries.mapNotNull { entry ->
+ when {
+ // Sigle fields are identity, never touched in a merge
+ entry.origin == KrillJsonGenerator.FieldOrigin.SIGLE -> null
+ // Dates inherited from corpus/doc headers must not clobber per-text
+ // dates in the tar; only a new text-level header may replace them.
+ entry.origin == KrillJsonGenerator.FieldOrigin.HEADER &&
+ (entry.key == "creationDate" || entry.key == "pubDate") && !hasNewTextHeader ->
+ KrillJsonPatcher.FieldPatch(entry.key, entry.json, KrillJsonPatcher.FieldMode.APPEND_IF_MISSING)
+ else ->
+ KrillJsonPatcher.FieldPatch(entry.key, entry.json, KrillJsonPatcher.FieldMode.REPLACE_OR_APPEND)
+ }
+ }
+ if (fieldPatches.isNotEmpty()) {
+ result = KrillJsonPatcher.patchFields(result, fieldPatches)
+ }
+
+ // ---- annotation foundries ----
+ if (!annotationEntries.isNullOrEmpty()) {
+ val entryDocId = annotationEntries.first().second.second
+ annotationEntries.forEach { (zipInfo, foundryInfo, entry) ->
+ processZipEntry(zipInfo.first, zipInfo.second, foundryInfo.first, entry, false)
+ }
+ val textData = krillData[entryDocId]
+ if (textData != null) {
+ result = KrillJsonPatcher.patchFoundries(result, textData)
+ } else {
+ LOGGER.warning("No annotation data collected for $entryDocId; foundries unchanged")
+ }
+ // Free per-text state
+ krillData.remove(entryDocId)
+ fnames.remove(entryDocId)
+ sentences.remove(entryDocId)
+ tokens.remove(entryDocId)
+ texts.remove(entryDocId)
+ morpho.remove(entryDocId)
+ }
+
+ return result
+ }
+
+ /**
* Submit a single text's entries for a specific foundry to the work-stealing queue.
*/
private fun submitTextForFoundry(zipFile: ApacheZipFile, zipPath: String, foundry: String, textId: String, textEntries: List<ZipArchiveEntry>) {
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KrillTarMerger.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KrillTarMerger.kt
new file mode 100644
index 0000000..a1e8134
--- /dev/null
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KrillTarMerger.kt
@@ -0,0 +1,170 @@
+package de.ids_mannheim.korapxmltools
+
+import org.apache.commons.compress.archivers.tar.TarArchiveEntry
+import org.apache.commons.compress.archivers.tar.TarArchiveInputStream
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream
+import java.io.BufferedInputStream
+import java.io.BufferedOutputStream
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.File
+import java.io.FileInputStream
+import java.io.FileOutputStream
+import java.nio.charset.StandardCharsets
+import java.util.concurrent.ArrayBlockingQueue
+import java.util.concurrent.ExecutorService
+import java.util.concurrent.Executors
+import java.util.concurrent.Future
+import java.util.concurrent.ThreadPoolExecutor
+import java.util.concurrent.TimeUnit
+import java.util.logging.Logger
+import java.util.zip.GZIPInputStream
+import java.util.zip.GZIPOutputStream
+
+/**
+ * Updates an existing Krill tar from new inputs by streaming it entry by entry
+ * ("merge mode", see KrillJsonPatcher).
+ *
+ * Entries that are not affected by the update are copied through without even
+ * being decompressed, so they stay byte-identical and cost only I/O. Affected
+ * entries are decompressed, patched and recompressed on a worker pool while the
+ * main thread keeps reading and writing, preserving the original entry order
+ * with bounded memory (at most a few texts in flight).
+ *
+ * The output is always a new tar file; the input tar is never modified, so an
+ * interrupted merge cannot damage existing data.
+ */
+class KrillTarMerger(
+ private val logger: Logger,
+ private val threads: Int
+) {
+ fun interface TextPatcher {
+ /**
+ * Return a JSON patch function for the tar entry with this normalized text id
+ * (the entry base name, e.g. "REI-RBR-00473"), or null when the text is not
+ * affected and its bytes should be copied through unchanged.
+ */
+ fun patcherFor(normalizedTextId: String): ((String) -> String)?
+ }
+
+ data class Stats(
+ var entries: Int = 0,
+ var patched: Int = 0,
+ var copied: Int = 0,
+ val seenTextIds: MutableSet<String> = mutableSetOf()
+ )
+
+ private class OutEntry(val name: String, val modTime: Long, val bytes: Future<ByteArray>)
+
+ /**
+ * Stream [inputTar] to [outputTar], patching affected texts via [patcher].
+ * [onProgress] is called after each written entry with the number of bytes
+ * read from the input tar so far.
+ */
+ fun merge(
+ inputTar: File,
+ outputTar: File,
+ patcher: TextPatcher,
+ onProgress: ((bytesRead: Long) -> Unit)? = null
+ ): Stats {
+ val stats = Stats()
+ val pool: ExecutorService = Executors.newFixedThreadPool(threads.coerceAtLeast(1)) { r ->
+ Thread(r, "KrillMergeWorker").apply { isDaemon = true }
+ }
+ // Bounded number of in-flight patch jobs keeps memory flat on huge tars.
+ val maxInFlight = (threads.coerceAtLeast(1)) * 2 + 1
+ val pending = ArrayDeque<OutEntry>()
+
+ try {
+ TarArchiveInputStream(BufferedInputStream(FileInputStream(inputTar), 1 shl 20)).use { tarIn ->
+ TarArchiveOutputStream(BufferedOutputStream(FileOutputStream(outputTar), 1 shl 20)).use { tarOut ->
+ tarOut.setLongFileMode(TarArchiveOutputStream.LONGFILE_POSIX)
+
+ fun drainOne() {
+ val out = pending.removeFirst()
+ val bytes = out.bytes.get()
+ val entry = TarArchiveEntry(out.name)
+ entry.size = bytes.size.toLong()
+ entry.setModTime(out.modTime)
+ tarOut.putArchiveEntry(entry)
+ tarOut.write(bytes)
+ tarOut.closeArchiveEntry()
+ onProgress?.invoke(tarIn.bytesRead)
+ }
+
+ var entry: TarArchiveEntry? = tarIn.nextEntry
+ while (entry != null) {
+ if (!entry.isFile) {
+ entry = tarIn.nextEntry
+ continue
+ }
+ stats.entries++
+ val name = entry.name
+ val baseName = name.substringAfterLast('/')
+ val compression = when {
+ baseName.endsWith(".json.gz") -> Compression.GZIP
+ baseName.endsWith(".json.lz4") -> Compression.LZ4
+ else -> null
+ }
+ val normalizedId = when (compression) {
+ Compression.GZIP -> baseName.removeSuffix(".json.gz")
+ Compression.LZ4 -> baseName.removeSuffix(".json.lz4")
+ null -> null
+ }
+ if (normalizedId != null) {
+ stats.seenTextIds.add(normalizedId)
+ }
+ val patchFn = normalizedId?.let { patcher.patcherFor(it) }
+ // The entry's bytes must be consumed before advancing to the next
+ // entry, so reading happens here; only patching is parallel.
+ val raw = tarIn.readAllBytes()
+ val modTime = entry.lastModifiedDate.time
+
+ if (patchFn == null || compression == null) {
+ stats.copied++
+ pending.addLast(OutEntry(name, modTime, java.util.concurrent.CompletableFuture.completedFuture(raw)))
+ } else {
+ stats.patched++
+ pending.addLast(OutEntry(name, modTime, pool.submit<ByteArray> {
+ recompress(patchFn(decompress(raw, compression)), compression)
+ }))
+ }
+ while (pending.size >= maxInFlight) drainOne()
+ entry = tarIn.nextEntry
+ }
+ while (pending.isNotEmpty()) drainOne()
+ tarOut.finish()
+ }
+ }
+ } finally {
+ pool.shutdownNow()
+ }
+ return stats
+ }
+
+ private enum class Compression { GZIP, LZ4 }
+
+ private fun decompress(bytes: ByteArray, compression: Compression): String {
+ val input = when (compression) {
+ Compression.GZIP -> GZIPInputStream(ByteArrayInputStream(bytes))
+ Compression.LZ4 -> net.jpountz.lz4.LZ4FrameInputStream(ByteArrayInputStream(bytes))
+ }
+ return input.use { it.readAllBytes().toString(StandardCharsets.UTF_8) }
+ }
+
+ private fun recompress(json: String, compression: Compression): ByteArray {
+ val byteOut = ByteArrayOutputStream(json.length / 2)
+ val out = when (compression) {
+ // Same gzip level as KorapXmlTool.compressKrillJson, so patched entries
+ // match what a fresh krill export would produce.
+ Compression.GZIP -> object : GZIPOutputStream(byteOut) {
+ init {
+ def.setLevel(1)
+ }
+ }
+ Compression.LZ4 -> net.jpountz.lz4.LZ4FrameOutputStream(byteOut)
+ }
+ out.use { it.write(json.toByteArray(StandardCharsets.UTF_8)) }
+ return byteOut.toByteArray()
+ }
+}
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt
index a7540c0..6568d0b 100644
--- a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt
@@ -3,6 +3,7 @@
import de.ids_mannheim.korapxmltools.KorapXmlTool.MorphoSpan
import de.ids_mannheim.korapxmltools.KorapXmlTool.Span
import de.ids_mannheim.korapxmltools.NonBmpString
+import java.util.SortedSet
import java.util.logging.Logger
/**
@@ -100,6 +101,281 @@
return sb.toString()
}
+ /** Short layer prefix used in the token stream and layerInfos, or null for "base". */
+ fun foundryPrefix(foundry: String): String? = when (foundry) {
+ "base" -> null
+ "tree_tagger" -> "tt"
+ "marmot-malt" -> "marmot"
+ else -> foundry
+ }
+
+ /** Full foundry name used in the "foundries" summary string. */
+ fun foundryFullNameForPrefix(prefix: String): String =
+ if (prefix == "tt") "treetagger" else prefix
+
+ /**
+ * Layer descriptors (e.g. "p=tokens", "d=rels") a foundry's morpho data
+ * contributes to layerInfos. Shared between [generateTo] and the tar merge path.
+ */
+ fun computeFoundryLayers(foundry: String, morphoData: Collection<MorphoSpan>?): SortedSet<String> {
+ val layers = sortedSetOf<String>()
+
+ // Check if this foundry has dependency annotations
+ val hasDependencies = morphoData?.any {
+ it.head != null && it.head != "_" && it.deprel != null && it.deprel != "_"
+ } ?: false
+
+ if (hasDependencies) {
+ layers.add("d=rels")
+ }
+
+ // Check if this foundry has lemma annotations
+ val hasLemma = morphoData?.any {
+ it.lemma != null && it.lemma != "_"
+ } ?: false
+ if (hasLemma) {
+ layers.add("l=tokens")
+ }
+
+ // Check if this foundry has POS annotations (xpos or upos)
+ val hasPos = morphoData?.any {
+ (it.xpos != null && it.xpos != "_") || (it.upos != null && it.upos != "_")
+ } ?: false
+ if (hasPos) {
+ layers.add("p=tokens")
+ }
+
+ // Check if this foundry has morphological features
+ val hasFeatures = morphoData?.any {
+ it.feats != null && it.feats != "_"
+ } ?: false
+ if (hasFeatures) {
+ layers.add("m=tokens")
+ }
+
+ // Check if this foundry has UPOS (skip for tree_tagger)
+ if (foundry != "tree_tagger") {
+ val hasUpos = morphoData?.any {
+ it.upos != null && it.upos != "_"
+ } ?: false
+ if (hasUpos) {
+ layers.add("u=tokens")
+ }
+ }
+
+ // Extra inline w-attribute layers (norm, orig, phon, trans) are
+ // rare; a single short-circuiting scan gates the four per-attribute
+ // checks so corpora without them pay at most one extra pass.
+ val hasExtra = morphoData?.any {
+ (it.norm != null && it.norm != "_") || (it.orig != null && it.orig != "_") ||
+ (it.phon != null && it.phon != "_") || (it.trans != null && it.trans != "_")
+ } ?: false
+ if (hasExtra) {
+ // hasExtra is only true when morphoData is non-null, so the
+ // compiler smart-casts it here (no safe call needed).
+ if (morphoData.any { it.norm != null && it.norm != "_" }) layers.add("norm=tokens")
+ if (morphoData.any { it.orig != null && it.orig != "_" }) layers.add("orig=tokens")
+ if (morphoData.any { it.phon != null && it.phon != "_" }) layers.add("phon=tokens")
+ if (morphoData.any { it.trans != null && it.trans != "_" }) layers.add("trans=tokens")
+ }
+ return layers
+ }
+
+ /** Resolve a CoNLL-U or offset-based head reference to a 0-based token index, or null. */
+ internal fun resolveHeadIndex(headStr: String, offsetToIndex: Map<String, Int>): Int? =
+ if (headStr.contains("-")) {
+ offsetToIndex[headStr]
+ } else {
+ val idx = headStr.toIntOrNull()
+ if (idx != null && idx > 0) idx - 1 else null
+ }
+
+ internal fun isRootHead(headStr: String): Boolean =
+ headStr == "0" || (headStr.contains("-") && headStr.startsWith("0-"))
+
+ /** Inverse dependency edge annotation ("<:foundry/d:rel$<b>32<i>dependent"). */
+ internal fun inverseDependencyAnnotation(prefix: String, deprel: String, dependentIndex: Int): String =
+ "<:$prefix/d:${deprel.escapeKrillValue()}\$<b>32<i>$dependentIndex"
+
+ /** Span annotation for a structural span whose token range is already resolved. */
+ internal fun structureSpanAnnotation(span: StructureSpan): String =
+ if (span.attributes.isEmpty()) {
+ "<>:${span.layer}\$<b>64<i>${span.from}<i>${span.to}<i>${span.tokenTo}<b>${span.depth}"
+ } else {
+ "<>:${span.layer}\$<b>64<i>${span.from}<i>${span.to}<i>${span.tokenTo}<b>${span.depth}<s>${span.depth}"
+ }
+
+ /**
+ * Resolve tokenFrom/tokenTo (exclusive) of a structural span against the token
+ * list, mirroring the resolution used during full generation.
+ */
+ fun resolveStructureSpanTokenRange(span: StructureSpan, tokens: List<Span>): StructureSpan {
+ if (span.tokenFrom >= 0 && span.tokenTo >= 0) {
+ return span
+ }
+ var tokenFrom = lowerBoundTokenFrom(tokens, span.from)
+ if (tokenFrom >= tokens.size || tokens[tokenFrom].from >= span.to) {
+ tokenFrom = -1
+ }
+
+ var lastTokenIndex = upperBoundTokenTo(tokens, span.to) - 1
+ if (lastTokenIndex < 0 || tokens[lastTokenIndex].to <= span.from) {
+ lastTokenIndex = -1
+ }
+
+ // Handle edge cases
+ if (tokenFrom == -1) tokenFrom = 0
+ if (lastTokenIndex == -1) lastTokenIndex = tokens.size - 1
+
+ // tokenTo is exclusive: one past the last token
+ return span.copy(tokenFrom = tokenFrom, tokenTo = lastTokenIndex + 1)
+ }
+
+ /**
+ * The raw (unquoted) stream annotations a single foundry contributes to one
+ * token: morphological features, POS, lemma, UPOS, extra w-attribute layers,
+ * and the outgoing/ROOT dependency edge. Shared between [forEachStreamItem]
+ * and the tar merge path so both emit identical strings.
+ */
+ internal fun morphoAnnotationsForToken(
+ prefix: String?,
+ foundry: String,
+ morphoSpan: MorphoSpan,
+ token: Span,
+ index: Int,
+ offsetToIndex: Map<String, Int>
+ ): List<String> {
+ val tokenAnnotations = mutableListOf<String>()
+
+ if (prefix != null) {
+ // Morphological features (sorted)
+ if (morphoSpan.feats != null && morphoSpan.feats != "_") {
+ val features = mutableListOf<String>()
+ morphoSpan.feats!!.split("|").forEach { feat ->
+ val parts = feat.split("=")
+ if (parts.size == 2) {
+ val key = parts[0].lowercase().escapeKrillValue()
+ val value = parts[1].lowercase().escapeKrillValue()
+ features.add("$prefix/m:$key:$value")
+ }
+ }
+ features.sorted().forEach { tokenAnnotations.add(it) }
+ }
+
+ // POS (xpos) with optional byte encoding - sorted by descending probability
+ if (morphoSpan.xpos != null && morphoSpan.xpos != "_") {
+ val xposList = morphoSpan.xpos!!.split("|")
+ val miscList = if (morphoSpan.misc != null && morphoSpan.misc != "_") {
+ morphoSpan.misc!!.split("|")
+ } else {
+ emptyList()
+ }
+
+ // Sort by descending probability if probabilities are available
+ val sortedPairs = if (miscList.size == xposList.size &&
+ miscList.all { it.toDoubleOrNull() != null }) {
+ xposList.mapIndexed { index, xpos ->
+ val certainty = miscList[index].toDoubleOrNull() ?: 0.0
+ Pair(xpos, certainty)
+ }.sortedByDescending { it.second }
+ } else {
+ // If probabilities don't match, keep original order
+ xposList.mapIndexed { index, xpos ->
+ val certainty = if (index < miscList.size) {
+ miscList[index].toDoubleOrNull()
+ } else {
+ null
+ }
+ Pair(xpos, certainty)
+ }
+ }
+
+ sortedPairs.forEach { (xpos, certainty) ->
+ if (certainty != null && sortedPairs.size > 1) {
+ val payload = kotlin.math.round(certainty * 255).toInt()
+ tokenAnnotations.add("$prefix/p:${xpos.escapeKrillValue()}\$<b>129<b>$payload")
+ } else {
+ tokenAnnotations.add("$prefix/p:${xpos.escapeKrillValue()}")
+ }
+ }
+ }
+
+ // Lemma - sorted by descending probability if probabilities are available
+ if (morphoSpan.lemma != null && morphoSpan.lemma != "_") {
+ val lemmaList = morphoSpan.lemma!!.split("|").distinct()
+ val miscList = if (morphoSpan.misc != null && morphoSpan.misc != "_") {
+ morphoSpan.misc!!.split("|")
+ } else {
+ emptyList()
+ }
+
+ // Extract probabilities from misc (exclude Offset= parts)
+ val probabilities = miscList.filter { !it.startsWith("Offset=") }
+ .mapNotNull { it.toDoubleOrNull() }
+
+ val sortedLemmas = if (probabilities.size == lemmaList.size) {
+ // Sort by descending probability
+ lemmaList.mapIndexed { index, lemma ->
+ val certainty = probabilities.getOrNull(index) ?: 0.0
+ Pair(lemma, certainty)
+ }.sortedByDescending { it.second }.map { it.first }
+ } else {
+ // If probabilities don't match, keep original order
+ lemmaList
+ }
+
+ sortedLemmas.forEach { lemma ->
+ tokenAnnotations.add("$prefix/l:${lemma.escapeKrillValue()}")
+ }
+ }
+
+ // UPOS (skip for tree_tagger as it only has xpos)
+ if (morphoSpan.upos != null && morphoSpan.upos != "_" && foundry != "tree_tagger") {
+ tokenAnnotations.add("$prefix/u:${morphoSpan.upos!!.escapeKrillValue()}")
+ }
+
+ // Extra inline w-attribute layers (norm, orig, phon, trans),
+ // each emitted under its own key like pos (p:) and lemma (l:).
+ if (morphoSpan.norm != null && morphoSpan.norm != "_") {
+ morphoSpan.norm!!.split("|").forEach {
+ tokenAnnotations.add("$prefix/norm:${it.escapeKrillValue()}")
+ }
+ }
+ if (morphoSpan.orig != null && morphoSpan.orig != "_") {
+ morphoSpan.orig!!.split("|").forEach {
+ tokenAnnotations.add("$prefix/orig:${it.escapeKrillValue()}")
+ }
+ }
+ if (morphoSpan.phon != null && morphoSpan.phon != "_") {
+ morphoSpan.phon!!.split("|").forEach {
+ tokenAnnotations.add("$prefix/phon:${it.escapeKrillValue()}")
+ }
+ }
+ if (morphoSpan.trans != null && morphoSpan.trans != "_") {
+ morphoSpan.trans!!.split("|").forEach {
+ tokenAnnotations.add("$prefix/trans:${it.escapeKrillValue()}")
+ }
+ }
+ }
+
+ // Dependency relations
+ if (morphoSpan.head != null && morphoSpan.head != "_" && morphoSpan.deprel != null && morphoSpan.deprel != "_") {
+ // Head can be either an offset (e.g., "100-110") or a token index (e.g., "1")
+ val headStr = morphoSpan.head!!
+ val resolvedHeadIndex = resolveHeadIndex(headStr, offsetToIndex)
+
+ if (resolvedHeadIndex != null) {
+ // Regular dependency - outgoing edge to head
+ tokenAnnotations.add(">:$prefix/d:${morphoSpan.deprel!!.escapeKrillValue()}\$<b>32<i>$resolvedHeadIndex")
+ } else if (isRootHead(headStr)) {
+ // ROOT node - use incoming edge format with full span info
+ tokenAnnotations.add("<:$prefix/d:${morphoSpan.deprel!!.escapeKrillValue()}\$<b>34<i>${token.from}<i>${token.to}<i>$index<i>1")
+ }
+ }
+
+ return tokenAnnotations
+ }
+
// Krill metadata field keys that were named misleadingly early on. By default we
// emit the corrected names; the original names map to their proper meaning:
// textClass -> dmozDomain (DMOZ-based topic-domain classification)
@@ -110,49 +386,49 @@
"textDomain" to "idsColumn"
)
- fun generateTo(
- out: Appendable,
+ /** Where a metadata field object originated. The Krill tar merge path uses this
+ * to apply different replacement policies per origin. */
+ enum class FieldOrigin { SIGLE, HEADER, STANDOFF }
+
+ /** A ready-to-emit metadata field: its emitted key and complete JSON object. */
+ data class FieldEntry(val key: String, val json: String, val origin: FieldOrigin)
+
+ /**
+ * Build the metadata field objects for a text, in emission order (sigles, header
+ * fields, stand-off fields). Shared between [generateTo] and the Krill tar merge
+ * path so both produce identical field JSON.
+ */
+ fun buildMetadataFieldEntries(
textData: KrillTextData,
- corpusMetadata: Map<String, MutableMap<String, Any>>,
- docMetadata: Map<String, MutableMap<String, Any>>,
- includeNonWordTokens: Boolean,
+ corpusMetadata: Map<String, Map<String, Any>>,
+ docMetadata: Map<String, Map<String, Any>>,
legacyFieldNames: Boolean = false
- ) {
- val sb = StringBuilder()
- sb.append("{")
-
- // @context, @type, and version
- sb.append("\"@context\":\"http://korap.ids-mannheim.de/ns/koral/0.4/context.jsonld\",")
- sb.append("\"@type\":\"koral:corpus\",")
- sb.append("\"version\":\"0.4\",")
-
- // fields (metadata)
- sb.append("\"fields\":[")
- val fields = mutableListOf<String>()
+ ): List<FieldEntry> {
+ val fields = mutableListOf<FieldEntry>()
// Extract corpus, doc, and text sigle from textId (e.g., "WUD24_I0083.95367")
// Convert underscores to slashes for proper format
val textIdWithSlashes = textData.textId.replace("_", "/").replace(".", "/")
val sigleParts = textIdWithSlashes.split("/")
if (sigleParts.size >= 3) {
- fields.add(jsonObject(listOf(
+ fields.add(FieldEntry("corpusSigle", jsonObject(listOf(
"value" to jsonString(sigleParts[0]),
"type" to jsonString("type:string"),
"@type" to jsonString("koral:field"),
"key" to jsonString("corpusSigle")
- )))
- fields.add(jsonObject(listOf(
+ )), FieldOrigin.SIGLE))
+ fields.add(FieldEntry("docSigle", jsonObject(listOf(
"@type" to jsonString("koral:field"),
"value" to jsonString("${sigleParts[0]}/${sigleParts[1]}"),
"type" to jsonString("type:string"),
"key" to jsonString("docSigle")
- )))
- fields.add(jsonObject(listOf(
+ )), FieldOrigin.SIGLE))
+ fields.add(FieldEntry("textSigle", jsonObject(listOf(
"@type" to jsonString("koral:field"),
"type" to jsonString("type:string"),
"value" to jsonString(textIdWithSlashes),
"key" to jsonString("textSigle")
- )))
+ )), FieldOrigin.SIGLE))
}
val resolvedHeaderMetadata = resolveHeaderMetadata(
@@ -256,12 +532,12 @@
}
val outKey = if (legacyFieldNames) key else CORRECTED_FIELD_NAMES[key] ?: key
- fields.add(jsonObject(listOf(
+ fields.add(FieldEntry(outKey, jsonObject(listOf(
"key" to jsonString(outKey),
"@type" to jsonString("koral:field"),
"value" to fieldValue,
"type" to jsonString(fieldType)
- )))
+ )), FieldOrigin.HEADER))
}
// Stand-off metadata fields (already typed/selected upstream). Skip any key
@@ -275,15 +551,37 @@
is List<*> -> jsonArray(v.map { jsonString(it.toString()) })
else -> jsonString(v.toString())
}
- fields.add(jsonObject(listOf(
+ fields.add(FieldEntry(field.key, jsonObject(listOf(
"key" to jsonString(field.key),
"@type" to jsonString("koral:field"),
"value" to fieldValue,
"type" to jsonString(field.type)
- )))
+ )), FieldOrigin.STANDOFF))
}
- sb.append(fields.joinToString(","))
+ return fields
+ }
+
+ fun generateTo(
+ out: Appendable,
+ textData: KrillTextData,
+ corpusMetadata: Map<String, MutableMap<String, Any>>,
+ docMetadata: Map<String, MutableMap<String, Any>>,
+ includeNonWordTokens: Boolean,
+ legacyFieldNames: Boolean = false
+ ) {
+ val sb = StringBuilder()
+ sb.append("{")
+
+ // @context, @type, and version
+ sb.append("\"@context\":\"http://korap.ids-mannheim.de/ns/koral/0.4/context.jsonld\",")
+ sb.append("\"@type\":\"koral:corpus\",")
+ sb.append("\"version\":\"0.4\",")
+
+ // fields (metadata)
+ sb.append("\"fields\":[")
+ sb.append(buildMetadataFieldEntries(textData, corpusMetadata, docMetadata, legacyFieldNames)
+ .joinToString(",") { it.json })
sb.append("],")
// data section
@@ -307,74 +605,10 @@
// Collect layers by foundry type (checking what data actually exists)
val foundryLayers = mutableMapOf<String, MutableSet<String>>()
textData.morphoByFoundry.keys.sorted().forEach { foundry ->
- val shortFoundry = when(foundry) {
- "base" -> null
- "tree_tagger" -> "tt"
- "marmot-malt" -> "marmot"
- else -> foundry
- }
+ val shortFoundry = foundryPrefix(foundry)
if (shortFoundry != null) {
- val layers = foundryLayers.getOrPut(shortFoundry) { mutableSetOf() }
- val morphoData = textData.morphoByFoundry[foundry]?.values
-
- // Check if this foundry has dependency annotations
- val hasDependencies = morphoData?.any {
- it.head != null && it.head != "_" && it.deprel != null && it.deprel != "_"
- } ?: false
-
- if (hasDependencies) {
- layers.add("d=rels")
- }
-
- // Check if this foundry has lemma annotations
- val hasLemma = morphoData?.any {
- it.lemma != null && it.lemma != "_"
- } ?: false
- if (hasLemma) {
- layers.add("l=tokens")
- }
-
- // Check if this foundry has POS annotations (xpos or upos)
- val hasPos = morphoData?.any {
- (it.xpos != null && it.xpos != "_") || (it.upos != null && it.upos != "_")
- } ?: false
- if (hasPos) {
- layers.add("p=tokens")
- }
-
- // Check if this foundry has morphological features
- val hasFeatures = morphoData?.any {
- it.feats != null && it.feats != "_"
- } ?: false
- if (hasFeatures) {
- layers.add("m=tokens")
- }
-
- // Check if this foundry has UPOS (skip for tree_tagger)
- if (foundry != "tree_tagger") {
- val hasUpos = morphoData?.any {
- it.upos != null && it.upos != "_"
- } ?: false
- if (hasUpos) {
- layers.add("u=tokens")
- }
- }
-
- // Extra inline w-attribute layers (norm, orig, phon, trans) are
- // rare; a single short-circuiting scan gates the four per-attribute
- // checks so corpora without them pay at most one extra pass.
- val hasExtra = morphoData?.any {
- (it.norm != null && it.norm != "_") || (it.orig != null && it.orig != "_") ||
- (it.phon != null && it.phon != "_") || (it.trans != null && it.trans != "_")
- } ?: false
- if (hasExtra) {
- // hasExtra is only true when morphoData is non-null, so the
- // compiler smart-casts it here (no safe call needed).
- if (morphoData.any { it.norm != null && it.norm != "_" }) layers.add("norm=tokens")
- if (morphoData.any { it.orig != null && it.orig != "_" }) layers.add("orig=tokens")
- if (morphoData.any { it.phon != null && it.phon != "_" }) layers.add("phon=tokens")
- if (morphoData.any { it.trans != null && it.trans != "_" }) layers.add("trans=tokens")
- }
+ foundryLayers.getOrPut(shortFoundry) { mutableSetOf() }
+ .addAll(computeFoundryLayers(foundry, textData.morphoByFoundry[foundry]?.values))
}
}
@@ -523,12 +757,7 @@
.map { (foundry, morphoSpans) ->
FoundryMorphoData(
foundry = foundry,
- prefix = when (foundry) {
- "tree_tagger" -> "tt"
- "marmot-malt" -> "marmot"
- "base" -> null
- else -> foundry
- },
+ prefix = foundryPrefix(foundry),
morphoSpans = morphoSpans
)
}
@@ -552,13 +781,8 @@
val prefix = foundryData.prefix ?: foundryData.foundry
// Check if this is a ROOT dependency (head == 0)
- if (!(headStr == "0" || (headStr.contains("-") && headStr.startsWith("0-")))) {
- val resolvedHeadIndex = if (headStr.contains("-")) {
- offsetToIndex[headStr]
- } else {
- val idx = headStr.toIntOrNull()
- if (idx != null && idx > 0) idx - 1 else null
- }
+ if (!isRootHead(headStr)) {
+ val resolvedHeadIndex = resolveHeadIndex(headStr, offsetToIndex)
if (resolvedHeadIndex != null) {
inverseDeps.getOrPut(resolvedHeadIndex) { mutableListOf() }
@@ -642,29 +866,7 @@
// Resolve tokenFrom and tokenTo for structural spans
// Note: tokenTo is exclusive (one past the last token index)
val resolvedStructureSpans = allStructureSpans.map { span ->
- if (span.tokenFrom >= 0 && span.tokenTo >= 0) {
- // Already resolved
- span
- } else {
- var tokenFrom = lowerBoundTokenFrom(tokens, span.from)
- if (tokenFrom >= tokens.size || tokens[tokenFrom].from >= span.to) {
- tokenFrom = -1
- }
-
- var lastTokenIndex = upperBoundTokenTo(tokens, span.to) - 1
- if (lastTokenIndex < 0 || tokens[lastTokenIndex].to <= span.from) {
- lastTokenIndex = -1
- }
-
- // Handle edge cases
- if (tokenFrom == -1) tokenFrom = 0
- if (lastTokenIndex == -1) lastTokenIndex = tokens.size - 1
-
- // tokenTo is exclusive: one past the last token
- val tokenTo = lastTokenIndex + 1
-
- span.copy(tokenFrom = tokenFrom, tokenTo = tokenTo)
- }
+ resolveStructureSpanTokenRange(span, tokens)
}
val resolvedSentenceFoundries = resolvedStructureSpans.foundriesWithSentenceSpans()
@@ -708,14 +910,7 @@
// Add all structural spans that start at token 0 or cover the whole document
val spansAtZero = spansByToken[0].orEmpty()
spansAtZero.forEach { span ->
- val spanAnnotation = if (span.attributes.isEmpty()) {
- "<>:${span.layer}\$<b>64<i>${span.from}<i>${span.to}<i>${span.tokenTo}<b>${span.depth}"
- } else {
- // Spans with attributes get a unique ID
- val attrId = span.depth
- "<>:${span.layer}\$<b>64<i>${span.from}<i>${span.to}<i>${span.tokenTo}<b>${span.depth}<s>$attrId"
- }
- tokenAnnotations.add(jsonString(spanAnnotation))
+ tokenAnnotations.add(jsonString(structureSpanAnnotation(span)))
// Add attribute annotations
span.attributes.forEach { (key, value) ->
@@ -730,12 +925,7 @@
} else {
// Add structural spans that start at this token
spansByToken[index]?.forEach { span ->
- val spanAnnotation = if (span.attributes.isEmpty()) {
- "<>:${span.layer}\$<b>64<i>${span.from}<i>${span.to}<i>${span.tokenTo}<b>${span.depth}"
- } else {
- "<>:${span.layer}\$<b>64<i>${span.from}<i>${span.to}<i>${span.tokenTo}<b>${span.depth}<s>${span.depth}"
- }
- tokenAnnotations.add(jsonString(spanAnnotation))
+ tokenAnnotations.add(jsonString(structureSpanAnnotation(span)))
span.attributes.forEach { (key, value) ->
val attrAnnotation = if (value.isEmpty()) {
@@ -771,148 +961,16 @@
// Add inverse dependency annotations (<:) for dependents pointing to this token as head
inverseDeps[index]?.sortedBy { "${it.foundry}/${it.deprel}" }?.forEach { inv ->
- tokenAnnotations.add(jsonString("<:${inv.foundry}/d:${inv.deprel.escapeKrillValue()}\$<b>32<i>${inv.dependentIndex}"))
+ tokenAnnotations.add(jsonString(inverseDependencyAnnotation(inv.foundry, inv.deprel, inv.dependentIndex)))
}
// Collect annotations from all foundries for this token
sortedFoundries.forEach { foundryData ->
- val foundry = foundryData.foundry
val morphoSpan = foundryData.morphoSpans[spanKey]
if (morphoSpan != null) {
- val prefix = foundryData.prefix
-
- if (prefix != null) {
- // Morphological features (sorted)
- if (morphoSpan.feats != null && morphoSpan.feats != "_") {
- val features = mutableListOf<String>()
- morphoSpan.feats!!.split("|").forEach { feat ->
- val parts = feat.split("=")
- if (parts.size == 2) {
- val key = parts[0].lowercase().escapeKrillValue()
- val value = parts[1].lowercase().escapeKrillValue()
- features.add("$prefix/m:$key:$value")
- }
- }
- features.sorted().forEach { tokenAnnotations.add(jsonString(it)) }
- }
-
- // POS (xpos) with optional byte encoding - sorted by descending probability
- if (morphoSpan.xpos != null && morphoSpan.xpos != "_") {
- val xposList = morphoSpan.xpos!!.split("|")
- val miscList = if (morphoSpan.misc != null && morphoSpan.misc != "_") {
- morphoSpan.misc!!.split("|")
- } else {
- emptyList()
- }
-
- // Sort by descending probability if probabilities are available
- val sortedPairs = if (miscList.size == xposList.size &&
- miscList.all { it.toDoubleOrNull() != null }) {
- xposList.mapIndexed { index, xpos ->
- val certainty = miscList[index].toDoubleOrNull() ?: 0.0
- Pair(xpos, certainty)
- }.sortedByDescending { it.second }
- } else {
- // If probabilities don't match, keep original order
- xposList.mapIndexed { index, xpos ->
- val certainty = if (index < miscList.size) {
- miscList[index].toDoubleOrNull()
- } else {
- null
- }
- Pair(xpos, certainty)
- }
- }
-
- sortedPairs.forEach { (xpos, certainty) ->
- if (certainty != null && sortedPairs.size > 1) {
- val payload = kotlin.math.round(certainty * 255).toInt()
- tokenAnnotations.add(jsonString("$prefix/p:${xpos.escapeKrillValue()}\$<b>129<b>$payload"))
- } else {
- tokenAnnotations.add(jsonString("$prefix/p:${xpos.escapeKrillValue()}"))
- }
- }
- }
-
- // Lemma - sorted by descending probability if probabilities are available
- if (morphoSpan.lemma != null && morphoSpan.lemma != "_") {
- val lemmaList = morphoSpan.lemma!!.split("|").distinct()
- val miscList = if (morphoSpan.misc != null && morphoSpan.misc != "_") {
- morphoSpan.misc!!.split("|")
- } else {
- emptyList()
- }
-
- // Extract probabilities from misc (exclude Offset= parts)
- val probabilities = miscList.filter { !it.startsWith("Offset=") }
- .mapNotNull { it.toDoubleOrNull() }
-
- val sortedLemmas = if (probabilities.size == lemmaList.size) {
- // Sort by descending probability
- lemmaList.mapIndexed { index, lemma ->
- val certainty = probabilities.getOrNull(index) ?: 0.0
- Pair(lemma, certainty)
- }.sortedByDescending { it.second }.map { it.first }
- } else {
- // If probabilities don't match, keep original order
- lemmaList
- }
-
- sortedLemmas.forEach { lemma ->
- tokenAnnotations.add(jsonString("$prefix/l:${lemma.escapeKrillValue()}"))
- }
- }
-
- // UPOS (skip for tree_tagger as it only has xpos)
- if (morphoSpan.upos != null && morphoSpan.upos != "_" && foundry != "tree_tagger") {
- tokenAnnotations.add(jsonString("$prefix/u:${morphoSpan.upos!!.escapeKrillValue()}"))
- }
-
- // Extra inline w-attribute layers (norm, orig, phon, trans),
- // each emitted under its own key like pos (p:) and lemma (l:).
- if (morphoSpan.norm != null && morphoSpan.norm != "_") {
- morphoSpan.norm!!.split("|").forEach {
- tokenAnnotations.add(jsonString("$prefix/norm:${it.escapeKrillValue()}"))
- }
- }
- if (morphoSpan.orig != null && morphoSpan.orig != "_") {
- morphoSpan.orig!!.split("|").forEach {
- tokenAnnotations.add(jsonString("$prefix/orig:${it.escapeKrillValue()}"))
- }
- }
- if (morphoSpan.phon != null && morphoSpan.phon != "_") {
- morphoSpan.phon!!.split("|").forEach {
- tokenAnnotations.add(jsonString("$prefix/phon:${it.escapeKrillValue()}"))
- }
- }
- if (morphoSpan.trans != null && morphoSpan.trans != "_") {
- morphoSpan.trans!!.split("|").forEach {
- tokenAnnotations.add(jsonString("$prefix/trans:${it.escapeKrillValue()}"))
- }
- }
- }
-
- // Dependency relations
- if (morphoSpan.head != null && morphoSpan.head != "_" && morphoSpan.deprel != null && morphoSpan.deprel != "_") {
- // Head can be either an offset (e.g., "100-110") or a token index (e.g., "1")
- val headStr = morphoSpan.head!!
- val resolvedHeadIndex = if (headStr.contains("-")) {
- // Offset format - resolve to token index
- offsetToIndex[headStr]
- } else {
- // Already a token index (1-based CoNLL-U format)
- val idx = headStr.toIntOrNull()
- if (idx != null && idx > 0) idx - 1 else null // Convert 1-based to 0-based
- }
-
- if (resolvedHeadIndex != null) {
- // Regular dependency - outgoing edge to head
- tokenAnnotations.add(jsonString(">:$prefix/d:${morphoSpan.deprel!!.escapeKrillValue()}\$<b>32<i>$resolvedHeadIndex"))
- } else if (headStr == "0" || (headStr.contains("-") && headStr.startsWith("0-"))) {
- // ROOT node - use incoming edge format with full span info
- tokenAnnotations.add(jsonString("<:$prefix/d:${morphoSpan.deprel!!.escapeKrillValue()}\$<b>34<i>${token.from}<i>${token.to}<i>$index<i>1"))
- }
- }
+ morphoAnnotationsForToken(
+ foundryData.prefix, foundryData.foundry, morphoSpan, token, index, offsetToIndex
+ ).forEach { tokenAnnotations.add(jsonString(it)) }
}
}
@@ -1045,6 +1103,9 @@
private fun jsonString(value: String): String = "\"${value.escapeJson()}\""
+ /** Quote and escape a stream annotation exactly like full generation does (for the merge path). */
+ internal fun quoteJson(value: String): String = jsonString(value)
+
private fun jsonArray(items: List<String>): String = items.joinToString(",", "[", "]")
private fun jsonObject(pairs: List<Pair<String, String>>): String {
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonPatcher.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonPatcher.kt
new file mode 100644
index 0000000..f437cf2
--- /dev/null
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonPatcher.kt
@@ -0,0 +1,626 @@
+package de.ids_mannheim.korapxmltools.formatters
+
+import de.ids_mannheim.korapxmltools.KorapXmlTool
+import java.util.SortedSet
+
+/**
+ * Surgical, lexical-level patching of existing Krill JSON documents, used when an
+ * existing Krill tar is given as input and should be updated from KorAP-XML ZIPs
+ * and/or stand-off metadata files ("merge mode").
+ *
+ * The patcher never re-serializes untouched parts of a document: it locates the
+ * regions to change with a small JSON scanner and splices replacement text in,
+ * so texts and annotations that are not affected by the update are preserved
+ * byte-for-byte. This keeps the merge robust for tars produced by older versions
+ * of this tool or by korapxml2krill (Perl).
+ */
+object KrillJsonPatcher {
+
+ /** How a new field is merged into the existing "fields" array. */
+ enum class FieldMode {
+ /** Replace the existing field with the same (or alias) key, else append. */
+ REPLACE_OR_APPEND,
+
+ /** Only append if no field with the same (or alias) key exists. */
+ APPEND_IF_MISSING
+ }
+
+ data class FieldPatch(val key: String, val json: String, val mode: FieldMode)
+
+ // Metadata keys that were renamed (see KrillJsonGenerator.CORRECTED_FIELD_NAMES).
+ // Replacing a field under one name must also remove its counterpart, so a tar
+ // written with legacy names never ends up carrying both variants.
+ private val FIELD_KEY_ALIASES = mapOf(
+ "textClass" to "dmozDomain", "dmozDomain" to "textClass",
+ "textDomain" to "idsColumn", "idsColumn" to "textDomain"
+ )
+
+ /**
+ * Merge [patches] into the top-level "fields" array of [json].
+ *
+ * Existing fields keep their position; a REPLACE_OR_APPEND patch replaces the
+ * first field whose key (or alias key) matches and removes later duplicates;
+ * patches without a match are appended at the end in the given order.
+ */
+ fun patchFields(json: String, patches: List<FieldPatch>): String {
+ if (patches.isEmpty()) return json
+ val fieldsRegion = findTopLevelMemberValue(json, "fields")
+ ?: throw IllegalArgumentException("No top-level \"fields\" array found in Krill JSON")
+ val elements = parseArrayElements(json, fieldsRegion)
+ val existingKeys = elements.map { extractMemberString(json, it, "key") }
+
+ // Resolve which existing element index each replacement patch targets.
+ val replacementByIndex = mutableMapOf<Int, FieldPatch>()
+ val dropIndices = mutableSetOf<Int>()
+ val toAppend = mutableListOf<FieldPatch>()
+ patches.forEach { patch ->
+ val targetKeys = setOf(patch.key, FIELD_KEY_ALIASES[patch.key] ?: patch.key)
+ val matches = existingKeys.withIndex().filter { (_, k) -> k != null && k in targetKeys }
+ when {
+ matches.isEmpty() -> toAppend.add(patch)
+
+ patch.mode == FieldMode.APPEND_IF_MISSING -> { /* exists: keep as is */ }
+
+ else -> {
+ replacementByIndex[matches.first().index] = patch
+ // Remove later duplicates and alias-named variants of the same field
+ matches.drop(1).forEach { dropIndices.add(it.index) }
+ }
+ }
+ }
+ if (replacementByIndex.isEmpty() && dropIndices.isEmpty() && toAppend.isEmpty()) return json
+
+ val newArray = StringBuilder()
+ var first = true
+ elements.forEachIndexed { idx, range ->
+ if (idx in dropIndices) return@forEachIndexed
+ if (!first) newArray.append(',')
+ newArray.append(replacementByIndex[idx]?.json ?: json.substring(range.first, range.last + 1))
+ first = false
+ }
+ toAppend.forEach { patch ->
+ if (!first) newArray.append(',')
+ newArray.append(patch.json)
+ first = false
+ }
+
+ return json.substring(0, fieldsRegion.first + 1) + newArray +
+ json.substring(fieldsRegion.last)
+ }
+
+ /**
+ * Extract the value of the top-level "textSigle" field from the "fields" array,
+ * e.g. "REI/RBR/00473", or null if absent. Used by merge mode to recover the
+ * text id of a tar entry independent of its (lossily normalized) file name.
+ */
+ fun extractTextSigle(json: String): String? {
+ val fieldsRegion = findTopLevelMemberValue(json, "fields") ?: return null
+ parseArrayElements(json, fieldsRegion).forEach { el ->
+ if (extractMemberString(json, el, "key") == "textSigle") {
+ return extractMemberString(json, el, "value")
+ }
+ }
+ return null
+ }
+
+ // ------------------------------------------------------------------
+ // Foundry (token stream) patching
+ // ------------------------------------------------------------------
+
+ /**
+ * Replace or add annotation foundries in an existing Krill JSON document.
+ *
+ * [textData] carries the new annotations collected from KorAP-XML ZIP entries
+ * (morpho/dependency/sentences/constituency) for this text. Every foundry it
+ * contains is treated as authoritative: all existing stream annotations of
+ * that foundry are removed and the new ones inserted at the positions full
+ * generation would put them; layerInfos and foundries summaries are updated.
+ * Token offsets come from the existing stream, so the original base ZIP is
+ * not needed. Annotations whose offsets match no stream token (tokenization
+ * drift, filtered non-word tokens) are silently skipped, like in full
+ * generation.
+ */
+ fun patchFoundries(json: String, textData: KrillJsonGenerator.KrillTextData): String {
+ val foundries = (textData.morphoByFoundry.keys +
+ textData.sentencesCollectedByFoundry +
+ textData.constituencyCollectedByFoundry)
+ .filterNot { it == "base" || it == "dereko" }
+ .toSortedSet()
+ if (foundries.isEmpty()) return json
+
+ val dataRange = findTopLevelMemberValue(json, "data")
+ ?: throw IllegalArgumentException("No top-level \"data\" object found in Krill JSON")
+ val streamRange = findMemberValue(json, "stream", dataRange.first)
+ ?: throw IllegalArgumentException("No \"stream\" array found in Krill JSON data")
+ val layerInfosRange = findMemberValue(json, "layerInfos", dataRange.first)
+ val foundriesRange = findMemberValue(json, "foundries", dataRange.first)
+
+ // Parse the stream into per-token lists of raw (escaped, unquoted) annotation strings
+ val tokenArrays: MutableList<MutableList<String>> = parseArrayElements(json, streamRange).map { tokRange ->
+ parseArrayElements(json, tokRange).map { el ->
+ require(json[el.first] == '"') { "Non-string stream annotation at offset ${el.first}" }
+ json.substring(el.first + 1, el.last)
+ }.toMutableList()
+ }.toMutableList()
+
+ // Token offsets from the existing stream ("_<i>$<i>from<i>to")
+ val offsetRegex = Regex("""^_\d+\$<i>(\d+)<i>(\d+)$""")
+ val tokens = tokenArrays.mapIndexed { idx, anns ->
+ val m = anns.firstNotNullOfOrNull { offsetRegex.matchEntire(it) }
+ ?: throw IllegalArgumentException("Stream token $idx has no offset annotation")
+ KorapXmlTool.Span(m.groupValues[1].toInt(), m.groupValues[2].toInt())
+ }
+ val offsetToIndex = HashMap<String, Int>(tokens.size * 2)
+ tokens.forEachIndexed { index, t -> offsetToIndex["${t.from}-${t.to}"] = index }
+
+ var layerInfoTokens = layerInfosRange?.let {
+ unescapeJsonString(json.substring(it.first + 1, it.last)).split(" ").filter { t -> t.isNotEmpty() }
+ } ?: emptyList()
+ var foundriesTokens = foundriesRange?.let {
+ unescapeJsonString(json.substring(it.first + 1, it.last)).split(" ").filter { t -> t.isNotEmpty() }
+ } ?: emptyList()
+
+ foundries.forEach { foundry ->
+ val contribution = buildContribution(foundry, textData, tokens, offsetToIndex)
+ removeFoundryAnnotations(tokenArrays, contribution.names)
+ insertFoundryAnnotations(tokenArrays, contribution)
+ layerInfoTokens = rebuildLayerInfos(layerInfoTokens, contribution)
+ foundriesTokens = rebuildFoundries(foundriesTokens, contribution)
+ }
+
+ // Re-serialize the three patched regions, splicing right-to-left so offsets stay valid
+ val replacements = mutableListOf<Pair<IntRange, String>>()
+ replacements.add(streamRange to tokenArrays.joinToString(",", "[", "]") { anns ->
+ anns.joinToString(",", "[", "]") { "\"$it\"" }
+ })
+ layerInfosRange?.let { replacements.add(it to KrillJsonGenerator.quoteJson(layerInfoTokens.joinToString(" "))) }
+ foundriesRange?.let { replacements.add(it to KrillJsonGenerator.quoteJson(foundriesTokens.joinToString(" "))) }
+ replacements.sortByDescending { it.first.first }
+
+ val sb = StringBuilder(json)
+ replacements.forEach { (range, text) ->
+ sb.replace(range.first, range.last + 1, text)
+ }
+ return sb.toString()
+ }
+
+ /** Everything one foundry contributes to a text's stream and summary strings. */
+ private class FoundryContribution(
+ val foundry: String,
+ val prefix: String,
+ val fullName: String,
+ /** All name variants whose existing annotations must be removed. */
+ val names: Set<String>,
+ val sentenceCount: Int,
+ /** Resolved structural spans (sentences/constituency), keyed by start token. */
+ val spansByToken: Map<Int, List<KrillJsonGenerator.StructureSpan>>,
+ val hasConstituency: Boolean,
+ /** Inverse dependency annotations, keyed by head token: (sortKey, raw annotation). */
+ val inverseByToken: Map<Int, List<Pair<String, String>>>,
+ /** Per-token morpho/dependency block (raw annotation strings). */
+ val morphoByToken: Map<Int, List<String>>,
+ /** layerInfos descriptors like "p=tokens" for the morpho prefix. */
+ val morphoLayers: SortedSet<String>
+ )
+
+ private fun buildContribution(
+ foundry: String,
+ textData: KrillJsonGenerator.KrillTextData,
+ tokens: List<KorapXmlTool.Span>,
+ offsetToIndex: Map<String, Int>
+ ): FoundryContribution {
+ val prefix = KrillJsonGenerator.foundryPrefix(foundry) ?: foundry
+ val fullName = KrillJsonGenerator.foundryFullNameForPrefix(prefix)
+ val morphoSpans = textData.morphoByFoundry[foundry]
+
+ // Structural spans of this foundry (sentences "f/s:s", constituency "f/c:X")
+ val spans = textData.structureSpans
+ .filter { it.layer.startsWith("$foundry/") }
+ .map { KrillJsonGenerator.resolveStructureSpanTokenRange(it, tokens) }
+ .filter { it.tokenFrom >= 0 }
+ val spansByToken = spans.groupBy { it.tokenFrom }
+ .mapValues { (_, list) ->
+ list.sortedWith(compareByDescending<KrillJsonGenerator.StructureSpan> { it.depth }.thenBy { it.layer })
+ }
+ val sentenceCount = spans.count { it.layer == "$foundry/s:s" }
+ val hasConstituency = spans.any { it.layer.substringAfter('/').startsWith("c:") }
+
+ val morphoByToken = HashMap<Int, List<String>>()
+ val inverseByToken = HashMap<Int, MutableList<Pair<String, String>>>()
+ if (morphoSpans != null) {
+ tokens.forEachIndexed { index, token ->
+ val spanKey = "${token.from}-${token.to}"
+ val morphoSpan = morphoSpans[spanKey] ?: return@forEachIndexed
+ val anns = KrillJsonGenerator.morphoAnnotationsForToken(
+ prefix, foundry, morphoSpan, token, index, offsetToIndex
+ )
+ if (anns.isNotEmpty()) {
+ morphoByToken[index] = anns
+ }
+ // Inverse dependency edges pointing at this token's head
+ val headStr = morphoSpan.head
+ val deprel = morphoSpan.deprel
+ if (headStr != null && headStr != "_" && deprel != null && deprel != "_" &&
+ !KrillJsonGenerator.isRootHead(headStr)
+ ) {
+ val headIndex = KrillJsonGenerator.resolveHeadIndex(headStr, offsetToIndex)
+ if (headIndex != null) {
+ inverseByToken.getOrPut(headIndex) { mutableListOf() }
+ .add("$prefix/$deprel" to KrillJsonGenerator.inverseDependencyAnnotation(prefix, deprel, index))
+ }
+ }
+ }
+ inverseByToken.values.forEach { list -> list.sortBy { it.first } }
+ }
+
+ return FoundryContribution(
+ foundry = foundry,
+ prefix = prefix,
+ fullName = fullName,
+ names = setOf(foundry, prefix, fullName),
+ sentenceCount = sentenceCount,
+ spansByToken = spansByToken,
+ hasConstituency = hasConstituency,
+ inverseByToken = inverseByToken,
+ morphoByToken = morphoByToken,
+ morphoLayers = KrillJsonGenerator.computeFoundryLayers(foundry, morphoSpans?.values)
+ )
+ }
+
+ // Position classes of stream annotations, in the order full generation emits them
+ private const val CLS_COUNTS = 0 // "-:..." (token 0 only)
+ private const val CLS_SPANS = 1 // "<>:..." and their "@:..." attributes
+ private const val CLS_OFFSET = 2 // "_<i>$<i>f<i>t"
+ private const val CLS_LOWER = 3 // "i:..."
+ private const val CLS_NONWORD = 4 // "base/p:_"
+ private const val CLS_INVDEP = 5 // "<:X/d:...$<b>32<i>n"
+ private const val CLS_MORPHO = 6 // foundry blocks: "X/...", ">:X/...", root "<:...<b>34..."
+ private const val CLS_SURFACE = 7 // "s:..."
+
+ private fun classify(raw: String): Int = when {
+ raw.startsWith("-:") -> CLS_COUNTS
+ raw.startsWith("<>:") || raw.startsWith("@:") -> CLS_SPANS
+ raw.startsWith("_") -> CLS_OFFSET
+ raw.startsWith("i:") -> CLS_LOWER
+ raw == "base/p:_" -> CLS_NONWORD
+ raw.startsWith("<:") && raw.contains("\$<b>32") -> CLS_INVDEP
+ raw.startsWith("s:") -> CLS_SURFACE
+ else -> CLS_MORPHO
+ }
+
+ /** Foundry name variant an annotation belongs to, or null (base/structural/bookkeeping). */
+ private fun annotationOwner(raw: String): String? {
+ val body = when {
+ raw.startsWith("<>:") -> raw.substring(3)
+ raw.startsWith(">:") || raw.startsWith("-:") || raw.startsWith("@:") -> raw.substring(2)
+ raw.startsWith("<:") -> raw.substring(2)
+ else -> raw
+ }
+ val slash = body.indexOf('/')
+ if (slash <= 0) return null
+ return body.substring(0, slash)
+ }
+
+ private fun removeFoundryAnnotations(tokenArrays: MutableList<MutableList<String>>, names: Set<String>) {
+ tokenArrays.forEach { anns ->
+ anns.removeAll { raw ->
+ classify(raw) != CLS_NONWORD && annotationOwner(raw) in names
+ }
+ }
+ }
+
+ private fun insertFoundryAnnotations(
+ tokenArrays: MutableList<MutableList<String>>,
+ contribution: FoundryContribution
+ ) {
+ tokenArrays.forEachIndexed { index, anns ->
+ // Sentence count at token 0, among "-:X/sentences" sorted by foundry, before "-:tokens"
+ if (index == 0 && contribution.sentenceCount > 0) {
+ val countAnn = "-:${contribution.foundry}/sentences\$<i>${contribution.sentenceCount}"
+ var pos = 0
+ while (pos < anns.size && classify(anns[pos]) == CLS_COUNTS) {
+ val raw = anns[pos]
+ val owner = annotationOwner(raw)
+ if (raw.startsWith("-:tokens") || (owner != null && owner != "base" && owner > contribution.foundry)) break
+ pos++
+ }
+ anns.add(pos, escapeAnnotation(countAnn))
+ }
+
+ // Structural spans among "<>:" entries, ordered by (depth desc, layer)
+ contribution.spansByToken[index]?.forEach { span ->
+ val ann = KrillJsonGenerator.structureSpanAnnotation(span)
+ var pos = anns.indexOfFirst { classify(it) == CLS_SPANS }
+ if (pos < 0) {
+ // No spans at this token yet: spans go right before the offset annotation
+ pos = anns.indexOfFirst { classify(it) == CLS_OFFSET }
+ if (pos < 0) pos = anns.size
+ } else {
+ while (pos < anns.size && classify(anns[pos]) == CLS_SPANS) {
+ val existing = parseSpanOrder(anns[pos])
+ if (existing != null &&
+ (existing.first < span.depth ||
+ (existing.first == span.depth && existing.second > span.layer))
+ ) break
+ pos++
+ }
+ }
+ anns.add(pos, escapeAnnotation(ann))
+ }
+
+ // Inverse dependency edges, sorted by "prefix/deprel" among existing <b>32 edges
+ contribution.inverseByToken[index]?.forEach { (sortKey, ann) ->
+ var pos = anns.indexOfFirst { classify(it) == CLS_INVDEP }
+ if (pos < 0) {
+ // Zone is empty: it sits after i:/base/p:_ and before the morpho blocks
+ pos = anns.indexOfFirst { classify(it) == CLS_MORPHO || classify(it) == CLS_SURFACE }
+ if (pos < 0) pos = anns.size
+ } else {
+ while (pos < anns.size && classify(anns[pos]) == CLS_INVDEP) {
+ val existingKey = parseInverseDepKey(anns[pos])
+ if (existingKey != null && existingKey > sortKey) break
+ pos++
+ }
+ }
+ anns.add(pos, escapeAnnotation(ann))
+ }
+
+ // The foundry's morpho/dependency block, between blocks sorted by foundry name
+ contribution.morphoByToken[index]?.let { block ->
+ var pos = -1
+ var i = 0
+ while (i < anns.size) {
+ val cls = classify(anns[i])
+ if (cls == CLS_SURFACE) {
+ if (pos < 0) pos = i
+ break
+ }
+ if (cls == CLS_MORPHO) {
+ val owner = annotationOwner(anns[i])
+ val ownerKey = owner?.let { foundrySortKey(it) }
+ if (ownerKey != null && ownerKey > contribution.foundry) {
+ pos = i
+ break
+ }
+ }
+ i++
+ }
+ if (pos < 0) pos = anns.size
+ anns.addAll(pos, block.map { escapeAnnotation(it) })
+ }
+ }
+ }
+
+ /** Sort key of a stream-annotation owner: map layer prefixes back to foundry names. */
+ private fun foundrySortKey(owner: String): String = when (owner) {
+ "tt", "treetagger" -> "tree_tagger"
+ else -> owner
+ }
+
+ /** (depth, layer) of an existing "<>:" span annotation, for ordered insertion. */
+ private fun parseSpanOrder(raw: String): Pair<Int, String>? {
+ val m = Regex("""^<>:([^$]+)\$<b>64(?:<i>-?\d+){3}<b>(\d+)""").find(raw) ?: return null
+ return m.groupValues[2].toInt() to m.groupValues[1]
+ }
+
+ /** "foundry/deprel" sort key of an existing inverse dependency annotation. */
+ private fun parseInverseDepKey(raw: String): String? {
+ val m = Regex("""^<:([^/]+)/d:([^$]*)\$""").find(raw) ?: return null
+ return "${m.groupValues[1]}/${m.groupValues[2]}"
+ }
+
+ /** JSON-escape a raw annotation exactly like the generator (content without quotes). */
+ private fun escapeAnnotation(ann: String): String {
+ val quoted = KrillJsonGenerator.quoteJson(ann)
+ return quoted.substring(1, quoted.length - 1)
+ }
+
+ private fun rebuildLayerInfos(existing: List<String>, c: FoundryContribution): List<String> {
+ val kept = existing.filterNot { it.substringBefore('/') in c.names }
+ val added = mutableListOf<String>()
+ if (c.sentenceCount > 0) added.add("${c.foundry}/s=spans")
+ if (c.hasConstituency) added.add("${c.foundry}/c=spans")
+ c.morphoLayers.forEach { added.add("${c.prefix}/$it") }
+ return (kept + added).sortedWith(compareBy({ layerInfoRank(it) }, { it.substringBefore('/') }, { it.substringAfter('/') }))
+ }
+
+ private fun layerInfoRank(token: String): Int = when {
+ token == "dereko/s=spans" -> 0
+ token == "base/p=tokens" -> 1
+ token.endsWith("/s=spans") -> 2
+ token.endsWith("/c=spans") -> 3
+ else -> 4
+ }
+
+ private fun rebuildFoundries(existing: List<String>, c: FoundryContribution): List<String> {
+ val kept = existing.filterNot { it.substringBefore('/') in c.names }
+
+ // Reconstruct the sets the generator derives its ordering from
+ val sentFoundries = kept.filter { it.endsWith("/sentences") }.map { it.substringBefore('/') }.toSortedSet()
+ val constitFoundries = kept.filter { it.endsWith("/structure") && it.substringBefore('/') != "dereko" }
+ .map { it.substringBefore('/') }.toSortedSet()
+ val annLayers = sortedMapOf<String, SortedSet<String>>()
+ kept.forEach {
+ val name = it.substringBefore('/')
+ val layer = it.substringAfter('/', "")
+ if (layer == "morpho" || layer == "dependency") {
+ annLayers.getOrPut(name) { sortedSetOf() }.add(layer)
+ }
+ }
+
+ if (c.sentenceCount > 0) sentFoundries.add(c.foundry)
+ if (c.hasConstituency) constitFoundries.add(c.foundry)
+ if (c.morphoLayers.isNotEmpty()) {
+ val layers = annLayers.getOrPut(c.fullName) { sortedSetOf() }
+ layers.clear()
+ if (c.morphoLayers.any { it == "d=rels" }) layers.add("dependency")
+ if (c.morphoLayers.any { it != "d=rels" }) layers.add("morpho")
+ }
+
+ val result = mutableListOf<String>()
+ // dereko block stays as-is (order is fixed)
+ kept.filter { it == "dereko" || it.startsWith("dereko/") }.forEach { result.add(it) }
+ // structure-advertised foundries: bare name, X/sentences, X/structure
+ (sentFoundries + constitFoundries).toSortedSet().forEach { f ->
+ if (!result.contains(f)) result.add(f)
+ if (f in sentFoundries) {
+ val e = "$f/sentences"
+ if (!result.contains(e)) result.add(e)
+ }
+ if (f in constitFoundries) {
+ val e = "$f/structure"
+ if (!result.contains(e)) result.add(e)
+ }
+ }
+ // annotation foundries, sorted like the generator (by layer prefix)
+ annLayers.keys.sortedBy { name -> if (name == "treetagger") "tt" else name }.forEach { name ->
+ result.add(name)
+ annLayers[name]!!.forEach { layer ->
+ val e = "$name/$layer"
+ if (!result.contains(e)) result.add(e)
+ }
+ }
+ return result
+ }
+
+ // ------------------------------------------------------------------
+ // Minimal JSON scanning utilities.
+ //
+ // These operate directly on the document string and return index ranges
+ // (inclusive first, inclusive last) so callers can splice text without
+ // re-serializing what they don't touch.
+ // ------------------------------------------------------------------
+
+ /** Index just past a JSON string that starts at [start] (which must be '"'). */
+ private fun skipString(s: String, start: Int): Int {
+ var i = start + 1
+ while (i < s.length) {
+ when (s[i]) {
+ '\\' -> i += 2
+ '"' -> return i + 1
+ else -> i++
+ }
+ }
+ throw IllegalArgumentException("Unterminated JSON string at offset $start")
+ }
+
+ /** Index just past the JSON value starting at [start] (skips leading whitespace). */
+ private fun skipValue(s: String, start: Int): Int {
+ var i = skipWhitespace(s, start)
+ return when (s[i]) {
+ '"' -> skipString(s, i)
+ '{', '[' -> {
+ val open = s[i]
+ val close = if (open == '{') '}' else ']'
+ var depth = 0
+ while (i < s.length) {
+ when (s[i]) {
+ '"' -> {
+ i = skipString(s, i)
+ continue
+ }
+ open -> depth++
+ close -> {
+ depth--
+ if (depth == 0) return i + 1
+ }
+ }
+ i++
+ }
+ throw IllegalArgumentException("Unbalanced JSON value at offset $start")
+ }
+ else -> { // number, true, false, null
+ while (i < s.length && s[i] !in charArrayOf(',', '}', ']') && !s[i].isWhitespace()) i++
+ i
+ }
+ }
+ }
+
+ private fun skipWhitespace(s: String, start: Int): Int {
+ var i = start
+ while (i < s.length && s[i].isWhitespace()) i++
+ return i
+ }
+
+ /**
+ * Find the value of member [key] in the object starting at [objStart]
+ * (default: the root object). Returns the value's inclusive index range,
+ * or null if the key is not present at this object's top level.
+ */
+ fun findMemberValue(s: String, key: String, objStart: Int = 0): IntRange? {
+ var i = skipWhitespace(s, objStart)
+ require(i < s.length && s[i] == '{') { "Expected object at offset $objStart" }
+ i++
+ while (i < s.length) {
+ i = skipWhitespace(s, i)
+ if (s[i] == '}') return null
+ require(s[i] == '"') { "Expected member key at offset $i" }
+ val keyEnd = skipString(s, i)
+ val memberKey = unescapeJsonString(s.substring(i + 1, keyEnd - 1))
+ i = skipWhitespace(s, keyEnd)
+ require(s[i] == ':') { "Expected ':' at offset $i" }
+ i = skipWhitespace(s, i + 1)
+ val valueEnd = skipValue(s, i)
+ if (memberKey == key) return IntRange(i, valueEnd - 1)
+ i = skipWhitespace(s, valueEnd)
+ if (i < s.length && s[i] == ',') i++ else if (i < s.length && s[i] == '}') return null
+ }
+ return null
+ }
+
+ /** Find the value range of a member of the root object. */
+ fun findTopLevelMemberValue(s: String, key: String): IntRange? = findMemberValue(s, key, 0)
+
+ /** Inclusive ranges of the elements of the array spanning [arrayRange]. */
+ fun parseArrayElements(s: String, arrayRange: IntRange): List<IntRange> {
+ var i = skipWhitespace(s, arrayRange.first)
+ require(s[i] == '[') { "Expected array at offset ${arrayRange.first}" }
+ i = skipWhitespace(s, i + 1)
+ val elements = mutableListOf<IntRange>()
+ if (i <= arrayRange.last && s[i] == ']') return elements
+ while (i <= arrayRange.last) {
+ val end = skipValue(s, i)
+ elements.add(IntRange(skipWhitespace(s, i), end - 1))
+ i = skipWhitespace(s, end)
+ if (i > arrayRange.last || s[i] == ']') break
+ require(s[i] == ',') { "Expected ',' in array at offset $i" }
+ i = skipWhitespace(s, i + 1)
+ }
+ return elements
+ }
+
+ /** The string value of member [key] of the object at [objRange], unescaped; null if absent or not a string. */
+ fun extractMemberString(s: String, objRange: IntRange, key: String): String? {
+ val valueRange = findMemberValue(s, key, objRange.first) ?: return null
+ if (s[valueRange.first] != '"') return null
+ return unescapeJsonString(s.substring(valueRange.first + 1, valueRange.last))
+ }
+
+ fun unescapeJsonString(escaped: String): String {
+ if ('\\' !in escaped) return escaped
+ val sb = StringBuilder(escaped.length)
+ var i = 0
+ while (i < escaped.length) {
+ val c = escaped[i]
+ if (c != '\\') {
+ sb.append(c); i++; continue
+ }
+ i++
+ when (val e = escaped[i]) {
+ '"', '\\', '/' -> sb.append(e)
+ 'b' -> sb.append('\b')
+ 'f' -> sb.append('\u000C')
+ 'n' -> sb.append('\n')
+ 'r' -> sb.append('\r')
+ 't' -> sb.append('\t')
+ 'u' -> {
+ sb.append(escaped.substring(i + 1, i + 5).toInt(16).toChar())
+ i += 4
+ }
+ else -> throw IllegalArgumentException("Invalid JSON escape '\\$e'")
+ }
+ i++
+ }
+ return sb.toString()
+ }
+}
diff --git a/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillTarMergeTest.kt b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillTarMergeTest.kt
new file mode 100644
index 0000000..22b561b
--- /dev/null
+++ b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillTarMergeTest.kt
@@ -0,0 +1,342 @@
+package de.ids_mannheim.korapxmltools
+
+import org.apache.commons.compress.archivers.tar.TarArchiveInputStream
+import org.junit.After
+import org.junit.AfterClass
+import org.junit.Before
+import java.io.ByteArrayInputStream
+import java.io.ByteArrayOutputStream
+import java.io.File
+import java.io.PrintStream
+import java.net.URL
+import java.util.zip.GZIPInputStream
+import kotlin.test.Test
+import kotlin.test.assertContains
+import kotlin.test.assertEquals
+import kotlin.test.assertFalse
+import kotlin.test.assertNotEquals
+import kotlin.test.assertTrue
+
+/**
+ * Tests for Krill tar merge mode: updating an existing Krill tar from
+ * KorAP-XML ZIPs and stand-off metadata files.
+ */
+class KrillTarMergeTest {
+
+ companion object {
+ private val tempDirs = mutableListOf<File>()
+
+ private fun newTempDir(key: String): File =
+ File.createTempFile(key, "").apply {
+ delete()
+ mkdirs()
+ tempDirs.add(this)
+ }
+
+ // Generated once and shared between tests (inputs are never modified)
+ private val baseTar: File by lazy { generateTar("merge_base", "rei_sample.krill.tar", resource("rei_sample.zip")) }
+ private val ttTar: File by lazy {
+ generateTar("merge_tt", "rei_sample.krill.tar", resource("rei_sample.zip"), resource("rei_sample.tree_tagger.zip"))
+ }
+
+ private fun resource(path: String): String {
+ val url: URL = Thread.currentThread().contextClassLoader.getResource(path)
+ ?: throw IllegalArgumentException("Resource $path not found")
+ return File(url.toURI()).path
+ }
+
+ private fun generateTar(key: String, tarName: String, vararg inputs: String): File {
+ val outputDir = newTempDir(key)
+ val exitCode = debug(arrayOf("-t", "krill", "-q", "-D", outputDir.path) + inputs)
+ assertEquals(0, exitCode, "Krill conversion should succeed for '$key'")
+ val tar = File(outputDir, tarName)
+ assertTrue(tar.exists(), "Expected $tarName for '$key'")
+ return tar
+ }
+
+ /** Raw (still compressed) tar entries by name, in order. */
+ private fun readTarEntries(tar: File): LinkedHashMap<String, ByteArray> {
+ val entries = LinkedHashMap<String, ByteArray>()
+ TarArchiveInputStream(tar.inputStream().buffered()).use { tarIn ->
+ var entry = tarIn.nextEntry
+ while (entry != null) {
+ if (entry.isFile) {
+ entries[entry.name] = tarIn.readAllBytes()
+ }
+ entry = tarIn.nextEntry
+ }
+ }
+ return entries
+ }
+
+ /** Decompressed JSON entries by name, in order. */
+ private fun readTarJson(tar: File): LinkedHashMap<String, String> {
+ val result = LinkedHashMap<String, String>()
+ readTarEntries(tar).forEach { (name, bytes) ->
+ if (name.endsWith(".json.gz")) {
+ result[name] = GZIPInputStream(ByteArrayInputStream(bytes)).bufferedReader().use { it.readText() }
+ }
+ }
+ return result
+ }
+
+ @JvmStatic
+ @AfterClass
+ fun cleanupTempDirs() {
+ tempDirs.forEach { it.deleteRecursively() }
+ tempDirs.clear()
+ }
+ }
+
+ private val outContent = ByteArrayOutputStream(10000000)
+ private val errContent = ByteArrayOutputStream()
+ private val originalOut: PrintStream = System.out
+ private val originalErr: PrintStream = System.err
+
+ @Before
+ fun setUpStreams() {
+ System.setOut(PrintStream(outContent))
+ System.setErr(PrintStream(errContent))
+ }
+
+ @After
+ fun restoreStreams() {
+ System.setOut(originalOut)
+ System.setErr(originalErr)
+ }
+
+ private fun mergeTar(key: String, inputTar: File, vararg inputs: String): Pair<File, File> {
+ val outputDir = newTempDir(key)
+ val exitCode = debug(arrayOf("-t", "krill", "-q", "-D", outputDir.path, inputTar.path) + inputs)
+ assertEquals(0, exitCode, "Krill tar merge should succeed for '$key'")
+ val outputs = outputDir.listFiles { f -> f.name.endsWith(".tar") }.orEmpty()
+ assertEquals(1, outputs.size, "Expected exactly one merged tar for '$key'")
+ val log = File(outputs[0].path.replace(Regex("\\.tar$"), ".log"))
+ return outputs[0] to log
+ }
+
+ // ------------------------------------------------------------------
+ // Phase 1: metadata merge
+ // ------------------------------------------------------------------
+
+ @Test
+ fun mergedStandoffMetadataMatchesFullGeneration() {
+ val standoff = resource("rei_sample.domains.meta.xml")
+ val full = generateTar("standoff_full", "rei_sample.krill.tar", resource("rei_sample.zip"), standoff)
+ val (merged, _) = mergeTar("standoff_merge", baseTar, standoff)
+
+ val fullJson = readTarJson(full)
+ val mergedJson = readTarJson(merged)
+ // Full generation writes texts in compression-completion order, which is not
+ // deterministic; only the set of entries and their contents must match.
+ assertEquals(fullJson.keys, mergedJson.keys, "Entry names should match")
+ fullJson.forEach { (name, expected) ->
+ assertEquals(expected, mergedJson[name], "Merged JSON should equal full generation for $name")
+ }
+ assertContains(mergedJson.values.first(), "wikiDomain")
+ }
+
+ @Test
+ fun mergeLeavesUnaffectedTextsByteIdentical() {
+ // Stand-off metadata for a single text only
+ val standoffFile = File(newTempDir("standoff_partial_input"), "partial.meta.xml")
+ standoffFile.writeText(
+ """
+ <standOff xmlns="http://www.tei-c.org/ns/1.0">
+ <metadataLayer xml:id="testDomain" type="classification">
+ <taxonomy xml:id="testtaxonomy">
+ <category xml:id="Politics"><catDesc>Politics</catDesc></category>
+ </taxonomy>
+ <textRef target="REI_RBR.00473">
+ <catRef scheme="#testtaxonomy" target="#Politics" n="1" cert="0.9"/>
+ </textRef>
+ </metadataLayer>
+ </standOff>
+ """.trimIndent()
+ )
+ val (merged, _) = mergeTar("standoff_partial", baseTar, standoffFile.path)
+
+ val baseEntries = readTarEntries(baseTar)
+ val mergedEntries = readTarEntries(merged)
+ assertEquals(baseEntries.keys.toList(), mergedEntries.keys.toList())
+
+ baseEntries.forEach { (name, bytes) ->
+ if (name.startsWith("REI-RBR-00473")) {
+ assertFalse(bytes.contentEquals(mergedEntries[name]!!), "$name should have been patched")
+ } else {
+ assertTrue(bytes.contentEquals(mergedEntries[name]!!),
+ "$name should be copied through byte-identically")
+ }
+ }
+ val patched = readTarJson(merged).getValue("REI-RBR-00473.json.gz")
+ assertContains(patched, "\"key\":\"testDomain\"")
+ assertContains(patched, "\"Politics\"")
+ }
+
+ @Test
+ fun mergeReplacesExistingStandoffField() {
+ // First give all texts wikiDomain fields, then merge an updated classification
+ // for one text and check it replaces (not duplicates) the old field.
+ val withDomains = generateTar(
+ "standoff_replace_base", "rei_sample.krill.tar",
+ resource("rei_sample.zip"), resource("rei_sample.domains.meta.xml")
+ )
+ val updateFile = File(newTempDir("standoff_replace_input"), "update.meta.xml")
+ updateFile.writeText(
+ """
+ <standOff xmlns="http://www.tei-c.org/ns/1.0">
+ <metadataLayer xml:id="wikiDomain" type="classification">
+ <taxonomy xml:id="wikitaxonomy">
+ <category xml:id="UpdatedTopic"><catDesc>Updated topic</catDesc></category>
+ </taxonomy>
+ <textRef target="REI_RBR.00473">
+ <catRef scheme="#wikitaxonomy" target="#UpdatedTopic" n="1" cert="0.99"/>
+ </textRef>
+ </metadataLayer>
+ </standOff>
+ """.trimIndent()
+ )
+ val (merged, _) = mergeTar("standoff_replace", withDomains, updateFile.path)
+ val patched = readTarJson(merged).getValue("REI-RBR-00473.json.gz")
+ assertContains(patched, "UpdatedTopic")
+ assertEquals(1, Regex("\"key\":\"wikiDomain\"").findAll(patched).count(),
+ "wikiDomain must be replaced, not duplicated")
+ }
+
+ @Test
+ fun mergeWarnsAndIgnoresTextsNotInTar() {
+ val wud24Tar = File(resource("wud24_sample.krill.tar"))
+ val (merged, log) = mergeTar("ignore_missing", wud24Tar, resource("rei_sample.tree_tagger.zip"))
+
+ // Nothing to patch: all entries must be byte-identical copies
+ val inEntries = readTarEntries(wud24Tar)
+ val outEntries = readTarEntries(merged)
+ assertEquals(inEntries.keys.toList(), outEntries.keys.toList())
+ inEntries.forEach { (name, bytes) ->
+ assertTrue(bytes.contentEquals(outEntries[name]!!), "$name should be unchanged")
+ }
+
+ assertTrue(log.exists(), "Merge log file should exist")
+ val logText = log.readText()
+ assertContains(logText, "Ignoring text REI_RBR.00473")
+ assertContains(logText, "not present in")
+ }
+
+ // ------------------------------------------------------------------
+ // Phase 2: annotation foundry merge
+ // ------------------------------------------------------------------
+
+ @Test
+ fun mergedTreeTaggerFoundryMatchesFullGeneration() {
+ val (merged, _) = mergeTar("tt_merge", baseTar, resource("rei_sample.tree_tagger.zip"))
+
+ val fullJson = readTarJson(ttTar)
+ val mergedJson = readTarJson(merged)
+ assertEquals(fullJson.keys, mergedJson.keys)
+ fullJson.forEach { (name, expected) ->
+ assertEquals(expected, mergedJson[name], "Merged JSON should equal full generation for $name")
+ }
+ }
+
+ @Test
+ fun mergedMaltDependenciesMatchFullGeneration() {
+ val full = generateTar(
+ "malt_full", "rei_sample.krill.tar",
+ resource("rei_sample.zip"), resource("rei_sample.malt.zip")
+ )
+ val (merged, _) = mergeTar("malt_merge", baseTar, resource("rei_sample.malt.zip"))
+
+ val fullJson = readTarJson(full)
+ val mergedJson = readTarJson(merged)
+ assertEquals(fullJson.keys, mergedJson.keys)
+ fullJson.forEach { (name, expected) ->
+ assertEquals(expected, mergedJson[name], "Merged JSON should equal full generation for $name")
+ }
+ }
+
+ @Test
+ fun mergedMultipleFoundriesMatchFullGeneration() {
+ val full = generateTar(
+ "multi_full", "rei_sample.krill.tar",
+ resource("rei_sample.zip"), resource("rei_sample.tree_tagger.zip"),
+ resource("rei_sample.malt.zip"), resource("rei_sample.opennlp.zip")
+ )
+ val (merged, _) = mergeTar(
+ "multi_merge", baseTar,
+ resource("rei_sample.tree_tagger.zip"), resource("rei_sample.malt.zip"),
+ resource("rei_sample.opennlp.zip")
+ )
+
+ val fullJson = readTarJson(full)
+ val mergedJson = readTarJson(merged)
+ assertEquals(fullJson.keys, mergedJson.keys)
+ fullJson.forEach { (name, expected) ->
+ assertEquals(expected, mergedJson[name], "Merged JSON should equal full generation for $name")
+ }
+ }
+
+ @Test
+ fun mergeReplacesExistingFoundry() {
+ // Re-merging the same tree_tagger annotations into a tar that already has
+ // them must be idempotent (replace, not duplicate).
+ val (merged, _) = mergeTar("tt_replace", ttTar, resource("rei_sample.tree_tagger.zip"))
+
+ val fullJson = readTarJson(ttTar)
+ val mergedJson = readTarJson(merged)
+ assertEquals(fullJson.keys, mergedJson.keys)
+ fullJson.forEach { (name, expected) ->
+ assertEquals(expected, mergedJson[name], "Replacing a foundry with itself must be idempotent for $name")
+ }
+ }
+
+ @Test
+ fun mergeSupportsLz4CompressedTars() {
+ val outputDir = newTempDir("lz4_base")
+ val exitCode = debug(arrayOf("-t", "krill", "-q", "--lz4", "-D", outputDir.path, resource("rei_sample.zip")))
+ assertEquals(0, exitCode)
+ val lz4Tar = File(outputDir, "rei_sample.krill.tar")
+ assertTrue(lz4Tar.exists())
+
+ val (merged, _) = mergeTar("lz4_merge", lz4Tar, resource("rei_sample.tree_tagger.zip"))
+ val entries = readTarEntries(merged)
+ assertTrue(entries.keys.all { it.endsWith(".json.lz4") }, "Entries should stay LZ4-compressed")
+ val json = net.jpountz.lz4.LZ4FrameInputStream(
+ ByteArrayInputStream(entries.getValue("REI-RBR-00473.json.lz4"))
+ ).bufferedReader().use { it.readText() }
+ assertContains(json, "tt/p:")
+ assertContains(json, "treetagger/morpho")
+ }
+
+ @Test
+ fun remergingHeadersFromBaseZipIsIdempotent() {
+ // The base ZIP carries the same headers the tar was built from, so re-merging
+ // it must leave every text's JSON unchanged (data/tokens/structure entries are
+ // ignored, header-derived fields regenerate to identical values).
+ val (merged, log) = mergeTar("header_remerge", baseTar, resource("rei_sample.zip"))
+
+ val baseJson = readTarJson(baseTar)
+ val mergedJson = readTarJson(merged)
+ assertEquals(baseJson.keys, mergedJson.keys)
+ baseJson.forEach { (name, expected) ->
+ assertEquals(expected, mergedJson[name], "Re-merging identical headers must not change $name")
+ }
+ assertContains(log.readText(), "cannot change the base tokenization")
+ }
+
+ // ------------------------------------------------------------------
+ // CLI semantics
+ // ------------------------------------------------------------------
+
+ @Test
+ fun mergeRejectsNonKrillOutputFormat() {
+ val exitCode = debug(arrayOf("-t", "conllu", baseTar.path, resource("rei_sample.tree_tagger.zip")))
+ assertNotEquals(0, exitCode, "Tar input without -t krill must be rejected")
+ }
+
+ @Test
+ fun mergeRequiresNewData() {
+ val outputDir = newTempDir("merge_no_data")
+ val exitCode = debug(arrayOf("-t", "krill", "-q", "-D", outputDir.path, baseTar.path))
+ assertNotEquals(0, exitCode, "Merging without any new input must be rejected")
+ }
+}