Derive externalLink for Krill when not explicitly encoded (#47)
In some KorAP-XML corpora the full-text link is not encoded as a
ref[@type=page_url]/link element. ExternalLinkResolver now derives it,
mirroring the predecessor tool (KorAP-XML-Krill, KorAP::XML::Meta::I5):
- Wikipedia (wpd17/wdd17): the article URL only appears in the
reference[type=complete] text; extracted with title "Wikipedia".
This is the bug reported in #47.
- DGD/AGD/FOLK: DGD access link built from the transcript title
(trailing _DF_<n> stripped), title "DGD".
- Süddeutsche Zeitung (U<yy> sigles): szarchiv link from the biblNote
"ID:", title "Süddeutsche Zeitung".
- Genios newspapers: genios.de link from the corpus sigle (mapped to
an orikuerzel via the generated GeniosKuerzelMap) plus the biblNote
"ID:", title "GENIOS". Genios detection requires a well-formed
[A-Z]{1,3}[0-9]{2} sigle so arbitrary sigles cannot trigger a link.
The rules are pure functions applied in priority order; explicitly
encoded links still take precedence. The link title is carried to the
Krill renderer via a new, non-emitted externalLinkTitle helper field.
Adds ExternalLinkResolverTest (unit tests for all four rules and
precedence) and two KrillJsonGeneratorTest cases: Wikipedia extraction
from wdd17sample.zip and the Genios wiring on a W24 header.
Resolves #47
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change-Id: I181254fc5367773701bb3502e55c8531f52c1258
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5c3eb56..a8020d3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,11 +4,13 @@
### Added
+- Krill `externalLink` is now derived for sources where the full-text link is not explicitly encoded as a `ref[@type=page_url]`/`link` element, mirroring the predecessor tool (KorAP-XML-Krill): DGD/AGD/FOLK transcripts (title `_DF_<n>` suffix stripped, title "DGD"), Süddeutsche Zeitung (`U<yy>` sigles → szarchiv link from the biblNote `ID:`, title "Süddeutsche Zeitung"), and Genios newspapers (corpus sigle → `orikuerzel` via a built-in mapping + biblNote `ID:` → genios.de link, title "GENIOS"). Explicitly encoded links still take precedence. Link titles are carried via a new `externalLinkTitle` helper field.
- Sentence segmentation fallback: when a corpus has no `s` spans in `structure.xml` (common for TEI conversions of custom corpora), sentence boundaries now fall back to other segment-like TEI elements, in order of preference: `posting` (chat/CMC), `l` (verse line), `seg` (segment), `u` (utterance). This makes integrated taggers/parsers and KorAP sentence-based queries work on such corpora.
- New `dck_sample.zip` test resource: two-text excerpt from the CC BY licensed Dortmunder Chat-Korpus with custom `cmc` tokenization/annotations (in one text the `s` spans are removed to exercise the sentence fallback)
### Fixed
+- Krill `externalLink` is now picked up from older Wikipedia corpora (wpd17/wdd17) where the article URL is only present in the `reference[type=complete]` text rather than encoded as a `ref`/`link` element ([#47](https://github.com/KorAP/korapxmltool/issues/47)). The URL is extracted with title "Wikipedia".
- `tokenSource` in Krill JSON output no longer resolves to a stand-off annotation foundry ([#48](https://github.com/KorAP/korapxmltool/issues/48)): token source identification and token-map updates are now restricted to base archive processing, so stand-off annotation foundries can no longer overwrite the base tokenization spans or steal the `tokenSource`.
- Krill output now drops texts that contain no tokens instead of emitting empty, unindexable documents ([#46](https://github.com/KorAP/korapxmltool/issues/46)). Such texts (e.g. articles with empty `data.xml`/`base/tokens.xml`) are skipped with a per-text warning and a summary count; texts with at least one token are kept. New `m21_empty_sample.zip` regression fixture (one empty text, one single-token text).
- Corpora with custom tokenization and annotations inside the base ZIP (e.g. `cmc/morpho.xml` from TEI conversions, with no `base/tokens.xml`) are now handled correctly: the foundry is derived from the annotation folder name instead of the ZIP file name, so Krill output indexes the annotations (e.g. `cmc/p`, `cmc/l`) instead of silently dropping them, the token stream is no longer empty (`tokenSource` is set to e.g. `cmc#morpho`), and CoNLL-U output reports `# foundry = cmc` instead of `# foundry = base`
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/ExternalLinkResolver.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/ExternalLinkResolver.kt
new file mode 100644
index 0000000..e89baac
--- /dev/null
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/ExternalLinkResolver.kt
@@ -0,0 +1,77 @@
+package de.ids_mannheim.korapxmltools
+
+/**
+ * Derives a text's external full-text link for Krill output in cases where the link is not
+ * explicitly encoded as a `ref[@type=page_url]`/`link` element, mirroring the behaviour of the
+ * predecessor tool (KorAP-XML-Krill, KorAP::XML::Meta::I5):
+ *
+ * - Wikipedia: the URL is only present in the `reference[type=complete]` text.
+ * - DGD/AGD/FOLK: a DGD access link is built from the corpus sigle and the transcript title.
+ * - Süddeutsche Zeitung (sigle `U<yy>`): an szarchiv link is built from the biblNote `ID:`.
+ * - Genios newspapers: a genios.de link is built from the corpus sigle (via [GeniosKuerzelMap])
+ * and the biblNote `ID:`.
+ *
+ * All rules are pure functions of the header values so they can be unit-tested without ZIP
+ * fixtures. [resolve] applies them in priority order; the first match wins.
+ */
+object ExternalLinkResolver {
+ data class ResolvedLink(val url: String, val title: String)
+
+ // Wikipedia references end in "... URL:<url>: Wikipedia, <year>" (http or https).
+ private val WIKIPEDIA_REFERENCE = Regex("""URL:(https?:.+?):\s+Wikipedia,\s+\d+\s*$""")
+ // biblNote carries the source id as "ID: <id> ..." (id may itself start with a letter, e.g. SZ).
+ private val BIBL_NOTE_ID = Regex("""ID:\s*(\S+)""")
+ // Local newspaper corpus sigles always look like [A-Z]{1,3}[0-9]{2}; the two digits are the
+ // year volume. Genios detection requires this shape so arbitrary sigles can't trigger a link.
+ private val NEWSPAPER_SIGLE = Regex("""^[A-Z]{1,3}\d{2}$""")
+ private val SUEDDEUTSCHE_SIGLE = Regex("""^U\d{2}$""")
+ private val DGD_SIGLE = Regex("""^(?:[AD]GD|FOLK)$""")
+ private val DGD_TRANSCRIPT_SUFFIX = Regex("""_DF_\d+$""", RegexOption.IGNORE_CASE)
+
+ /** Extract the `ID:` value from a biblNote text, or null if absent. */
+ fun biblNoteId(text: String?): String? =
+ text?.let { BIBL_NOTE_ID.find(it)?.groupValues?.get(1) }
+
+ /** The Genios botkuerzel for a corpus sigle: its leading letters, lowercased (e.g. W24 -> "w"). */
+ private fun botkuerzel(corpusSigle: String?): String? =
+ corpusSigle?.takeWhile { it.isLetter() }?.lowercase()?.takeIf { it.isNotEmpty() }
+
+ fun wikipedia(reference: String?): ResolvedLink? =
+ reference?.let { WIKIPEDIA_REFERENCE.find(it) }
+ ?.let { ResolvedLink(it.groupValues[1], "Wikipedia") }
+
+ fun dgd(corpusSigle: String?, title: String?): ResolvedLink? {
+ if (corpusSigle == null || !DGD_SIGLE.matches(corpusSigle)) return null
+ val transcript = title?.trim()?.takeIf { it.isNotEmpty() }
+ ?.replace(DGD_TRANSCRIPT_SUFFIX, "") ?: return null
+ return ResolvedLink(
+ "https://dgd.ids-mannheim.de/DGD2Web/ExternalAccessServlet?command=displayData&id=$transcript",
+ "DGD"
+ )
+ }
+
+ fun sueddeutsche(corpusSigle: String?, biblNoteId: String?): ResolvedLink? {
+ if (corpusSigle == null || !SUEDDEUTSCHE_SIGLE.matches(corpusSigle) || biblNoteId == null) return null
+ return ResolvedLink(
+ "https://archiv.szarchiv.de/Portal/restricted/Start.act?articleId=$biblNoteId",
+ "Süddeutsche Zeitung"
+ )
+ }
+
+ fun genios(corpusSigle: String?, biblNoteId: String?): ResolvedLink? {
+ if (corpusSigle == null || biblNoteId == null || !NEWSPAPER_SIGLE.matches(corpusSigle)) return null
+ val orikuerzel = botkuerzel(corpusSigle)?.let { GeniosKuerzelMap.byBotkuerzel[it] } ?: return null
+ return ResolvedLink("https://www.genios.de/document/${orikuerzel}__$biblNoteId", "GENIOS")
+ }
+
+ /**
+ * Try every derivation rule in priority order and return the first match. Wikipedia is
+ * content-detected and wins over the sigle-based rules; Süddeutsche is checked before Genios
+ * so `U<yy>` sigles resolve to szarchiv rather than a colliding Genios botkuerzel.
+ */
+ fun resolve(corpusSigle: String?, reference: String?, biblNoteId: String?, title: String?): ResolvedLink? =
+ wikipedia(reference)
+ ?: dgd(corpusSigle, title)
+ ?: sueddeutsche(corpusSigle, biblNoteId)
+ ?: genios(corpusSigle, biblNoteId)
+}
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/GeniosKuerzelMap.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/GeniosKuerzelMap.kt
new file mode 100644
index 0000000..714cace
--- /dev/null
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/GeniosKuerzelMap.kt
@@ -0,0 +1,320 @@
+// GENERATED FILE - DO NOT EDIT BY HAND.
+// Genios newspaper short-code (botkuerzel) -> Genios document prefix (orikuerzel) mapping,
+// generated from the IDS Genios metadata.xml. Used to build genios.de full-text links for
+// DeReKo corpora that originate from Genios (see ExternalLinkResolver.genios).
+// To regenerate, re-run the extraction over an updated metadata.xml.
+package de.ids_mannheim.korapxmltools
+
+object GeniosKuerzelMap {
+ // Keyed by the lowercased alphabetic prefix of a corpus sigle (the botkuerzel).
+ val byBotkuerzel: Map<String, String> = mapOf(
+ "a" to "STG",
+ "aan" to "AAN",
+ "aaz" to "AAZ",
+ "abm" to "ALBB",
+ "abo" to "AARB",
+ "aez" to "AEZT",
+ "afz" to "AFZ",
+ "agf" to "AGEF",
+ "agz" to "ED",
+ "aho" to "AUON",
+ "ait" to "AUTI",
+ "alz" to "ALLZ",
+ "art" to "GART",
+ "aue" to "AUEL",
+ "auf" to "AF",
+ "auh" to "AUTH",
+ "aui" to "AUIN",
+ "aup" to "AUPR",
+ "aut" to "AUTO",
+ "azm" to "MUAZ",
+ "b" to "BEZE",
+ "baz" to "BAZ",
+ "bdw" to "BIWI",
+ "bdz" to "BADZ",
+ "bee" to "BEEF",
+ "bei" to "BE",
+ "beo" to "BEOB",
+ "bez" to "BERN",
+ "bil" to "ABIL",
+ "bkr" to "BKR",
+ "bku" to "BKU",
+ "bla" to "BLAU",
+ "bli" to "BLI",
+ "bmo" to "BGM",
+ "bna" to "BNA",
+ "bot" to "BVH",
+ "boz" to "BOEZ",
+ "brg" to "BRIG",
+ "brm" to "BRIM",
+ "bru" to "BR",
+ "brw" to "BRIW",
+ "bsz" to "BSTZ",
+ "bue" to "BUER",
+ "bun" to "BUND",
+ "bup" to "BUPU",
+ "bwa" to "BWAI",
+ "bze" to "BZ",
+ "bzg" to "BEZG",
+ "cap" to "CAPI",
+ "cav" to "CAV",
+ "chk" to "CHEK",
+ "chm" to "CHMM",
+ "cho" to "CHMO",
+ "cht" to "CHET",
+ "cic" to "CICE",
+ "cid" to "CID",
+ "cio" to "CIOD",
+ "cit" to "CITL",
+ "cou" to "COU",
+ "ct" to "CT",
+ "ctb" to "COBU",
+ "dae" to "DAE",
+ "dak" to "DIAK",
+ "daz" to "DAZ",
+ "dbt" to "MCDB",
+ "dec" to "DECH",
+ "dgl" to "DGL",
+ "dib" to "DIBA",
+ "dki" to "DIKI",
+ "dnn" to "DNN",
+ "dnv" to "DNV",
+ "dog" to "DOGS",
+ "dol" to "DOL",
+ "dpr" to "PRIG",
+ "dsz" to "DSTZ",
+ "dtz" to "DTZ",
+ "dvz" to "DVZ",
+ "dww" to "DWW",
+ "e" to "TAG",
+ "edf" to "EDF",
+ "eft" to "ETB",
+ "eid" to "EID",
+ "ein" to "EVIN",
+ "elf" to "ELTF",
+ "eli" to "EIND",
+ "elo" to "ELEO",
+ "elt" to "ELTN",
+ "ema" to "EMA",
+ "emt" to "EMT",
+ "ene" to "ENER",
+ "epp" to "EPP",
+ "epr" to "EPRA",
+ "et" to "ETEC",
+ "ett" to "EUTD",
+ "eut" to "EUT",
+ "euw" to "EUZW",
+ "eze" to "EZE",
+ "fis" to "FRIM",
+ "fk" to "FK",
+ "flt" to "FALT",
+ "fmt" to "FORM",
+ "fnp" to "FNP",
+ "foc" to "FOCU",
+ "fom" to "FOCM",
+ "fpc" to "FEPR",
+ "fra" to "FRA",
+ "frt" to "FRT",
+ "fuw" to "FLOW",
+ "gal" to "GALA",
+ "gaz" to "GAZ",
+ "geo" to "GEO",
+ "ges" to "GEOS",
+ "ggt" to "GG",
+ "giz" to "GIAN",
+ "gng" to "GONG",
+ "gob" to "DAGO",
+ "gsp" to "GESP",
+ "gta" to "GTB",
+ "gtb" to "GETA",
+ "gwp" to "GWP",
+ "haa" to "HA",
+ "hab" to "HB",
+ "hat" to "HATA",
+ "hau" to "HAEU",
+ "hdz" to "HZ",
+ "hfz" to "HOFZ",
+ "hgz" to "AHGZ",
+ "hhz" to "HOZE",
+ "hkr" to "HK",
+ "hna" to "HNA",
+ "hrz" to "HOER",
+ "hst" to "HST",
+ "htb" to "HOT",
+ "hzs" to "HERZ",
+ "hzw" to "HZWI",
+ "hzz" to "HOZG",
+ "i" to "TITA",
+ "iee" to "IEE",
+ "imw" to "IMWI",
+ "imz" to "IMMO",
+ "itb" to "ITB",
+ "itt" to "ITT",
+ "ix" to "IX",
+ "jue" to "JUEZ",
+ "k" to "KLEI",
+ "kaz" to "KRAN",
+ "ke" to "KE",
+ "kem" to "KEM",
+ "kfz" to "KFZB",
+ "khh" to "KH",
+ "kir" to "KIRZ",
+ "kn" to "KN",
+ "kru" to "KR",
+ "ksa" to "KSTA",
+ "ktz" to "KTZ",
+ "kur" to "KUR",
+ "kxp" to "EXPR",
+ "l" to "BMP",
+ "lab" to "LB",
+ "lah" to "LIAH",
+ "lan" to "LAAN",
+ "laz" to "LAZE",
+ "lhz" to "LAZ",
+ "lit" to "LIN",
+ "lmd" to "LEMO",
+ "lmz" to "LMZ",
+ "ln" to "LBN",
+ "log" to "HLOG",
+ "lru" to "LR",
+ "ltb" to "LUXT",
+ "lvz" to "LVZ",
+ "mag" to "MAER",
+ "mav" to "MAV",
+ "maz" to "MAZ",
+ "mdr" to "MDR",
+ "mep" to "MEP",
+ "met" to "MUTE",
+ "mib" to "MIB",
+ "mid" to "MID",
+ "mm" to "MM",
+ "mme" to "MUME",
+ "mpo" to "MPW",
+ "msp" to "MASP",
+ "mt" to "MT",
+ "mtk" to "MATK",
+ "mut" to "MUAR",
+ "muv" to "MUVB",
+ "mwo" to "MAC",
+ "mze" to "MZ",
+ "n" to "SN",
+ "nas" to "NECH",
+ "nbk" to "NBK",
+ "ndo" to "NIDO",
+ "neo" to "NEON",
+ "neu" to "NEUL",
+ "new" to "ANEW",
+ "ngz" to "NGVZ",
+ "nku" to "NKU",
+ "nlz" to "NLZ",
+ "nnn" to "NNN",
+ "nnp" to "NNP",
+ "now" to "NOW",
+ "noz" to "NOZ",
+ "npr" to "NPHA",
+ "nun" to "NN",
+ "nuz" to "NZ",
+ "nvb" to "NVB",
+ "nvt" to "NVT",
+ "nwe" to "NEUW",
+ "nws" to "AGZ",
+ "nwt" to "NEWE",
+ "nwz" to "NWZ",
+ "nzf" to "FOLI",
+ "nzs" to "NZZS",
+ "nzz" to "NZZ",
+ "o" to "KRON",
+ "oaz" to "OAZ",
+ "ohz" to "OHEZ",
+ "osz" to "OSZ",
+ "otz" to "OTZ",
+ "ovz" to "OVZ",
+ "p" to "PRE",
+ "paz" to "PAZ",
+ "pco" to "PCW",
+ "pha" to "PHAP",
+ "pmm" to "GJPM",
+ "pnn" to "PNN",
+ "pnp" to "PNP",
+ "prd" to "PRTR",
+ "prf" to "PROF",
+ "qe" to "QE",
+ "r" to "FR",
+ "rbs" to "RBS",
+ "rga" to "GEA",
+ "rhz" to "RZTG",
+ "rlh" to "RLH",
+ "rln" to "RLNR",
+ "rn" to "RN",
+ "rpo" to "RP",
+ "rsw" to "NKR",
+ "rue" to "RELO",
+ "rvz" to "RVZ",
+ "saz" to "SAZE",
+ "sbl" to "SBLI",
+ "sbz" to "SAAR",
+ "sch" to "SHF",
+ "sct" to "SWT",
+ "scv" to "SVER",
+ "scw" to "SOWO",
+ "scz" to "SWAZ",
+ "sku" to "SK",
+ "smp" to "SOMP",
+ "soa" to "STP",
+ "spz" to "DSZ",
+ "sta" to "STA",
+ "stb" to "STAG",
+ "ste" to "STER",
+ "stg" to "STGL",
+ "stn" to "STN",
+ "stz" to "STZ",
+ "svz" to "SVZ",
+ "swp" to "SWP",
+ "sze" to "SZO",
+ "t" to "TAZ",
+ "tai" to "TAI",
+ "tas" to "TAS",
+ "tbz" to "TAB",
+ "ter" to "TERE",
+ "tet" to "TETE",
+ "tew" to "TWA",
+ "tha" to "TA",
+ "thb" to "THBE",
+ "tlz" to "TLZ",
+ "tnz" to "TZH",
+ "toz" to "TZ",
+ "tre" to "TRE",
+ "tru" to "TRUC",
+ "tsg" to "TSPG",
+ "tsp" to "TSP",
+ "tue" to "TUE",
+ "tvd" to "TVDT",
+ "tvf" to "TV",
+ "uan" to "USAN",
+ "v" to "VN",
+ "vbw" to "VBW",
+ "vku" to "VKU",
+ "vra" to "VRAG",
+ "vru" to "VR",
+ "vsw" to "VW",
+ "vzs" to "VZS",
+ "w" to "WELT",
+ "was" to "WAMS",
+ "wbl" to "WB",
+ "weo" to "WEON",
+ "wez" to "WEST",
+ "wfb" to "WFB",
+ "wiz" to "WZ",
+ "wku" to "WK",
+ "wor" to "WORZ",
+ "wtb" to "WITA",
+ "wwo" to "WEWO",
+ "wwt" to "WWT",
+ "x" to "OOEN",
+ "z" to "ZEIT",
+ "zca" to "ZTCS",
+ "zcw" to "CUW",
+ "zge" to "ZTGS",
+ "zwi" to "ZTWI",
+ )
+}
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
index b96ffbd..f339440 100644
--- a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/KorapXmlTool.kt
@@ -5664,6 +5664,25 @@
}
}
+ // Derive the external link for cases where it is not explicitly encoded (Wikipedia URL in
+ // the reference text, DGD/FOLK transcripts, Süddeutsche and Genios newspapers). See
+ // ExternalLinkResolver. Explicitly encoded links (above) always take precedence.
+ if (!metadata.containsKey("externalLink")) {
+ val corpusSigle = headerRoot.firstText("textSigle")?.substringBefore("/")
+ val biblNoteId = headerRoot.childElements("biblNote")
+ .mapNotNull { ExternalLinkResolver.biblNoteId(it.textContent) }
+ .firstOrNull()
+ ExternalLinkResolver.resolve(
+ corpusSigle = corpusSigle,
+ reference = metadata["reference"] as? String,
+ biblNoteId = biblNoteId,
+ title = metadata["title"] as? String
+ )?.let { resolved ->
+ metadata["externalLink"] = resolved.url
+ metadata["externalLinkTitle"] = resolved.title
+ }
+ }
+
val biblNoteElement = analytic.firstElement("biblNote") { it.getAttribute("n") == "url" }
?: monogr.firstElement("biblNote") { it.getAttribute("n") == "url" }
biblNoteElement?.let {
diff --git a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt
index cf22cbe..905a829 100644
--- a/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt
+++ b/app/src/main/kotlin/de/ids_mannheim/korapxmltools/formatters/KrillJsonGenerator.kt
@@ -231,8 +231,10 @@
}
"externalLink" -> {
val url = value.toString()
- // Extract title from corpus/publisher metadata if available
- val title = resolvedHeaderMetadata["publisher"]?.toString() ?: "Link"
+ // Prefer an explicit link title (e.g. "Wikipedia", "GENIOS", "Süddeutsche
+ // Zeitung", "DGD" set by ExternalLinkResolver), then publisher, then "Link".
+ val title = resolvedHeaderMetadata["externalLinkTitle"]?.toString()
+ ?: resolvedHeaderMetadata["publisher"]?.toString() ?: "Link"
val encodedUrl = url.replace(":", "%3A").replace("/", "%2F")
"type:attachement" to jsonString("data:application/x.korap-link;title=$title,$encodedUrl")
}
diff --git a/app/src/test/kotlin/de/ids_mannheim/korapxmltools/ExternalLinkResolverTest.kt b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/ExternalLinkResolverTest.kt
new file mode 100644
index 0000000..ab3380b
--- /dev/null
+++ b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/ExternalLinkResolverTest.kt
@@ -0,0 +1,111 @@
+package de.ids_mannheim.korapxmltools
+
+import kotlin.test.Test
+import kotlin.test.assertEquals
+import kotlin.test.assertNull
+
+/**
+ * Unit tests for the external-link derivation rules (issue #47). These are pure functions of the
+ * header values, so they are tested directly without ZIP fixtures.
+ */
+class ExternalLinkResolverTest {
+
+ @Test
+ fun extractsBiblNoteId() {
+ assertEquals("216199263", ExternalLinkResolver.biblNoteId("ID: 216199263 file: Originaldaten/2024/x@ Categories: ..."))
+ assertEquals("A124349227", ExternalLinkResolver.biblNoteId("ID: A124349227"))
+ assertNull(ExternalLinkResolver.biblNoteId("no id here"))
+ assertNull(ExternalLinkResolver.biblNoteId(null))
+ }
+
+ @Test
+ fun derivesWikipediaLinkFromReferenceText() {
+ val reference = "WDD17/B06.45592 Diskussion:Berlinische Grammatik, In: Wikipedia - " +
+ "URL:http://de.wikipedia.org/wiki/Diskussion:Berlinische_Grammatik: Wikipedia, 2017"
+ val link = ExternalLinkResolver.wikipedia(reference)
+ assertEquals("http://de.wikipedia.org/wiki/Diskussion:Berlinische_Grammatik", link?.url)
+ assertEquals("Wikipedia", link?.title)
+ }
+
+ @Test
+ fun ignoresNonWikipediaReference() {
+ assertNull(ExternalLinkResolver.wikipedia("U24/JUN.00442 Süddeutsche Zeitung, 06.06.2024, S. 14"))
+ assertNull(ExternalLinkResolver.wikipedia(null))
+ }
+
+ @Test
+ fun derivesGeniosLinkFromSigleAndId() {
+ // W24 -> botkuerzel "w" -> orikuerzel WELT
+ val link = ExternalLinkResolver.genios("W24", "216199263")
+ assertEquals("https://www.genios.de/document/WELT__216199263", link?.url)
+ assertEquals("GENIOS", link?.title)
+ }
+
+ @Test
+ fun returnsNoGeniosLinkForUnknownSigle() {
+ assertNull(ExternalLinkResolver.genios("ZZ99", "123"))
+ assertNull(ExternalLinkResolver.genios("W24", null))
+ }
+
+ @Test
+ fun geniosRequiresWellFormedNewspaperSigle() {
+ // "w" maps to WELT, but only a [A-Z]{1,3}[0-9]{2} sigle may trigger a Genios link.
+ assertNull(ExternalLinkResolver.genios("W", "216199263")) // no year volume
+ assertNull(ExternalLinkResolver.genios("W2024", "216199263")) // three digits
+ assertNull(ExternalLinkResolver.genios("w24", "216199263")) // lowercase
+ assertNull(ExternalLinkResolver.genios("WELT24", "216199263")) // four letters
+ assertNull(ExternalLinkResolver.genios("W24/SEP.00359", "216199263")) // full text sigle, not corpus prefix
+ // The well-formed sigle still works.
+ assertEquals("https://www.genios.de/document/WELT__216199263", ExternalLinkResolver.genios("W24", "216199263")?.url)
+ }
+
+ @Test
+ fun derivesSueddeutscheLinkFromSigleAndId() {
+ val link = ExternalLinkResolver.sueddeutsche("U24", "A124349227")
+ assertEquals("https://archiv.szarchiv.de/Portal/restricted/Start.act?articleId=A124349227", link?.url)
+ assertEquals("Süddeutsche Zeitung", link?.title)
+ }
+
+ @Test
+ fun sueddeutscheRequiresYearVolumeSigle() {
+ // The two digits stand for the year volume, so a bare "U" must not match.
+ assertNull(ExternalLinkResolver.sueddeutsche("U", "A1"))
+ assertNull(ExternalLinkResolver.sueddeutsche("UXX", "A1"))
+ }
+
+ @Test
+ fun derivesDgdLinkAndStripsTranscriptSuffix() {
+ val link = ExternalLinkResolver.dgd("FOLK", "FOLK_E_00010_SE_01_DF_01")
+ assertEquals(
+ "https://dgd.ids-mannheim.de/DGD2Web/ExternalAccessServlet?command=displayData&id=FOLK_E_00010_SE_01",
+ link?.url
+ )
+ assertEquals("DGD", link?.title)
+ assertNull(ExternalLinkResolver.dgd("W24", "whatever"))
+ }
+
+ @Test
+ fun resolvePrefersWikipediaOverIdBasedRules() {
+ // A text that is both a Wikipedia reference and carries an ID + Genios sigle resolves to Wikipedia.
+ val reference = "X URL:http://de.wikipedia.org/wiki/Foo: Wikipedia, 2017"
+ val link = ExternalLinkResolver.resolve(
+ corpusSigle = "W24", reference = reference, biblNoteId = "216199263", title = "Foo"
+ )
+ assertEquals("http://de.wikipedia.org/wiki/Foo", link?.url)
+ assertEquals("Wikipedia", link?.title)
+ }
+
+ @Test
+ fun resolvePrefersSueddeutscheOverGenios() {
+ // U-sigle texts go to szarchiv even though "u" could collide with a Genios botkuerzel.
+ val link = ExternalLinkResolver.resolve(
+ corpusSigle = "U24", reference = null, biblNoteId = "A124349227", title = null
+ )
+ assertEquals("Süddeutsche Zeitung", link?.title)
+ }
+
+ @Test
+ fun resolveReturnsNullWhenNothingMatches() {
+ assertNull(ExternalLinkResolver.resolve("ZZZ", "plain reference", null, "title"))
+ }
+}
diff --git a/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillJsonGeneratorTest.kt b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillJsonGeneratorTest.kt
index 1316c14..9cbc88b 100644
--- a/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillJsonGeneratorTest.kt
+++ b/app/src/test/kotlin/de/ids_mannheim/korapxmltools/KrillJsonGeneratorTest.kt
@@ -1315,4 +1315,62 @@
"log should warn about the dropped empty text"
)
}
+
+ /**
+ * Regression test for https://github.com/KorAP/korapxmltool/issues/47
+ *
+ * In older Wikipedia corpora (wdd17/wpd17) the article URL is only present in the
+ * reference[type=complete] text, not encoded as a ref/link element. It must still be picked up
+ * as the Krill externalLink, with title "Wikipedia".
+ */
+ @Test
+ fun krillExtractsWikipediaExternalLinkFromReference() {
+ val baseZip = loadResource("wdd17sample.zip").path
+ val tar = ensureKrillTar("wdd17_externallink", "wdd17sample.krill.tar") { outputDir ->
+ arrayOf("-t", "krill", "-q", "-D", outputDir.path, baseZip)
+ }
+
+ val json = readKrillJson(tar).getValue("WDD17-B06-45592.json")
+ val externalLink = krillFieldValue(json, "externalLink")
+ assertNotNull(externalLink, "Wikipedia text should have an externalLink field")
+ assertTrue(externalLink.contains("title=Wikipedia"), "link title should be Wikipedia: $externalLink")
+ assertTrue(
+ externalLink.contains("de.wikipedia.org%2Fwiki%2FDiskussion%3ABerlinische_Grammatik"),
+ "link should contain the encoded Wikipedia URL: $externalLink"
+ )
+ }
+
+ /**
+ * Wiring test for the Genios newspaper case (issue #47): a corpus sigle whose botkuerzel is
+ * known (W24 -> WELT) plus a biblNote "ID:" yields a genios.de full-text link with title GENIOS.
+ */
+ @Test
+ fun krillDerivesGeniosExternalLinkForNewspaper() {
+ val tool = KorapXmlTool()
+ val metadata = collectKrillMetadata(
+ tool,
+ "W24_SEP.00359",
+ headerElement(
+ """
+ <idsHeader>
+ <fileDesc>
+ <titleStmt><textSigle>W24/SEP.00359</textSigle></titleStmt>
+ <sourceDesc>
+ <biblStruct>
+ <analytic>
+ <h.title type="main">Wir brauchen ein Smartphone-Verbot in Schulen</h.title>
+ <biblNote n="1">ID: 216199263 file: Originaldaten/2024/WELT.xml.zip@ Categories: Ressort: Forum</biblNote>
+ </analytic>
+ <monogr><h.title type="main">Die Welt</h.title></monogr>
+ </biblStruct>
+ <reference type="complete" assemblage="regular">W24/SEP.00359 Die Welt, 12.09.2024, S. 7.</reference>
+ </sourceDesc>
+ </fileDesc>
+ </idsHeader>
+ """
+ )
+ )
+ assertEquals("https://www.genios.de/document/WELT__216199263", metadata["externalLink"])
+ assertEquals("GENIOS", metadata["externalLinkTitle"])
+ }
}