=pod

=encoding utf8

=head1 NAME

korapxml2krill - Merge KorAP-XML data and create Krill documents


=head1 SYNOPSIS

  $ korapxml2krill [archive|extract] --input <directory|archive> [options]


=head1 DESCRIPTION

L<KorAP::XML::Krill> is a library to convert KorAP-XML documents to files
compatible with the L<Krill|https://github.com/KorAP/Krill> indexer.
The C<korapxml2krill> command line tool is a simple wrapper of this library.


=head1 INSTALLATION

The preferred way to install L<KorAP::XML::Krill> is to use L<cpanm|App::cpanminus>.

  $ cpanm https://github.com/KorAP/KorAP-XML-Krill.git

In case everything went well, the C<korapxml2krill> tool will
be available on your command line immediately.
Minimum requirement for L<KorAP::XML::Krill> is Perl 5.32.
Optionally installing L<Archive::Tar::Builder> speeds up archive building.
Optional support for L<Sys::Info> to calculate available cores is available.
In addition to work with zip archives, the C<unzip> tool needs to be present.

=head1 ARGUMENTS

  $ korapxml2krill -z --input <directory> --output <filename>

Without arguments, C<korapxml2krill> converts a directory of a single KorAP-XML document.
It expects the input to point to the text level folder.

=over 2

=item B<archive>

  $ korapxml2krill archive -z --input <directory|archive> --output <directory|tar>

Converts an archive of KorAP-XML documents. It expects a directory
(pointing to the corpus level folder) or one or more zip files as input.

=item B<extract>

  $ korapxml2krill extract --input <archive> --output <directory> --sigle <SIGLE>

Extracts KorAP-XML documents from a zip file.

=item B<serial>

  $ korapxml2krill serial -i <archive1> -i <archive2> -o <directory> -cfg <config-file>

Convert archives in serial. The inputs are not merged but treated
as they are (so they may be premerged or globs).
the C<--out> directory is treated as the base directory where subdirectories
are created based on the archive name. In case the C<--to-tar> flag is given,
the output will be a tar file.


=item B<slimlog>

  $ korapxml2krill slimlog <logfile> > <logfile-slim>

Filters out all useless aka succesfull information from logs, to simplify
log checks. Expects no further options.


=back


=head1 OPTIONS

=over 2

=item B<--input|-i> <directory|zip file>

Directory or zip file(s) of documents to convert.

Without arguments, C<korapxml2krill> expects a folder of a single KorAP-XML
document, while C<archive> expects a KorAP-XML corpus folder or a zip
file to batch process multiple files.
C<extract> expects zip files only.

C<archive> supports multiple input zip files with the constraint
that the first archive listed contains all primary data files
and all meta data files.

  -i file/news.zip -i file/news.malt.zip -i "#file/news.tt.zip"

Input may also be defined using BSD glob wildcards.

  -i 'file/news*.zip'

The extended input array will be sorted in length order, so the shortest
path needs to contain all primary data files and all meta data files.

(The directory structure follows the base directory format
that may include a C<.> root folder.
In this case further archives lacking a C<.> root folder
need to be passed with a hash sign in front of the archive's name.
This may require to quote the parameter.)

To support zip files, a version of C<unzip> needs to be installed that is
compatible with the archive file.

B<The root folder switch using the hash sign is experimental and
may vanish in future versions.>


=item B<--input-base|-ib> <directory>

The base directory for inputs.


=item B<--output|-o> <directory|file>

Output folder for archive processing or
document name for single output (optional),
writes to C<STDOUT> by default
(in case C<output> is not mandatory due to further options).

=item B<--overwrite|-w>

Overwrite files that already exist.


=item B<--token|-t> <foundry>#<file>

Define the default tokenization by specifying
the name of the foundry and optionally the name
of the layer-file. Defaults to C<OpenNLP#tokens>.
This will directly take the file instead of running
the layer implementation!


=item B<--base-sentences|-bs> <foundry>#<layer>

Define the layer for base sentences.
If given, this will be used instead of using C<Base#Sentences>.
Currently C<DeReKo#Structure> and C<DGD#Structure> are the only additional
layers supported.

 Defaults to unset.


=item B<--base-paragraphs|-bp> <foundry>#<layer>

Define the layer for base paragraphs.
If given, this will be used instead of using C<Base#Paragraphs>.
Currently C<DeReKo#Structure> and C<DGD#Structure> are the only additional
layer supported.

 Defaults to unset.


=item B<--base-pagebreaks|-bpb> <foundry>#<layer>

Define the layer for base pagebreaks.
Currently C<DeReKo#Structure> is the only layer supported.

 Defaults to unset.


=item B<--skip|-s> <foundry>[#<layer>]

Skip specific annotations by specifying the foundry
(and optionally the layer with a C<#>-prefix),
e.g. C<Mate> or C<Mate#Morpho>. Alternatively you can skip C<#ALL>.
Can be set multiple times.


=item B<--anno|-a> <foundry>#<layer>

Convert specific annotations by specifying the foundry
(and optionally the layer with a C<#>-prefix),
e.g. C<Mate> or C<Mate#Morpho>.
Can be set multiple times.


=item B<--non-word-tokens|-nwt>

Tokenize non-word tokens like word tokens (defined as matching
C</[\d\w]/>). Useful to treat punctuations as tokens.

 Defaults to unset.


=item B<--non-verbal-tokens|-nvt>

Tokenize non-verbal tokens marked as in the primary data as 
the unicode symbol 'Black Vertical Rectangle' aka \x25ae.

 Defaults to unset.


=item B<--jobs|-j>

Define the number of spawned forks for concurrent jobs
of archive processing.
Defaults to C<0> (everything runs in a single process).

If C<sequential-extraction> is not set to true, this will
also apply to extraction.

Pass C<-1>, and the value will be set automatically to 5
times the number of available cores, in case L<Sys::Info>
is available and can read CPU count (see C<--job-count>).
Be aware, that the report of available cores
may not work in certain conditions. Benchmarking the processing
speed based on the number of jobs may be valuable.

This is I<experimental>.


=item B<--job-count|-jc>

Print job and core information that would be used if
C<-1> was passed to C<--jobs>.


=item B<--koral|-k>

Version of the output format. Supported versions are:
C<0> for legacy serialization, C<0.03> for serialization
with metadata fields as key-values on the root object,
C<0.4> for serialization with metadata fields as a list
of C<"@type":"koral:field"> objects.

Currently defaults to C<0.03>.


=item B<--sequential-extraction|-se>

Flag to indicate, if the C<jobs> value also applies to extraction.
Some systems may have problems with extracting multiple archives
to the same folder at the same time.
Can be flagged using C<--no-sequential-extraction> as well.
Defaults to C<false>.


=item B<--meta|-m>

Define the metadata parser to use. Defaults to C<I5>.
Metadata parsers can be defined in the C<KorAP::XML::Meta> namespace.
This is I<experimental>.


=item B<--gzip|-z>

Compress the output.
Expects a defined C<output> file in single processing.


=item B<--cache|-c>

File to mmap a cache (using L<Cache::FastMmap>).
Defaults to C<korapxml2krill.cache> in the calling directory.


=item B<--cache-size|-cs>

Size of the cache. Defaults to C<50m>.


=item B<--cache-init|-ci>

Initialize cache file.
Can be flagged using C<--no-cache-init> as well.
Defaults to C<true>.


=item B<--cache-delete|-cd>

Delete cache file after processing.
Can be flagged using C<--no-cache-delete> as well.
Defaults to C<true>.


=item B<--config|-cfg>

Configure the parameters of your call in a file
of key-value pairs with whitespace separator

  overwrite 1
  token     DeReKo#Structure
  ...

Supported parameters are:
C<overwrite>, C<gzip>, C<jobs>, C<input-base>,
C<token>, C<log>,
C<cache>, C<cache-size>, C<cache-init>, C<cache-delete>, C<meta>,
C<output>, C<koral>,
C<temporary-extract>, C<sequential-extraction>,
C<base-sentences>, C<base-paragraphs>,
C<base-pagebreaks>,
C<skip> (semicolon separated), C<sigle>
(semicolon separated), C<anno> (semicolon separated).

Configuration parameters will always be overwritten by
passed parameters.


=item B<--temporary-extract|-te>

Only valid for the C<archive> and C<serial>
commands.

This will first extract all files into a
directory and then will archive.
If the directory is given as C<:temp:>,
a temporary directory is used.
This is especially useful to avoid
massive unzipping and potential
network latency.


=item B<--to-tar>

Only valid for the C<archive> command.

Writes the output into a tar archive.
The tar needs to be opened with C<--ignore-zeros> afterwards.


=item B<--sigle|-sg>

Extract the given texts.
Can be set multiple times.
I<Currently only supported on C<extract>.>
Sigles have the structure C<Corpus>/C<Document>/C<Text>.
In case the C<Text> path is omitted, the whole document will be extracted.
On the document level, the postfix wildcard C<*> is supported.

=item B<--lang>

Preferred language for metadata fields. In case multiple titles are
given (on any level) with different C<xml:lang> attributes,
the language given is preferred.
Because titles may have different sources and different priorities,
non-specific language titles may still be preferred in case the title
source has a higher priority.


=item B<--log|-l>

The L<Log::Any> log level, defaults to C<ERROR>.


=item B<--quiet>

Silence all information (non-log) outputs.


=item B<--help|-h>

Print help information.


=item B<--version|-v>

Print version information.

=back

=head1 PERFORMANCE

There are some ways to improve performance for large tasks:

=over 2

=item First unpack

Using the archive or serial command on one or multiple zip files
can be very slow, as it needs to unpack small portions every time.
It's better to use C<--temporary-extract> to unpack the whole archive
first into a temprary directory and then read the extracted files.
This is especially important for remote archives

=item Limit annotations

Per default, all supported annotation layers are sought. This can be limited
by adding C<--skip '#ALL'> and only listing the expected annotations with C<--anno>.

=item Checking the parallel job count

By providing the number of parallel jobs using C<--jobs>, the execution can be tailored to specific
hardware environments.

=item Install ripunzip

For full extraction of data, L<ripunzip|https://github.com/google/ripunzip> can be
used for improved performance.

=back

=head1 ANNOTATION SUPPORT

L<KorAP::XML::Krill> has built-in importer for some annotation foundries and layers
developed in the KorAP project that are part of the KorAP preprocessing pipeline.
The base foundry with paragraphs, sentences, and the text element are mandatory for
L<Krill|https://github.com/KorAP/Krill>.

  Base
    #Paragraphs
    #Sentences

  Connexor
    #Morpho
    #Phrase
    #Sentences
    #Syntax

  CoreNLP
    #Constituency
    #Morpho
    #NamedEntities
    #Sentences

  CorpusExplorer
    #Morpho

  CMC
    #Morpho

  DeReKo
    #Structure

  DGD
    #Morpho
    #Structure

  DRuKoLa
    #Morpho

  Glemm
    #Morpho

  Gingko
    #Morpho

  HNC
    #Morpho

  LWC
    #Dependency

  Malt
    #Dependency

  MarMoT
    #Morpho

  Mate
    #Dependency
    #Morpho

  MDParser
    #Dependency

  NKJP
    #Morpho
    #NamedEntities

  OpenNLP
    #Morpho
    #Sentences

  RWK
    #Morpho
    #Structure

  Sgbr
    #Lemma
    #Morpho

  Spacy
    #Morpho
    #Dependency

  Talismane
    #Dependency
    #Morpho

  TreeTagger
    #Morpho
    #Sentences

  UDPipe
    #Dependency
    #Morpho

  XIP
    #Constituency
    #Morpho
    #Sentences


More importers are in preparation.
New annotation importers can be defined in the C<KorAP::XML::Annotation> namespace.
See the built-in annotation importers as examples.


=head1 METADATA SUPPORT

L<KorAP::XML::Krill> has built-in importer for some meta data variants
that are part of the KorAP preprocessing pipeline.

=over 2

=item B<I5>

Meta data for all I5 files

Environment variables:

=over 4

=item C<K2K_TRANSLATOR_TEXT>

Index the translator as a text field (attachement otherwise).

=item C<K2K_PUBLISHER_STRING>

Index the publisher as a string field (attachement otherwise).


=back

=item B<Sgbr>

Meta data from the Schreibgebrauch project

=item B<Gingko>

Meta data from the Gingko project in addition to I5

=item B<ICC>

Meta data for the ICC in addition to I5

=item B<NKJP>

Meta data for the NKJP corpora

=back

New meta data importers can be defined in the C<KorAP::XML::Meta> namespace.
See the built-in meta data importers as examples.

The I5 metadata definition is based on TEI-P5 and supports C<E<lt>xenoDataE<gt>>
with C<E<lt>metaE<gt>> elements like

  <meta type="..." name="..." project="..." desc="...">...</meta>

that are directly translated to Krill objects. The supported values are:

=over 2

=item C<type>

=over 4

=item C<string>

String meta data value

=item C<keyword>

String meta data value that can be given multiple times

=item C<text>

String meta data value that is tokenized and can be searched as token sequences

=item C<date>

Date meta data value (as "yyyy/mm/dd" with optional granularity)

=item C<integer>

Numerical meta data value

=item C<attachment>

Non-indexed meta data value (only retrievable)

=item C<uri>

Non-indexed attached URI, takes the desc as the title for links

=back

=item C<name>

The key of the meta object that may be prefixed by C<corpus> or C<doc>, in case the
C<E<lt>xenoDataE<gt>> information is located on these levels. The text level introduces
no prefixes.

=item C<project> (optional)

A prefixed namespace of the key

=item C<desc> (optional)

A description of the key

=item text content

The value of the meta object

=back


=head1 About KorAP-XML

KorAP-XML (Bański et al. 2012) is an implementation of the KorAP
data model (Bański et al. 2013), where text data are stored physically
separated from their interpretations (i.e. annotations).
A text document in KorAP-XML therefore consists of several files
containing primary data, metadata and annotations.

The structure of a single KorAP-XML document can be as follows:

  - data.xml
  - header.xml
    + base
      - tokens.xml
      - ...
    + struct
      - structure.xml
      - ...
    + corenlp
      - morpho.xml
      - constituency.xml
      - ...
    + tree_tagger
      - morpho.xml
      - ...
    - ...

The C<data.xml> contains the primary data, the C<header.xml> contains
the metadata, and the annotation layers are stored in subfolders
like C<base>, C<struct> or C<corenlp>
(so-called "foundries"; Bański et al. 2013).

Metadata is available in the TEI-P5 variant I5
(Lüngen and Sperberg-McQueen 2012). See the documentation in
L<KorAP::XML::Meta::I5> for translatable fields.

Annotations correspond to a variant of the TEI-P5 feature structures
(TEI Consortium; Lee et al. 2004).
Annotation feature structures refer to character sequences of the primary text
inside the C<text> element of the C<data.xml>.
A single annotation containing the lemma of a token can have the following structure:

  <span from="0" to="3">
    <fs type="lex" xmlns="http://www.tei-c.org/ns/1.0">
      <f name="lex">
        <fs>
          <f name="lemma">zum</f>
        </fs>
      </f>
    </fs>
  </span>

The C<from> and C<to> attributes are refering to the character span
in the primary text.
Depending on the kind of annotation (e.g. token-based, span-based, relation-based),
the structure may vary. See L<KorAP::XML::Annotation::*> for various
annotation preprocessors.

Multiple KorAP-XML documents are organized on three levels following
the "IDS Textmodell" (Lüngen and Sperberg-McQueen 2012):
corpus E<gt> document E<gt> text. On each level metadata information
can be stored, that C<korapxml2krill> will merge to a single metadata
object per text. A corpus is therefore structured as follows:

  + <corpus>
    - header.xml
    + <document>
      - header.xml
      + <text>
        - data.xml
        - header.xml
        - ...
    - ...

A single text can be identified by the concatenation of
the corpus identifier, the document identifier and the text identifier.
This identifier is called the text sigle
(e.g. a text with the identifier C<18486> in the document C<060> in the
corpus C<WPD17> has the text sigle C<WPD17/060/18486>, see C<--sigle>).

These corpora are often stored in zip files, with which C<korapxml2krill>
can deal with. Corpora may also be split in multiple zip archives
(e.g. one zip file per foundry), which is also supported (see C<--input>).

Examples for KorAP-XML files are included in L<KorAP::XML::Krill>
in form of a test suite.
The resulting JSON format merges all annotation layers
based on a single token stream.

=head2 References

Piotr Bański, Cyril Belica, Helge Krause, Marc Kupietz, Carsten Schnober, Oliver Schonefeld, and Andreas Witt (2011):
KorAP data model: first approximation, December.

Piotr Bański, Peter M. Fischer, Elena Frick, Erik Ketzan, Marc Kupietz, Carsten Schnober, Oliver Schonefeld and Andreas Witt (2012):
"The New IDS Corpus Analysis Platform: Challenges and Prospects",
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012).
L<PDF|http://www.lrec-conf.org/proceedings/lrec2012/pdf/789_Paper.pdf>

Piotr Bański, Elena Frick, Michael Hanl, Marc Kupietz, Carsten Schnober and Andreas Witt (2013):
"Robust corpus architecture: a new look at virtual collections and data access",
Corpus Linguistics 2013. Abstract Book. Lancaster: UCREL, pp. 23-25.
L<PDF|https://ids-pub.bsz-bw.de/frontdoor/deliver/index/docId/4485/file/Ba%c5%84ski_Frick_Hanl_Robust_corpus_architecture_2013.pdf>

Kiyong Lee, Lou Burnard, Laurent Romary, Eric de la Clergerie, Thierry Declerck,
Syd Bauman, Harry Bunt, Lionel Clément, Tomaz Erjavec, Azim Roussanaly and Claude Roux (2004):
"Towards an international standard on featurestructure representation",
Proceedings of the fourth International Conference on Language Resources and Evaluation (LREC 2004),
pp. 373-376.
L<PDF|http://www.lrec-conf.org/proceedings/lrec2004/pdf/687.pdf>

Harald Lüngen and C. M. Sperberg-McQueen (2012):
"A TEI P5 Document Grammar for the IDS Text Model",
Journal of the Text Encoding Initiative, Issue 3 | November 2012.
L<PDF|https://journals.openedition.org/jtei/pdf/508>

TEI Consortium, eds:
"Feature Structures",
Guidelines for Electronic Text Encoding and Interchange.
L<html|https://www.tei-c.org/release/doc/tei-p5-doc/en/html/FS.html>

=head1 AVAILABILITY

  https://github.com/KorAP/KorAP-XML-Krill


=head1 COPYRIGHT AND LICENSE

Copyright (C) 2015-2025, L<IDS Mannheim|https://www.ids-mannheim.de/>

Author: L<Nils Diewald|https://www.nils-diewald.de/>

Contributor: Eliza Margaretha, Marc Kupietz

L<KorAP::XML::Krill> is developed as part of the L<KorAP|https://korap.ids-mannheim.de/>
Corpus Analysis Platform at the
L<Leibniz Institute for the German Language (IDS)|https://www.ids-mannheim.de/>,
member of the
L<Leibniz-Gemeinschaft|http://www.leibniz-gemeinschaft.de/>.

This program is free software published under the
L<BSD-2 License|https://opensource.org/licenses/BSD-2-Clause>.

=cut