| =pod |
| |
| =encoding utf8 |
| |
| =head1 NAME |
| |
| korapxml2krill - Merge KorAP-XML data and create Krill documents |
| |
| |
| =head1 SYNOPSIS |
| |
| korapxml2krill [archive|extract] --input <directory|archive> [options] |
| |
| |
| =head1 DESCRIPTION |
| |
| L<KorAP::XML::Krill> is a library to convert KorAP-XML documents to files |
| compatible with the L<Krill|https://github.com/KorAP/Krill> indexer. |
| The C<korapxml2krill> command line tool is a simple wrapper of this library. |
| |
| |
| =head1 INSTALLATION |
| |
| The preferred way to install L<KorAP::XML::Krill> is to use L<cpanm|App::cpanminus>. |
| |
| $ cpanm https://github.com/KorAP/KorAP-XML-Krill.git |
| |
| In case everything went well, the C<korapxml2krill> tool will |
| be available on your command line immediately. |
| Minimum requirement for L<KorAP::XML::Krill> is Perl 5.32. |
| Optionally installing L<Archive::Tar::Builder> speeds up archive building. |
| Optional support for L<Sys::Info> to calculate available cores is available. |
| In addition to work with zip archives, the C<unzip> tool needs to be present. |
| |
| =head1 ARGUMENTS |
| |
| $ korapxml2krill -z --input <directory> --output <filename> |
| |
| Without arguments, C<korapxml2krill> converts a directory of a single KorAP-XML document. |
| It expects the input to point to the text level folder. |
| |
| =over 2 |
| |
| =item B<archive> |
| |
| $ korapxml2krill archive -z --input <directory|archive> --output <directory|tar> |
| |
| Converts an archive of KorAP-XML documents. It expects a directory |
| (pointing to the corpus level folder) or one or more zip files as input. |
| |
| =item B<extract> |
| |
| $ korapxml2krill extract --input <archive> --output <directory> --sigle <SIGLE> |
| |
| Extracts KorAP-XML documents from a zip file. |
| |
| =item B<serial> |
| |
| $ korapxml2krill serial -i <archive1> -i <archive2> -o <directory> -cfg <config-file> |
| |
| Convert archives sequentially. The inputs are not merged but treated |
| as they are (so they may be premerged or globs). |
| the C<--out> directory is treated as the base directory where subdirectories |
| are created based on the archive name. In case the C<--to-tar> flag is given, |
| the output will be a tar file. |
| |
| |
| =item B<slimlog> |
| |
| $ korapxml2krill slimlog <logfile> > <logfile-slim> |
| |
| Filters out all useless aka succesfull information from logs, to simplify |
| log checks. Expects no further options. |
| |
| |
| =back |
| |
| |
| =head1 OPTIONS |
| |
| =over 2 |
| |
| =item B<--input|-i> <directory|zip file> |
| |
| Directory or zip file(s) of documents to convert. |
| |
| Without arguments, C<korapxml2krill> expects a folder of a single KorAP-XML |
| document, while C<archive> expects a KorAP-XML corpus folder or a zip |
| file to batch process multiple files. |
| C<extract> expects zip files only. |
| |
| C<archive> supports multiple input zip files with the constraint, |
| that the first archive listed contains all primary data files |
| and all meta data files. |
| |
| -i file/news.zip -i file/news.malt.zip -i "#file/news.tt.zip" |
| |
| Input may also be defined using BSD glob wildcards. |
| |
| -i 'file/news*.zip' |
| |
| The extended input array will be sorted in length order, so the shortest |
| path needs to contain all primary data files and all meta data files. |
| |
| (The directory structure follows the base directory format, |
| that may include a C<.> root folder. |
| In this case further archives lacking a C<.> root folder |
| need to be passed with a hash sign in front of the archive's name. |
| This may require to quote the parameter.) |
| |
| To support zip files, a version of C<unzip> needs to be installed that is |
| compatible with the archive file. |
| |
| B<The root folder switch using the hash sign is experimental and |
| may vanish in future versions.> |
| |
| |
| =item B<--input-base|-ib> <directory> |
| |
| The base directory for inputs. |
| |
| |
| =item B<--output|-o> <directory|file> |
| |
| Output folder for archive processing or |
| document name for single output (optional), |
| writes to C<STDOUT> by default |
| (in case C<output> is not mandatory due to further options). |
| |
| =item B<--overwrite|-w> |
| |
| Overwrite files that already exist. |
| |
| |
| =item B<--token|-t> <foundry>#<file> |
| |
| Define the default tokenization by specifying |
| the name of the foundry and optionally the name |
| of the layer-file. Defaults to C<OpenNLP#tokens>. |
| This will directly take the file instead of running |
| the layer implementation! |
| |
| |
| =item B<--base-sentences|-bs> <foundry>#<layer> |
| |
| Define the layer for base sentences. |
| If given, this will be used instead of using C<Base#Sentences>. |
| Currently C<DeReKo#Structure> and C<DGD#Structure> are the only additional |
| layers supported. |
| |
| Defaults to unset. |
| |
| |
| =item B<--base-paragraphs|-bp> <foundry>#<layer> |
| |
| Define the layer for base paragraphs. |
| If given, this will be used instead of using C<Base#Paragraphs>. |
| Currently C<DeReKo#Structure> and C<DGD#Structure> are the only additional |
| layer supported. |
| |
| Defaults to unset. |
| |
| |
| =item B<--base-pagebreaks|-bpb> <foundry>#<layer> |
| |
| Define the layer for base pagebreaks. |
| Currently C<DeReKo#Structure> is the only layer supported. |
| |
| Defaults to unset. |
| |
| |
| =item B<--skip|-s> <foundry>[#<layer>] |
| |
| Skip specific annotations by specifying the foundry |
| (and optionally the layer with a C<#>-prefix), |
| e.g. C<Mate> or C<Mate#Morpho>. Alternatively you can skip C<#ALL>. |
| Can be set multiple times. |
| |
| |
| =item B<--anno|-a> <foundry>#<layer> |
| |
| Convert specific annotations by specifying the foundry |
| (and optionally the layer with a C<#>-prefix), |
| e.g. C<Mate> or C<Mate#Morpho>. |
| Can be set multiple times. |
| |
| |
| =item B<--non-word-tokens|-nwt> |
| |
| Tokenize non-word tokens like word tokens (defined as matching |
| C</[\d\w]/>). Useful to treat punctuations as tokens. |
| |
| Defaults to unset. |
| |
| |
| =item B<--non-verbal-tokens|-nvt> |
| |
| Tokenize non-verbal tokens marked as in the primary data as |
| the unicode symbol 'Black Vertical Rectangle' aka \x25ae. |
| |
| Defaults to unset. |
| |
| |
| =item B<--jobs|-j> |
| |
| Define the number of concurrent jobs in seperated forks |
| for archive processing. |
| Defaults to C<0> (everything runs in a single process). |
| |
| If C<sequential-extraction> is not set to true, this will |
| also apply to extraction. |
| |
| Pass -1, and the value will be set automatically to 5 |
| times the number of available cores, in case L<Sys::Info> |
| is available. |
| This is I<experimental>. |
| |
| |
| =item B<--koral|-k> |
| |
| Version of the output format. Supported versions are: |
| C<0> for legacy serialization, C<0.03> for serialization |
| with metadata fields as key-values on the root object, |
| C<0.4> for serialization with metadata fields as a list |
| of C<"@type":"koral:field"> objects. |
| |
| Currently defaults to C<0.03>. |
| |
| |
| =item B<--sequential-extraction|-se> |
| |
| Flag to indicate, if the C<jobs> value also applies to extraction. |
| Some systems may have problems with extracting multiple archives |
| to the same folder at the same time. |
| Can be flagged using C<--no-sequential-extraction> as well. |
| Defaults to C<false>. |
| |
| |
| =item B<--meta|-m> |
| |
| Define the metadata parser to use. Defaults to C<I5>. |
| Metadata parsers can be defined in the C<KorAP::XML::Meta> namespace. |
| This is I<experimental>. |
| |
| |
| =item B<--gzip|-z> |
| |
| Compress the output. |
| Expects a defined C<output> file in single processing. |
| |
| |
| =item B<--cache|-c> |
| |
| File to mmap a cache (using L<Cache::FastMmap>). |
| Defaults to C<korapxml2krill.cache> in the calling directory. |
| |
| |
| =item B<--cache-size|-cs> |
| |
| Size of the cache. Defaults to C<50m>. |
| |
| |
| =item B<--cache-init|-ci> |
| |
| Initialize cache file. |
| Can be flagged using C<--no-cache-init> as well. |
| Defaults to C<true>. |
| |
| |
| =item B<--cache-delete|-cd> |
| |
| Delete cache file after processing. |
| Can be flagged using C<--no-cache-delete> as well. |
| Defaults to C<true>. |
| |
| |
| =item B<--config|-cfg> |
| |
| Configure the parameters of your call in a file |
| of key-value pairs with whitespace separator |
| |
| overwrite 1 |
| token DeReKo#Structure |
| ... |
| |
| Supported parameters are: |
| C<overwrite>, C<gzip>, C<jobs>, C<input-base>, |
| C<token>, C<log>, C<cache>, C<cache-size>, C<cache-delete>, C<meta>, |
| C<output>, C<koral>, |
| C<temporary-extract>, C<sequential-extraction>, |
| C<base-sentences>, C<base-paragraphs>, |
| C<base-pagebreaks>, |
| C<skip> (semicolon separated), C<sigle> |
| (semicolon separated), C<anno> (semicolon separated). |
| |
| Configuration parameters will always be overwritten by |
| passed parameters. |
| |
| |
| =item B<--temporary-extract|-te> |
| |
| Only valid for the C<archive> and C<serial> |
| commands. |
| |
| This will first extract all files into a |
| directory and then will archive. |
| If the directory is given as C<:temp:>, |
| a temporary directory is used. |
| This is especially useful to avoid |
| massive unzipping and potential |
| network latency. |
| |
| |
| =item B<--to-tar> |
| |
| Only valid for the C<archive> command. |
| |
| Writes the output into a tar archive. |
| |
| |
| =item B<--sigle|-sg> |
| |
| Extract the given texts. |
| Can be set multiple times. |
| I<Currently only supported on C<extract>.> |
| Sigles have the structure C<Corpus>/C<Document>/C<Text>. |
| In case the C<Text> path is omitted, the whole document will be extracted. |
| On the document level, the postfix wildcard C<*> is supported. |
| |
| =item B<--lang> |
| |
| Preferred language for metadata fields. In case multiple titles are |
| given (on any level) with different C<xml:lang> attributes, |
| the language given is preferred. |
| Because titles may have different sources and different priorities, |
| non-specific language titles may still be preferred in case the title |
| source has a higher priority. |
| |
| |
| =item B<--log|-l> |
| |
| The L<Log::Any> log level, defaults to C<ERROR>. |
| |
| |
| =item B<--help|-h> |
| |
| Print help information. |
| |
| |
| =item B<--version|-v> |
| |
| Print version information. |
| |
| =back |
| |
| |
| =head1 ANNOTATION SUPPORT |
| |
| L<KorAP::XML::Krill> has built-in importer for some annotation foundries and layers |
| developed in the KorAP project that are part of the KorAP preprocessing pipeline. |
| The base foundry with paragraphs, sentences, and the text element are mandatory for |
| L<Krill|https://github.com/KorAP/Krill>. |
| |
| Base |
| #Paragraphs |
| #Sentences |
| |
| Connexor |
| #Morpho |
| #Phrase |
| #Sentences |
| #Syntax |
| |
| CoreNLP |
| #Constituency |
| #Morpho |
| #NamedEntities |
| #Sentences |
| |
| CMC |
| #Morpho |
| |
| DeReKo |
| #Structure |
| |
| DGD |
| #Morpho |
| #Structure |
| |
| DRuKoLa |
| #Morpho |
| |
| Glemm |
| #Morpho |
| |
| Gingko |
| #Morpho |
| |
| HNC |
| #Morpho |
| |
| LWC |
| #Dependency |
| |
| Malt |
| #Dependency |
| |
| MarMoT |
| #Morpho |
| |
| Mate |
| #Dependency |
| #Morpho |
| |
| MDParser |
| #Dependency |
| |
| NKJP |
| #Morpho |
| #NamedEntities |
| |
| OpenNLP |
| #Morpho |
| #Sentences |
| |
| RWK |
| #Morpho |
| #Structure |
| |
| Sgbr |
| #Lemma |
| #Morpho |
| |
| Talismane |
| #Dependency |
| #Morpho |
| |
| TreeTagger |
| #Morpho |
| #Sentences |
| |
| UDPipe |
| #Dependency |
| #Morpho |
| |
| XIP |
| #Constituency |
| #Morpho |
| #Sentences |
| |
| |
| More importers are in preparation. |
| New annotation importers can be defined in the C<KorAP::XML::Annotation> namespace. |
| See the built-in annotation importers as examples. |
| |
| |
| =head1 METADATA SUPPORT |
| |
| L<KorAP::XML::Krill> has built-in importer for some meta data variants |
| developed in the KorAP project that are part of the KorAP preprocessing pipeline. |
| |
| =over 2 |
| |
| =item I5 - Meta data for all I5 files |
| |
| =item Sgbr - Meta data from the Schreibgebrauch project |
| |
| =item Gingko - Meta data from the Gingko project in addition to I5 |
| |
| =item ICC - Meta data for the ICC in addition to I5 |
| |
| =back |
| |
| More importers are in preparation. |
| New meta data importers can be defined in the C<KorAP::XML::Meta> namespace. |
| See the built-in meta data importers as examples. |
| |
| |
| =head1 About KorAP-XML |
| |
| KorAP-XML (Banski et al. 2012) is an implementation of the KorAP |
| data model (Banski et al. 2013), where text data are stored physically |
| separated from their interpretations (i.e. annotations). |
| A text document in KorAP-XML therefore consists of several files |
| containing primary data, metadata and annotations. |
| |
| The structure of a single KorAP-XML document can be as follows: |
| |
| - data.xml |
| - header.xml |
| + base |
| - tokens.xml |
| - ... |
| + struct |
| - structure.xml |
| - ... |
| + corenlp |
| - morpho.xml |
| - constituency.xml |
| - ... |
| + tree_tagger |
| - morpho.xml |
| - ... |
| - ... |
| |
| The C<data.xml> contains the primary data, the C<header.xml> contains |
| the metadata, and the annotation layers are stored in subfolders |
| like C<base>, C<struct> or C<corenlp> |
| (so-called "foundries"; Banski et al. 2013). |
| |
| Metadata is available in the TEI-P5 variant I5 |
| (Lüngen and Sperberg-McQueen 2012). See the documentation in |
| L<KorAP::XML::Meta::I5> for translatable fields. |
| |
| Annotations correspond to a variant of the TEI-P5 feature structures |
| (TEI Consortium; Lee et al. 2004). |
| Annotation feature structures refer to character sequences of the primary text |
| inside the C<text> element of the C<data.xml>. |
| A single annotation containing the lemma of a token can have the following structure: |
| |
| <span from="0" to="3"> |
| <fs type="lex" xmlns="http://www.tei-c.org/ns/1.0"> |
| <f name="lex"> |
| <fs> |
| <f name="lemma">zum</f> |
| </fs> |
| </f> |
| </fs> |
| </span> |
| |
| The C<from> and C<to> attributes are refering to the character span |
| in the primary text. |
| Depending on the kind of annotation (e.g. token-based, span-based, relation-based), |
| the structure may vary. See L<KorAP::XML::Annotation::*> for various |
| annotation preprocessors. |
| |
| Multiple KorAP-XML documents are organized on three levels following |
| the "IDS Textmodell" (Lüngen and Sperberg-McQueen 2012): |
| corpus E<gt> document E<gt> text. On each level metadata information |
| can be stored, that C<korapxml2krill> will merge to a single metadata |
| object per text. A corpus is therefore structured as follows: |
| |
| + <corpus> |
| - header.xml |
| + <document> |
| - header.xml |
| + <text> |
| - data.xml |
| - header.xml |
| - ... |
| - ... |
| |
| A single text can be identified by the concatenation of |
| the corpus identifier, the document identifier and the text identifier. |
| This identifier is called the text sigle |
| (e.g. a text with the identifier C<18486> in the document C<060> in the |
| corpus C<WPD17> has the text sigle C<WPD17/060/18486>, see C<--sigle>). |
| |
| These corpora are often stored in zip files, with which C<korapxml2krill> |
| can deal with. Corpora may also be split in multiple zip archives |
| (e.g. one zip file per foundry), which is also supported (see C<--input>). |
| |
| Examples for KorAP-XML files are included in L<KorAP::XML::Krill> |
| in form of a test suite. |
| The resulting JSON format merges all annotation layers |
| based on a single token stream. |
| |
| =head2 References |
| |
| Piotr Banski, Cyril Belica, Helge Krause, Marc Kupietz, Carsten Schnober, Oliver Schonefeld, and Andreas Witt (2011): |
| KorAP data model: first approximation, December. |
| |
| Piotr Banski, Peter M. Fischer, Elena Frick, Erik Ketzan, Marc Kupietz, Carsten Schnober, Oliver Schonefeld and Andreas Witt (2012): |
| "The New IDS Corpus Analysis Platform: Challenges and Prospects", |
| Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012). |
| L<PDF|http://www.lrec-conf.org/proceedings/lrec2012/pdf/789_Paper.pdf> |
| |
| Piotr Banski, Elena Frick, Michael Hanl, Marc Kupietz, Carsten Schnober and Andreas Witt (2013): |
| "Robust corpus architecture: a new look at virtual collections and data access", |
| Corpus Linguistics 2013. Abstract Book. Lancaster: UCREL, pp. 23-25. |
| L<PDF|https://ids-pub.bsz-bw.de/frontdoor/deliver/index/docId/4485/file/Ba%c5%84ski_Frick_Hanl_Robust_corpus_architecture_2013.pdf> |
| |
| Kiyong Lee, Lou Burnard, Laurent Romary, Eric de la Clergerie, Thierry Declerck, |
| Syd Bauman, Harry Bunt, Lionel Clément, Tomaz Erjavec, Azim Roussanaly and Claude Roux (2004): |
| "Towards an international standard on featurestructure representation", |
| Proceedings of the fourth International Conference on Language Resources and Evaluation (LREC 2004), |
| pp. 373-376. |
| L<PDF|http://www.lrec-conf.org/proceedings/lrec2004/pdf/687.pdf> |
| |
| Harald Lüngen and C. M. Sperberg-McQueen (2012): |
| "A TEI P5 Document Grammar for the IDS Text Model", |
| Journal of the Text Encoding Initiative, Issue 3 | November 2012. |
| L<PDF|https://journals.openedition.org/jtei/pdf/508> |
| |
| TEI Consortium, eds: |
| "Feature Structures", |
| Guidelines for Electronic Text Encoding and Interchange. |
| L<html|https://www.tei-c.org/release/doc/tei-p5-doc/en/html/FS.html> |
| |
| =head1 AVAILABILITY |
| |
| https://github.com/KorAP/KorAP-XML-Krill |
| |
| |
| =head1 COPYRIGHT AND LICENSE |
| |
| Copyright (C) 2015-2023, L<IDS Mannheim|https://www.ids-mannheim.de/> |
| |
| Author: L<Nils Diewald|https://www.nils-diewald.de/> |
| |
| Contributor: Eliza Margaretha |
| |
| L<KorAP::XML::Krill> is developed as part of the L<KorAP|https://korap.ids-mannheim.de/> |
| Corpus Analysis Platform at the |
| L<Leibniz Institute for the German Language (IDS)|https://www.ids-mannheim.de/>, |
| member of the |
| L<Leibniz-Gemeinschaft|http://www.leibniz-gemeinschaft.de/>. |
| |
| This program is free software published under the |
| L<BSD-2 License|https://opensource.org/licenses/BSD-2-Clause>. |
| |
| =cut |