Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 1 | =pod |
| 2 | |
| 3 | =encoding utf8 |
| 4 | |
| 5 | =head1 NAME |
| 6 | |
| 7 | tei2korapxml - Conversion of TEI P5 based formats to KorAP-XML |
| 8 | |
| 9 | =head1 SYNOPSIS |
| 10 | |
Akron | 1148478 | 2021-11-03 20:12:14 +0100 | [diff] [blame] | 11 | cat corpus.i5.xml | tei2korapxml - > corpus.korapxml.zip |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 12 | |
| 13 | =head1 DESCRIPTION |
| 14 | |
| 15 | C<tei2korapxml> is a script to convert TEI P5 and |
Akron | d72baca | 2021-07-23 13:25:32 +0200 | [diff] [blame] | 16 | L<I5|https://www.ids-mannheim.de/digspra/kl/projekte/korpora/textmodell> |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 17 | based documents to the |
| 18 | L<KorAP-XML format|https://github.com/KorAP/KorAP-XML-Krill#about-korap-xml>. |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 19 | |
| 20 | This program is usually called from inside another script. |
| 21 | |
| 22 | =head1 FORMATS |
| 23 | |
| 24 | =head2 Input restrictions |
| 25 | |
| 26 | =over 2 |
| 27 | |
| 28 | =item |
| 29 | |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 30 | TEI P5 formatted input with certain restrictions: |
| 31 | |
| 32 | =over 4 |
| 33 | |
| 34 | =item |
| 35 | |
| 36 | B<mandatory>: text-header with integrated textsigle, text-body |
| 37 | |
| 38 | =item |
| 39 | |
| 40 | B<optional>: corp-header with integrated corpsigle, |
| 41 | doc-header with integrated docsigle |
| 42 | |
| 43 | =back |
| 44 | |
| 45 | =item |
| 46 | |
| 47 | All tokens inside the primary text may not be |
| 48 | newline seperated, because newlines are removed |
| 49 | (see L<KorAP::XML::TEI::Data>) and a conversion of newlines |
| 50 | into blanks between 2 tokens could lead to additional blanks, |
| 51 | where there should be none (e.g.: punctuation characters like C<,> or |
| 52 | C<.> should not be seperated from their predecessor token). |
Akron | 8a0c4bf | 2021-03-16 16:51:21 +0100 | [diff] [blame] | 53 | (see also code section C<~ whitespace handling ~> in C<script/tei2korapxml>). |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 54 | |
Akron | 940ca6f | 2021-10-11 12:38:39 +0200 | [diff] [blame] | 55 | =item |
| 56 | |
| 57 | Header types, like C<E<lt>idsHeader [...] type="document" [...] E<gt>> |
| 58 | need to be defined in the same line as the header tag. |
| 59 | |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 60 | =back |
| 61 | |
| 62 | =head2 Notes on the output |
| 63 | |
| 64 | =over 2 |
| 65 | |
| 66 | =item |
| 67 | |
| 68 | zip file output (default on C<stdout>) with utf8 encoded entries |
| 69 | (which together form the KorAP-XML format) |
| 70 | |
| 71 | =back |
| 72 | |
| 73 | =head1 INSTALLATION |
| 74 | |
Marc Kupietz | e83a4e9 | 2021-03-16 20:51:26 +0100 | [diff] [blame] | 75 | C<tei2korapxml> requires L<libxml2-dev> bindings and L<File::ShareDir::Install> to be installed. |
| 76 | When these requirements are met, the preferred way to install the script is |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 77 | to use L<cpanm|App::cpanminus>. |
| 78 | |
| 79 | $ cpanm https://github.com/KorAP/KorAP-XML-TEI.git |
| 80 | |
| 81 | In case everything went well, the C<tei2korapxml> tool will |
| 82 | be available on your command line immediately. |
| 83 | |
| 84 | Minimum requirement for L<KorAP::XML::TEI> is Perl 5.16. |
| 85 | |
| 86 | =head1 OPTIONS |
| 87 | |
| 88 | =over 2 |
| 89 | |
Akron | 1148478 | 2021-11-03 20:12:14 +0100 | [diff] [blame] | 90 | =item B<--input|-i> |
| 91 | |
| 92 | The input file to process. If no specific input is defined and a single |
| 93 | dash C<-> is passed as an argument, data is read from C<STDIN>. |
| 94 | |
| 95 | |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 96 | =item B<--root|-r> |
| 97 | |
| 98 | The root directory for output. Defaults to C<.>. |
| 99 | |
| 100 | =item B<--help|-h> |
| 101 | |
| 102 | Print help information. |
| 103 | |
| 104 | =item B<--version|-v> |
| 105 | |
| 106 | Print version information. |
| 107 | |
| 108 | =item B<--tokenizer-call|-tc> |
| 109 | |
| 110 | Call an external tokenizer process, that will tokenize |
Akron | 1148478 | 2021-11-03 20:12:14 +0100 | [diff] [blame] | 111 | from STDIN and outputs the offsets of all tokens. |
| 112 | |
| 113 | Texts are separated using C<\x04\n>. The external process |
| 114 | should add a new line per text. |
| 115 | |
| 116 | If the L</--use-tokenizer-sentence-splits> option is activated, |
| 117 | sentences are marked by offset as well in new lines. |
| 118 | |
| 119 | To use L<Datok|https://github.com/KorAP/Datok> including sentence |
| 120 | splitting, call C<tei2korap> as follows: |
| 121 | |
| 122 | $ cat corpus.i5.xml | tei2korapxml -s \ |
| 123 | $ -tc 'datok tokenize \ |
| 124 | $ -t ./tokenizer.matok \ |
| 125 | $ -p --newline-after-eot --no-sentences \ |
| 126 | $ --no-tokens --sentence-positions -' - \ |
| 127 | $ > corpus.korapxml.zip |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 128 | |
| 129 | =item B<--tokenizer-korap|-tk> |
| 130 | |
| 131 | Use the standard KorAP/DeReKo tokenizer. |
| 132 | |
Akron | 6d7b8e4 | 2020-09-29 07:37:41 +0200 | [diff] [blame] | 133 | =item B<--tokenizer-internal|-ti> |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 134 | |
| 135 | Tokenize the data using two embedded tokenizers, |
| 136 | that will take an I<Aggressive> and a I<conservative> |
| 137 | approach. |
| 138 | |
Akron | 75d6314 | 2021-02-23 18:40:56 +0100 | [diff] [blame] | 139 | =item B<--skip-inline-tokens> |
| 140 | |
| 141 | Boolean flag indicating that inline tokens should not |
| 142 | be processed. Defaults to false (meaning inline tokens will be processed). |
| 143 | |
Akron | 692d17d | 2021-03-05 13:21:03 +0100 | [diff] [blame] | 144 | =item B<--skip-inline-token-annotations> |
| 145 | |
| 146 | Boolean flag indicating that inline token annotations should not |
| 147 | be processed. Defaults to true (meaning inline token annotations |
| 148 | won't be processed). |
| 149 | |
Akron | ca70a1d | 2021-02-25 16:21:31 +0100 | [diff] [blame] | 150 | =item B<--skip-inline-tags> <tags> |
Akron | 54c3ff1 | 2021-02-25 11:33:37 +0100 | [diff] [blame] | 151 | |
| 152 | Expects a comma-separated list of tags to be ignored when the structure |
| 153 | is parsed. Content of these tags however will be processed. |
| 154 | |
Akron | 1a5271a | 2021-02-18 13:18:15 +0100 | [diff] [blame] | 155 | =item B<--inline-tokens> <foundry>#[<file>] |
| 156 | |
| 157 | Define the foundry and file (without extension) |
| 158 | to store inline token information in. |
Akron | 8a0c4bf | 2021-03-16 16:51:21 +0100 | [diff] [blame] | 159 | Unless C<--skip-inline-token-annotations> is set, |
| 160 | this will contain annotations as well. |
Akron | 1a5271a | 2021-02-18 13:18:15 +0100 | [diff] [blame] | 161 | Defaults to C<tokens> and C<morpho>. |
| 162 | |
Akron | e2819a1 | 2021-10-12 15:52:55 +0200 | [diff] [blame] | 163 | The inline token data will also be stored in the |
| 164 | inline structures file (see I<--inline-structures>), |
| 165 | unless the inline token foundry is prepended |
| 166 | by an B<!> exclamation mark, indicating that inline |
| 167 | tokens are stored exclusively in the inline tokens |
| 168 | file. |
| 169 | |
| 170 | Example: |
| 171 | |
| 172 | tei2korapxml --inline-tokens '!gingko#morpho' < data.i5.xml > korapxml.zip |
| 173 | |
Akron | dd0be8f | 2021-02-18 19:29:41 +0100 | [diff] [blame] | 174 | =item B<--inline-structures> <foundry>#[<file>] |
| 175 | |
| 176 | Define the foundry and file (without extension) |
| 177 | to store inline structure information in. |
| 178 | Defaults to C<struct> and C<structures>. |
Akron | 75d6314 | 2021-02-23 18:40:56 +0100 | [diff] [blame] | 179 | |
Akron | 26a7152 | 2021-02-19 10:27:37 +0100 | [diff] [blame] | 180 | =item B<--base-foundry> <foundry> |
| 181 | |
| 182 | Define the base foundry to store newly generated |
| 183 | token information in. |
| 184 | Defaults to C<base>. |
| 185 | |
| 186 | =item B<--data-file> <file> |
| 187 | |
| 188 | Define the file (without extension) |
| 189 | to store primary data information in. |
| 190 | Defaults to C<data>. |
| 191 | |
| 192 | =item B<--header-file> <file> |
| 193 | |
| 194 | Define the file name (without extension) |
| 195 | to store header information on |
| 196 | the corpus, document, and text level in. |
| 197 | Defaults to C<header>. |
Akron | dd0be8f | 2021-02-18 19:29:41 +0100 | [diff] [blame] | 198 | |
Marc Kupietz | 985da0c | 2021-02-15 19:29:50 +0100 | [diff] [blame] | 199 | =item B<--use-tokenizer-sentence-splits|-s> |
| 200 | |
| 201 | Replace existing with, or add new, sentence boundary information |
Akron | 1148478 | 2021-11-03 20:12:14 +0100 | [diff] [blame] | 202 | provided by the tokenizer. |
| 203 | Currently KorAP-tokenizer and certain external tokenizers support |
| 204 | these boundaries. |
Marc Kupietz | 985da0c | 2021-02-15 19:29:50 +0100 | [diff] [blame] | 205 | |
Akron | 91705d7 | 2021-02-19 10:59:45 +0100 | [diff] [blame] | 206 | =item B<--tokens-file> <file> |
| 207 | |
| 208 | Define the file (without extension) |
| 209 | to store generated token information in |
| 210 | (either from the KorAP tokenizer or an externally called tokenizer). |
| 211 | Defaults to C<tokens>. |
| 212 | |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 213 | =item B<--log|-l> |
| 214 | |
| 215 | Loglevel for I<Log::Any>. Defaults to C<notice>. |
| 216 | |
| 217 | =back |
| 218 | |
Akron | b364947 | 2020-09-29 08:24:46 +0200 | [diff] [blame] | 219 | =head1 ENVIRONMENT VARIABLES |
| 220 | |
| 221 | =over 2 |
| 222 | |
| 223 | =item B<KORAPXMLTEI_DEBUG> |
| 224 | |
| 225 | Activate minimal debugging. |
| 226 | Defaults to C<false>. |
| 227 | |
Akron | b364947 | 2020-09-29 08:24:46 +0200 | [diff] [blame] | 228 | =back |
| 229 | |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 230 | =head1 COPYRIGHT AND LICENSE |
| 231 | |
Marc Kupietz | 985da0c | 2021-02-15 19:29:50 +0100 | [diff] [blame] | 232 | Copyright (C) 2021, L<IDS Mannheim|https://www.ids-mannheim.de/> |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 233 | |
| 234 | Author: Peter Harders |
| 235 | |
Akron | aabd095 | 2020-09-29 07:35:08 +0200 | [diff] [blame] | 236 | Contributors: Nils Diewald, Marc Kupietz, Carsten Schnober |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 237 | |
| 238 | L<KorAP::XML::TEI> is developed as part of the L<KorAP|https://korap.ids-mannheim.de/> |
| 239 | Corpus Analysis Platform at the |
Akron | d72baca | 2021-07-23 13:25:32 +0200 | [diff] [blame] | 240 | L<Leibniz Institute for the German Language (IDS)|https://www.ids-mannheim.de/>, |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 241 | member of the |
| 242 | L<Leibniz-Gemeinschaft|http://www.leibniz-gemeinschaft.de/>. |
| 243 | |
| 244 | This program is free software published under the |
Marc Kupietz | e955ecc | 2021-02-17 17:42:01 +0100 | [diff] [blame] | 245 | L<BSD-2 License|https://opensource.org/licenses/BSD-2-Clause>. |
Akron | 0c41ab3 | 2020-09-29 07:33:33 +0200 | [diff] [blame] | 246 | |
Akron | 692d17d | 2021-03-05 13:21:03 +0100 | [diff] [blame] | 247 | =cut |