| commit | 5b8f67494d65ade35833d3770407cc78c2ec9234 | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Tue Sep 07 18:48:47 2021 +0200 |
| committer | Gerrit Code Review <gerrit2@korap.ids-mannheim.de> | Tue Sep 07 18:48:47 2021 +0200 |
| tree | 4aa4f9a0673c8c49025cf96d0c675b3d83aed3f9 | |
| parent | 210fc240a02434fa7c0e00365a6c0011b3f80c11 [diff] | |
| parent | b7084ce4caa7aba342df3ffa5734f64d12bd0978 [diff] |
Merge "Simplified FNAME rule"
Interface and implementation of a tokenizer and sentence splitter that can be used
The included implementations (DerekoDfaTokenizer_de, DerekoDfaTokenizer_en, DerekoDfaTokenizer_fr) are highly efficient DFA tokenizers and sentence splitters with character offset output based on JFlex. The de-variant is used for the German Reference Corpus DeReKo. Being based on finite state automata, the tokenizers are potentially not as accurate as language model based ones, but with ~5 billion words per hour typically more efficient. An important feature in the DeReKo/KorAP context is also that token character offsets can be reported, which can be used for applying standoff annotations.
The include mplementations of the KorapTokenizer interface also implement the opennlp.tools.tokenize.Tokenizer and opennlp.tools.sentdetect.SentenceDetector interfaces and can thus be used as a drop-in replacements in OpenNLP applications.
The underlying scanner is based on the Lucene scanner with modifications from David Hall.
Our changes mainly concern a good coverage of German, or optionally of some English and French abbreviations, and some updates for handling computer mediated communication, optimized and tested, in the case of German, against the gold data from the EmpiriST 2015 shared task (Beißwenger et al. 2016).
mvn clean install
Because of the large table of abbreviations, the conversion from the jflex source to java, i.e. the calculation of the DFA, takes about 5 to 30 minutes, depending on your hardware, and requires a lot of heap space.
By default, KorAP tokenizer reads from standard input and writes to standard output. It supports multiple modes of operations.
$ echo "It's working." | java -jar target/KorAP-Tokenizer-2.2.0.9000-standalone.jar -l en It 's working .
$ echo "C'est une phrase. Ici, il s'agit d'une deuxième phrase." \ | java -jar target/KorAP-Tokenizer-2.2.0.9000-standalone.jar -s -l fr C' est une phrase . Ici , il s' agit d' une deuxième phrase .
With the --positions option, for example, the tokenizer prints all offsets of the first character of a token and the first character after a token. In order to end a text, flush the output and reset the character position, an EOT character (0x04) can be used.
$ echo -n -e 'This is a text.\x0a\x04\x0aAnd this is another text.\n\x04\n' |\
java -jar target/KorAP-Tokenizer-2.2.0.9000-standalone.jar --positions
This
is
a
text
.
0 4 5 7 8 9 10 14 14 15
And
this
is
another
text
.
0 3 4 8 9 11 12 19 20 24 24 25
echo -n -e ' This ist a start of a text. And this is a sentence!!! But what the hack????\x0a\x04\x0aAnd this is another text.' |\ java -jar target/KorAP-Tokenizer-2.2.0.9000-standalone.jar --no-tokens --positions --sentence-boundaries 1 5 6 9 10 11 12 17 18 20 21 22 23 27 27 28 29 32 33 37 38 40 41 42 43 51 51 54 55 58 59 63 64 67 68 72 72 76 1 28 29 54 55 76 0 3 4 8 9 11 12 19 20 24 24 25 0 25
To adapt the included implementations to more languages, take one of the language-specific_<language>.jflex-macro files as template and modify for example the macro for abbreviations SEABBR. Then add an execution section for the new language to the jcp (java-comment-preprocessor) artifact in pom.xml following the example of one of the configurations there. After building the project (see below) your added language specific tokenizer / sentence splitter should be selectable with the --language option.
Alternatively, you can also provide KorAPTokenizer implementations independently on the class path and select them with the --tokenizer-class option.
Authors:
Copyright (c) 2021, Leibniz Institute for the German Language, Mannheim, Germany
This package is developed as part of the KorAP Corpus Analysis Platform at the Leibniz Institute for German Language (IDS).
The package contains code from Apache Lucene with modifications by Jim Hall.
It is published under the Apache 2.0 License.
Contributions are very welcome!
Your contributions should ideally be committed via our Gerrit server to facilitate reviewing (see Gerrit Code Review - A Quick Introduction if you are not familiar with Gerrit). However, we are also happy to accept comments and pull requests via GitHub.