| commit | 52dc21b3b735d8f69e6e843c9a4b257e412aa9ee | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Sat Sep 05 13:51:22 2020 +0200 |
| committer | Peter Harders <harders@ids-mannheim.de> | Tue Sep 22 18:02:29 2020 +0200 |
| tree | 1b91a0caa47a320a51a00bd2654086c24c635614 | |
| parent | 59289de1b13d069ee2836a5acb1d1bf60e9eafbe [diff] |
Send <EOT>\n to external tokenizer for separating texts This is now the standard for the KorAP tokenizer. Change-Id: I30c2d6ca82211b1d312364899d4f56ea7908b4f8