Gitiles
Code Review Sign In
korap.ids-mannheim.de / KorAP / KorAP-XML-TEI / 56b8dbd954b5d47a6e8e2f8d7776aba948ad2210 / t / cmd / tokenizer.pl
  1. 1e882fb Add -tk option to use the standard KoAP tokenizer by Marc Kupietz · 4 years, 3 months ago
  2. 52dc21b Send <EOT>\n to external tokenizer for separating texts by Marc Kupietz · 4 years, 3 months ago
  3. b122717 clean up intern tokenization by Peter Harders · 4 years, 4 months ago
  4. 8b511f9 Establish tokenizer object for external base tokenization by Akron · 4 years, 5 months ago