Gitiles
Code Review Sign In
korap.ids-mannheim.de / KorAP / KorAP-XML-TEI / 3b4dcb1cc6da343c6d92d37e4c3abedd66e9406f / t / cmd
  1. cfaefb9 Add missing tokenizer_faulty.pl by Marc Kupietz · 5 months ago
  2. 1e882fb Add -tk option to use the standard KoAP tokenizer by Marc Kupietz · 6 years ago
  3. 52dc21b Send <EOT>\n to external tokenizer for separating texts by Marc Kupietz · 6 years ago
  4. b122717 clean up intern tokenization by Peter Harders · 6 years ago
  5. 8b511f9 Establish tokenizer object for external base tokenization by Akron · 6 years ago