Gitiles
Code Review
Sign In
korap.ids-mannheim.de
/
KorAP
/
KorAP-XML-TEI
/
4ee372a14e0437e7810f3d7a3eebe78d959c3832
/
t
/
cmd
/
tokenizer.pl
1e882fb
Add -tk option to use the standard KoAP tokenizer
by Marc Kupietz
· 4 years, 3 months ago
52dc21b
Send <EOT>\n to external tokenizer for separating texts
by Marc Kupietz
· 4 years, 3 months ago
b122717
clean up intern tokenization
by Peter Harders
· 4 years, 5 months ago
8b511f9
Establish tokenizer object for external base tokenization
by Akron
· 4 years, 5 months ago