Gitiles
Code Review
Sign In
korap.ids-mannheim.de
/
KorAP
/
KorAP-Tokenizer
/
5c6df037afcdc7a149f84d9db50c6c88a5b278db
/
src
/
main
5c6df03
List available KorapTokenizerImplementation options for -T
by Marc Kupietz
· 4 years, 10 months ago
ce48102
Recognize {LETTER}+str. as abbreviation for Straße in de-tokenizer
by Marc Kupietz
· 4 years, 10 months ago
67eed1c
Build language specific tokenizers: de, en
by Marc Kupietz
· 4 years, 10 months ago
8e197f3
Allow setting input encoding explicitely whith -e <encoding>
by Marc Kupietz
· 4 years, 10 months ago
751868b
Make tokenizer implementation exchangeable
by Marc Kupietz
· 4 years, 10 months ago
b9f45e0
Rename tokenizer class to KorAPDFATokenizer
by Marc Kupietz
· 4 years, 11 months ago
c419d5b
Add new command line options using picocli and sanitize code
by Marc Kupietz
· 4 years, 11 months ago
783e2a2
Ignore quoted email names like "John Doe"@xx.com
by Marc Kupietz
· 5 years ago
571c194
Empty text (<EOT><EOT>) -> empty output line
by Marc Kupietz
· 5 years ago
6afd121
Use standard EOT/EOF character x04 instead of magic escape \n\x03\n
by Marc Kupietz
· 5 years ago
b2666fc
Implement sentence splitter
by Marc Kupietz
· 5 years ago
8192509
Use original Span class and implement Tokenizer interface from OpenNLP
by Marc Kupietz
· 5 years ago
478632e
Clean up code
by Marc Kupietz
· 5 years ago
45dc0fe
Add Apache copyright NOTICE
by Marc Kupietz
· 5 years ago
3367773
Initial import from private/Ingestion
by Marc Kupietz
· 5 years ago