Gitiles
Code Review Sign In
korap.ids-mannheim.de / KorAP / KorAP-Tokenizer / 4fb896a5bc95fcd2eb02a6d63ef8d471ca801431 / target
  1. 4fb896a Amend English abbreviation macro by Marc Kupietz · 4 years, 10 months ago
  2. 74141b3 Add -l command line option to choose language by Marc Kupietz · 4 years, 10 months ago
  3. ce48102 Recognize {LETTER}+str. as abbreviation for Straße in de-tokenizer by Marc Kupietz · 4 years, 10 months ago
  4. 67eed1c Build language specific tokenizers: de, en by Marc Kupietz · 4 years, 10 months ago
  5. 751868b Make tokenizer implementation exchangeable by Marc Kupietz · 4 years, 10 months ago
  6. b9f45e0 Rename tokenizer class to KorAPDFATokenizer by Marc Kupietz · 4 years, 11 months ago
  7. c419d5b Add new command line options using picocli and sanitize code by Marc Kupietz · 4 years, 11 months ago
  8. 783e2a2 Ignore quoted email names like "John Doe"@xx.com by Marc Kupietz · 4 years, 11 months ago
  9. 571c194 Empty text (<EOT><EOT>) -> empty output line by Marc Kupietz · 5 years ago
  10. 6afd121 Use standard EOT/EOF character x04 instead of magic escape \n\x03\n by Marc Kupietz · 5 years ago
  11. b2666fc Implement sentence splitter by Marc Kupietz · 5 years ago
  12. 8192509 Use original Span class and implement Tokenizer interface from OpenNLP by Marc Kupietz · 5 years ago
  13. 478632e Clean up code by Marc Kupietz · 5 years ago
  14. 3367773 Initial import from private/Ingestion by Marc Kupietz · 5 years ago