Gitiles
Code Review Sign In
korap.ids-mannheim.de / KorAP / Datok / 96fdc9be352df941e7304b2e5ceeb8e2078d7307 / testdata / tokenizer.fst
  1. f6bdfdb Add trimming at the beginning of a text by Akron · 3 years, 2 months ago
  2. a854faa Introduce EOT (end-of-transmission) marker by Akron · 3 years, 2 months ago
  3. 4c2a1ad Introduce XML tests by Akron · 3 years, 4 months ago
  4. 235ea12 Update generated tokenizers by Akron · 3 years, 4 months ago
  5. e184a91 Add new generated automata by Akron · 3 years, 5 months ago
  6. 03c92fe Support for tokenend MCS symbol by Akron · 3 years, 5 months ago
  7. b4bbb47 Added sentence splitter capabilities by Akron · 3 years, 5 months ago
  8. c9d84a6 Sort alphabet prior to xCheck by Akron · 3 years, 5 months ago