Gitiles
Code Review Sign In
korap.ids-mannheim.de / KorAP / KorAP-XML-TEI / 2115eccf4fbf0c1bfb7170235c4c7fb0b8280364 / t / tokenization.t
  1. 3bdc0a3 Improve Zipper to support root directory and fix bug in root directory normalization by Akron · 5 years ago
  2. 079f2bd Add utf-8 test for tokenization by Akron · 5 years ago
  3. 09e0b2c Establish collection object for token annotations by Akron · 5 years ago
  4. edee6e5 Make tokenization chainable and remove unnecessary tokenization switch by Akron · 5 years ago
  5. 42e18a6 allow to specify both tokenizations (extern and intern) by Peter Harders · 5 years ago
  6. 994aff7 faster processing of UTF8-chars by Peter Harders · 5 years ago
  7. 854a115 bugfixing Conservative.pm by Peter Harders · 5 years ago
  8. 1d65f94 extend tests for wikipedia.txt in t/tokenization.t by Peter Harders · 5 years ago
  9. 4c6ff5b Merge "Bugfix: intern tokenization" by Peter Harders · 5 years ago
  10. 57c884e remove temp. files from tests per default by Peter Harders · 5 years ago
  11. 71f072b Bugfix: intern tokenization by Peter Harders · 5 years ago
  12. 41c3562 changed comments, variable- and function-name(s) by Peter Harders · 6 years ago
  13. d962747 Establish tokenizer objects for aggressive and conservative base tokenization by Akron · 6 years ago
  14. 510a88c Minor speedup in tokenization by merging array pushes by Akron · 6 years ago
  15. eac374d Separate dummy tokenization from main script with minimal changes by Akron · 6 years ago