- df27581 Make tokenizer robust and never failing by Akron · 2 years, 8 months ago
- e87906b Minor improvements by Akron · 3 years ago
- 4a6e0ff Fix newline after eot behaiour by Akron · 3 years, 1 month ago
- 274600e Fix buffer flushing to work with tei2korapxml by Akron · 3 years, 1 month ago
- 9fb63af Optimize tests by avoiding reload of tokenizers by Akron · 3 years, 1 month ago
- 7035d2e Fix sentence_pos handling by Akron · 3 years, 1 month ago
- 96fdc9b Fix TokenWriter regarding sentence boundaries and remove simple TokenWriter by Akron · 3 years, 1 month ago
- 0f087ea Parse command line options as bit flags by Akron · 3 years, 1 month ago
- fceddb6 Add sentence flags (for printing and offsets) by Akron · 3 years, 1 month ago
- a9e0c42 Introduce --[no]-tokens flag by Akron · 3 years, 1 month ago
- e9431ec Ignore newline after EOT with a flag by Akron · 3 years, 1 month ago
- 8cc2dd9 Fix buffer rewind at end of transmission by Akron · 3 years, 1 month ago
- 4f6b28c Support token offsets in token writer by Akron · 3 years, 1 month ago
- 32416ce Support offsets in token writer by Akron · 3 years, 1 month ago
- a854faa Introduce EOT (end-of-transmission) marker by Akron · 3 years, 1 month ago
- e396a93 Introduce token_writer object by Akron · 3 years, 1 month ago