| commit | d962747a4ac7e02a6040fad736e4a8a45a6b4431 | [log] [tgz] |
|---|---|---|
| author | Akron <nils@diewald-online.de> | Thu Jul 09 16:53:09 2020 +0200 |
| committer | Akron <nils@diewald-online.de> | Thu Jul 09 17:01:06 2020 +0200 |
| tree | 9f39e305982ab20660aaa09b669b8accfe00ddba | |
| parent | 95612c3d5a2aeea3d81915b7a846a8dc69d46df4 [diff] |
Establish tokenizer objects for aggressive and conservative base tokenization Change-Id: I702098185b0b6292c73217268d4516d55a2f95b5