Adopt tokenizer tests from KorAP-Tokenizer
1 file changed
tree: 60ce9e9c4a73e5c351be1d081bb217a8582ca5ae
  1. testdata/
  2. .gitignore
  3. datokenizer.go
  4. datokenizer_test.go
  5. go.mod
  6. go.sum