Gitiles
Code Review
Sign In
korap.ids-mannheim.de
/
KorAP
/
Datok
/
a854faa43b6c1d9f023cff0b5433e485f0331aa3
/
src
/
tokenizer.xfst
a854faa
Introduce EOT (end-of-transmission) marker
by Akron
· 3 years, 1 month ago
4c2a1ad
Introduce XML tests
by Akron
· 3 years, 3 months ago
3de361e
Improved newline and abbreviation handling
by Akron
· 3 years, 3 months ago
57d0161
Add known terms with special characters
by Akron
· 3 years, 4 months ago
e8837b5
Add file scheme
by Akron
· 3 years, 4 months ago
fd92d7e
Update abbreviations according to KorAP-Tokenizer
by Akron
· 3 years, 4 months ago
a0bded5
Add ordinals
by Akron
· 3 years, 4 months ago
4af79f1
Added support for streetnames
by Akron
· 3 years, 4 months ago
310905f
Add foma sources
by Akron
· 3 years, 4 months ago