tree e3d073f62394c006cba70c22ffb6623191d5218d
parent 2173013a1ef865a231026ca4fd81f041732c769f
author Marc Kupietz <kupietz@ids-mannheim.de> 1770470837 +0100
committer Marc Kupietz <kupietz@ids-mannheim.de> 1770480229 +0100

Add de_old tokenizer variant without gender-sensitive rules

Introduces a new German tokenizer variant 'de_old' that treats
gender-sensitive forms, separted by `:`, `(`, and `/` as separate
tokens.

With de_old:
- Nutzer:in → Nutzer : in (3 tokens)
- Nutzer/innen → Nutzer / innen (3 tokens)
- Kaufmann/frau → Kaufmann / frau (3 tokens)
- Nutzer(in) → Nutzer ( in ) (4 tokens)

The standard 'de' tokenizer continues to keep these as single tokens.

Includes tests for the new variant.

Change-Id: I1d2af97e92a8af36ac3b04c2807bce8422b85df1