tree efa93123637391ce7b6718a269495f9677717a64
parent b9f45e084ceca956b3378fe12d20d9d5b08edadc
author Marc Kupietz <kupietz@ids-mannheim.de> 1601049578 +0200
committer Marc Kupietz <kupietz@ids-mannheim.de> 1601276650 +0200

Make tokenizer implementation exchangeable

Allows for quite an easy adaptation and integration of other language
specializations and other tokenizers, also from the command line with
new -T option that allows to specify the tokenizer class name.

TODO: add documentation to the KorapTokenizer interface

Change-Id: Ib95793f47887a3b0829d68ebdbf8bc40815a0605