Make tokenizer implementation exchangeable
Allows for quite an easy adaptation and integration of other language
specializations and other tokenizers, also from the command line with
new -T option that allows to specify the tokenizer class name.
TODO: add documentation to the KorapTokenizer interface
Change-Id: Ib95793f47887a3b0829d68ebdbf8bc40815a0605
diff --git a/.gitignore b/.gitignore
index b109df0..b5ddfd2 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,7 +1,7 @@
.*
!/.gitignore
target
-!/target/generated-sources/jflex/de/ids_mannheim/korap/tokenizer/KorAPTokenizerImpl.java
+!/target/generated-sources/jflex/de/ids_mannheim/korap/tokenizer/DerekoDfaTokenizer.java
tmp
logs
cache_store