tree e7ecabaf61abdc58b0bbec0a57a45a5015d4fb98
parent e955eccda68b3ecffd4d4433cdf75a6d0830d603
author Marc Kupietz <kupietz@ids-mannheim.de> 1613587172 +0100
committer Marc Kupietz <kupietz@ids-mannheim.de> 1613635002 +0100

Fix possible IO deadlocks with KorAP tokenizer

Text separators should always have a newline in front of artificial EOTs
to make sure they are recognized and to avoid them being consumed
by regular expressions for tokens.

Change-Id: I528c903904da50312a7472c7a34775476b0955be