| commit | eed4cb1874a5318860af348d1def96fc11d64a60 | [log] [tgz] |
|---|---|---|
| author | Marc Kupietz <kupietz@ids-mannheim.de> | Wed Feb 17 19:39:32 2021 +0100 |
| committer | Marc Kupietz <kupietz@ids-mannheim.de> | Thu Feb 18 08:56:42 2021 +0100 |
| tree | e7ecabaf61abdc58b0bbec0a57a45a5015d4fb98 | |
| parent | e955eccda68b3ecffd4d4433cdf75a6d0830d603 [diff] |
Fix possible IO deadlocks with KorAP tokenizer Text separators should always have a newline in front of artificial EOTs to make sure they are recognized and to avoid them being consumed by regular expressions for tokens. Change-Id: I528c903904da50312a7472c7a34775476b0955be