tree aae1c3781aa0e28b5ef3b134c5017637830e6952
parent e7751b807975e77757a63456aa89094f490b9dce
author Akron <nils@diewald-online.de> 1634598373 +0200
committer Akron <nils@diewald-online.de> 1634749567 +0200

Introduce token_writer object

This change also removes final state
sensibility from the tokenizer. Tokens
now require a tokenend transition to
be treated as complete.