Improve documentation and support for external tokenizers

Change-Id: Ia65d4e9bcd2a28a7a77903dd49e2456dc566e7fe
3 files changed