Fixed bug in tokenizer to recognize non-word-tokenizations

Change-Id: I4d9d5ffaefc45dc2220c17273dee70e05080137e
11 files changed
tree: 3bd3a29820b69cc5d86fc9c6f954739b3fcacd80
  1. lib/
  2. script/
  3. t/
  4. .gitignore
  5. Changes
  6. LICENSE
  7. Makefile.PL
  8. MANIFEST
  9. Readme.pod