Changes:
- Add optional
toktokenizer (a faster but different option than nltk) (#10) - Fix naive vocab checking for tokens (#14)
- Allow lowercasing (#13)
List view
0 issues of 1 selected
- Status: Open (in progress).soaxelbrooke/python-bpenumber 11#11 In soaxelbrooke/python-bpe;