Skip to content

adamyedidia/tokenizer_tests

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

tokenizer_tests

Playing around with anomalous tokens

Run setup.sh and populate the created local_settings.py with your OpenaI API secret (don't worry, it's .gitignore'd)

I used Python 3.9.16 to run this code, though another version may work as well.

You can open an IPython shell with ./shell.py

About

Playing around with anomalous tokens

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •