tokenizer_tests

Playing around with anomalous tokens

To use this, you'll need to run:

pip install openai pip install tiktoken

I used Python 3.9.16 to run this code, though another version may work as well.

For the code to run, you'll need to make a new file called local_settings.py in the same directory as settings.py, and populate it with the following line:

OPENAI_SECRET_KEY = "[Your OpenAI secret key here]"