`tokenizer.ts` doesn't need a conditional import if you're using Vite's default bundler

[inspiredlabs]

tokenizer.ts doesn't need a conditional import if you're using Vite (npm run dev), but it is necessary when using (npm run build --debug && npm run preview).

So, why does tokenizer.ts require this conditional check to import GPT3TokenizerImport?

The code below (from the GitHub issue mentioned in the video), checks if you're using CommonJS or not.

const GPT3Tokenizer: typeof GPT3TokenizerImport =
 typeof GPT3TokenizerImport === 'function'
 ? GPT3TokenizerImport
 : (GPT3TokenizerImport as any).default

However – the following Just Works™️ when you're in npm run dev in Vite because it can handle mixed import syntax from ES6 module imports and CommonJS:

// $lib/tokenizer.ts
import GPT3Tokenizer from 'gpt3-tokenizer';

const tokenizer = new GPT3Tokenizer({ type: 'gpt3' });

export function getTokens(input: string): number {
  const tokens = tokenizer.encode(input);
  return tokens.text.length;
}

I hope this helps.

[inspiredlabs]

Rich Harris on the state of commonJS, see this: https://youtu.be/AOXq89h8saI?t=298

[inspiredlabs]

And this: https://youtu.be/xCeYmdukOKI?t=541