#

byte-pair

Here are 2 public repositories matching this topic...

VenkatRamaraju / polyglot

a multilingual tokenizer written from scratch in go

tokenizer lllm byte-pair

Updated Sep 15, 2025
Go

99degree / gpu_assisted_bpe

Use my old buddy iGPU HD620 to do token matching.

gpu vulkan tokens tokenization utf-32 byte-pair-encoding byte-pair tokenizerbpe utf8-utf32

Updated Aug 19, 2025
C++

Improve this page

Add a description, image, and links to the byte-pair topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the byte-pair topic, visit your repo's landing page and select "manage topics."