Under the hood · interactive

How tokenization works

A language model never sees your letters. It sees tokens — and the rules for making them explain a surprising amount of model behaviour, from pricing to why it can't count the r's in "strawberry."

Before a single layer of a transformer runs, your text is chopped into tokens: chunks of characters the model treats as atomic units. It doesn't read character by character (sequences would be enormous) and it doesn't read whole words (the vocabulary would be infinite). It reads something in between, built by an algorithm called byte-pair encoding — BPE. The neat part is that BPE isn't hand-written. It's learned from a pile of text by one dumb, repeated rule: find the most common adjacent pair, merge it, repeat.

Why not just words, or just letters?

Two obvious ideas both fail. One token per word means an unbounded vocabulary — every typo, name, and compound is a new word the model has never seen, with nowhere to put it. One token per character gives a tiny fixed vocabulary but makes every sequence brutally long, and forces the model to relearn that t-h-e is a word from scratch, everywhere. BPE splits the difference: frequent words end up as one token, and rare ones fall back to a handful of subword pieces. Nothing is ever truly out-of-vocabulary, because the pieces bottom out at single characters.

The whole algorithm

You just watched it. Start with the text as individual characters. Count every adjacent pair; the pair that shows up most (say t then h) gets merged into a new token th. Do it again — now maybe th and e merge into the. Repeat a fixed number of times. The ordered list of merges is the tokenizer: to encode new text, you replay the same merges in the same order. That's it. Real tokenizers like GPT's run tens of thousands of these merges over billions of bytes — but the rule on the slider is exactly the rule they use.

Why it can't count the letters in "strawberry"

Watch what happens to a word the corpus never saw: it stays shattered into little pieces, while common words collapse to one token. Now flip it around — to the model, a common word is a single opaque token. It was never given the letters s-t-r-a-w-b-e-r-r-y; it was given one or two chunks. Asking it to count the r's is like asking you to count the strokes in a Chinese character you can recognise but can't write. The famous failure isn't stupidity — it's that the letters were folded away before the model ever looked.

Why your bill depends on this

APIs charge per token, and tokenization is why the same idea can cost wildly different amounts. English prose sits right in the tokenizer's training distribution, so it packs tightly — often close to one token per short word. Code, JSON, rare names, non-English scripts, and emoji sit outside it, so they fragment into many tokens each. The slider shows the mechanism in miniature: text the corpus has seen compresses; text it hasn't stays long. Same characters, very different token count — and a different number on your invoice.

One honest caveat: the toy above is trained live on that little corpus, lowercased, letters and spaces only. A production tokenizer works on raw bytes (so it handles any language, punctuation, and emoji), keeps the leading space as part of a token, and ships a fixed vocabulary of ~100k merges learned once over a huge corpus. The algorithm is identical — only the scale and the byte-level details differ.