How tokenization works
A language model never sees your letters. It sees tokens, which explains your bill and why it can't count the r's in "strawberry."
Models read text as tokens: chunks of characters treated as one unit. An algorithm called byte-pair encoding (BPE) learns them from a pile of text with one rule: merge the most common pair of neighbors, and repeat.
BPE, live
Drag the slider to run merges on a small corpus, then type something to see how it splits.
At 0 merges, every character is a token. More merges fuse common chunks.
Why not words or letters?
With one token per word, every typo and name is a new word the model
has never seen. With one per letter, sequences get very long and the model
has to learn t-h-e everywhere. With BPE, common words become one token and rare
ones break into pieces. Since the pieces go down to single characters, no word is unknown.
The whole algorithm
Start with single characters. Merge the most common adjacent pair, say t
and h into th. Then maybe th and e into
the. Repeat a set number of times. That ordered list of merges is the
tokenizer, and encoding new text replays it. GPT's tokenizer does the same thing with tens
of thousands of merges over billions of bytes.
Why it can't count the letters in "strawberry"
A common word becomes one or two tokens. The model never sees
s-t-r-a-w-b-e-r-r-y. Asking it to count the r's is like asking you to count
the strokes in a Chinese character you can read but can't write.
Why your bill depends on it
APIs charge per token. English prose packs tightly, often one token per short word. Code, JSON, rare names, other scripts and emoji break into many.
About the toy. It trains on a small corpus of lowercase letters and spaces. Production tokenizers work on raw bytes, so they handle any language and emoji. They also keep the leading space in a token and use about 100k merges learned once. The algorithm is the same.