Tokenizer playground

Models do not read words or letters. They read tokens. Type below to see exactly how text is split.

What to notice

• Short, common words are usually a single token. Rare or long words split into pieces.
• “strawberry” breaks into pieces, which is why counting its letters is harder for a model than it looks.
• Numbers, other languages and emoji often take more tokens than you would expect.

This uses an OpenAI-style tokenizer (cl100k). Other model families split text differently, but the principle is the same. The dots mark spaces, which belong to the token that follows them. The tool loads about half a megabyte the first time.