Tokenizer
Also: tokenizers, tokenization
The tool that chops text into tokens and gives each piece a number. Different models chop differently.
“The contract is signed.”
becomes
“unbelievable”
becomes
Each piece is a token. The model never sees your words; it sees these pieces, each with its own number. Short common words are one piece; long or unusual words are cut into several. A page of text is roughly 500 to 700 tokens.
The fuller definition
A tokenizer converts text into tokens and numerical identifiers, and usually supports converting those identifiers back into text.
Go deeper
- Characters
Original text
- Segmentation
Find vocabulary pieces
- Encoding
Map pieces to token IDs
A simplified relationship, not a screenshot or a complete design.
A concrete example
A model's tokenizer may treat a leading space as part of the following token.
The distinction that matters
A word-count estimate is not an exact token count. Use the tokenizer for the intended model when precision matters.
Try it: words become pieces, and pieces have IDs
Choose an example and compare its coloured pieces. These results were computed in advance with OpenAI's tiktoken library using its o200k_base encoding, the one used by OpenAI's recent models. They are exact for that encoding; Claude, Gemini, and open-weight models each use their own tokenizer, so the pieces and counts differ.
5 tokens · 23 characters in this example
The ␠ marker makes a space visible. The numeric ID identifies a vocabulary entry; its magnitude is not a measure of meaning. A model converts these IDs into representations for computation.
Why a small edit can matter
Changing a word or punctuation can change the token boundaries and count. A model's context limit and API usage can depend on tokens, rather than the number of words a person sees.
What this does not show
This visual covers text tokenization. Images, audio, message formatting, and model-specific accounting can add other units or overhead. It is not a billing estimator.
Related terms
Token
A piece of text, often part of a word, that the model actually works with. Models read and write tokens, not whole words.
Documents & retrievalEmbedding
A list of numbers used to compare items such as texts. Texts with similar meanings may have similar lists.
FoundationsContext window
How much text the model can hold in mind at once: your question, the documents you added, and the conversation so far. Anything outside it is invisible to the model.
Where it is taught
What actually happens when you ask AI a question?Source and further reading
Google · Machine Learning GlossaryThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.