Token
Also: tokens
A piece of text, often part of a word, that the model actually works with. Models read and write tokens, not whole words.
“The contract is signed.”
becomes
“unbelievable”
becomes
Each piece is a token. The model never sees your words; it sees these pieces, each with its own number. Short common words are one piece; long or unusual words are cut into several. A page of text is roughly 500 to 700 tokens.
The fuller definition
A token is a unit in the numerical sequence a language model processes. It can represent a whole word, part of a word, punctuation, or other text, depending on the tokenizer.
Go deeper
- Text
A sentence
- Pieces
Tokenizer-specific segments
- IDs
Numbers supplied to the model
A simplified relationship, not a screenshot or a complete design.
A concrete example
One tokenizer may divide “unbelievable” into several pieces while representing a common word with one token.
The distinction that matters
Tokens are not universally words. Different tokenizers, languages, and formatting produce different counts.
Try it: words become pieces, and pieces have IDs
Choose an example and compare its coloured pieces. These results were computed in advance with OpenAI's tiktoken library using its o200k_base encoding, the one used by OpenAI's recent models. They are exact for that encoding; Claude, Gemini, and open-weight models each use their own tokenizer, so the pieces and counts differ.
5 tokens · 23 characters in this example
The ␠ marker makes a space visible. The numeric ID identifies a vocabulary entry; its magnitude is not a measure of meaning. A model converts these IDs into representations for computation.
Why a small edit can matter
Changing a word or punctuation can change the token boundaries and count. A model's context limit and API usage can depend on tokens, rather than the number of words a person sees.
What this does not show
This visual covers text tokenization. Images, audio, message formatting, and model-specific accounting can add other units or overhead. It is not a billing estimator.
Related terms
Tokenizer
The tool that chops text into tokens and gives each piece a number. Different models chop differently.
FoundationsContext window
How much text the model can hold in mind at once: your question, the documents you added, and the conversation so far. Anything outside it is invisible to the model.
FoundationsInference
The act of running a trained model to get an answer. In law the word means something else: a conclusion drawn from evidence.
Where it is taught
What actually happens when you ask AI a question?Source and further reading
Hugging Face · TokenizersThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.