Decode
Also: decoding phase, token generation, text generation speed
The model writing its answer one word-piece at a time. How fast this goes depends mostly on memory bandwidth.
The fuller definition
Decode is the phase in which a model writes its answer one token at a time. Each token requires reading the needed weights from memory again, so decode speed is limited mainly by memory bandwidth.
- Next token
One at a time
- Weights read
Every token
- Speed
Set by bandwidth
A simplified relationship, not a screenshot or a complete design.
A concrete example
A model writing 40 tokens per second is decoding at 40 tokens per second.
The distinction that matters
The speed formula, bandwidth divided by bytes read per token, predicts decode speed, not prefill speed.
Related terms
Prefill
The model reading your question and documents all at once before it starts answering. Long documents mean a longer wait for the first word.
Hardware & hostingMemory bandwidth
How fast data moves out of memory, in gigabytes per second. Think of it as the width of a loading door: wider means the model writes words faster.
Hardware & hostingTokens per second
How many word-pieces the model reads or writes each second. Around 20 or more feels comfortable to read along with.
Where it is taught
How models learn from examplesSource and further reading
Hugging Face · LLM courseThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.