Tokens per second
Also: tok/s, t/s
How many word-pieces the model reads or writes each second. Around 20 or more feels comfortable to read along with.
The fuller definition
Tokens per second is the standard measure of how fast a model reads or writes. Writing speed is usually quoted as decode tokens per second; reading speed as prompt-processing tokens per second.
- Read
Prompt tokens per second
- Write
Generated tokens per second
- Feel
Writing speed
A simplified relationship, not a screenshot or a complete design.
A concrete example
At 30 tokens per second, a 300-word answer streams in roughly 13 seconds.
The distinction that matters
Always check which phase a number refers to; prompt and generation speeds differ by many times.
Related terms
Decode
The model writing its answer one word-piece at a time. How fast this goes depends mostly on memory bandwidth.
Training & generationPrefill
The model reading your question and documents all at once before it starts answering. Long documents mean a longer wait for the first word.
APIs & softwarellama-bench
A small tool that measures how fast a model reads and writes on your computer.
Where it is taught
Understand the hardware a local model needsSource and further reading
Hugging Face · Model inference and quantizationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.