Throughput
Also: tokens per second
How much work a system can get through in a given time.
The fuller definition
Throughput measures how much work a system completes per unit time, such as generated tokens per second or requests per minute.
- Work
Tokens or requests
- Time
Measured interval
- Rate
Work divided by time
A simplified relationship, not a screenshot or a complete design.
A concrete example
A service processes many users efficiently by batching requests even if an individual request waits longer.
The distinction that matters
High aggregate throughput does not guarantee low latency for one user.
Related terms
Latency
The wait between sending a request and getting the answer back.
APIs & softwareBatch processing
A group of tasks sent or run together. Some services charge less if the tasks can wait.
FoundationsInference
The act of running a trained model to get an answer. In law the word means something else: a conclusion drawn from evidence.
Where it is taught
APIs and JSON: connect a model to a small programSource and further reading
MDN · Web technology glossaryThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.