Time to first token
Also: TTFT
How long you wait before the first word of the answer appears. It grows with the length of what you gave the model to read.
The fuller definition
Time to first token is the delay between sending a request and seeing the first word of the answer. It is dominated by the prefill phase, so it grows with the length of the prompt and documents.
- Request
Sent
- Prefill
Documents read
- First word
Answer starts
A simplified relationship, not a screenshot or a complete design.
A concrete example
A short question answers almost instantly; the same question with a long transcript attached may pause before the first word.
The distinction that matters
A system can have a long wait to start and still write quickly once it does.
Related terms
Prefill
The model reading your question and documents all at once before it starts answering. Long documents mean a longer wait for the first word.
Training & generationDecode
The model writing its answer one word-piece at a time. How fast this goes depends mostly on memory bandwidth.
APIs & softwareLatency
The wait between sending a request and getting the answer back.
Where it is taught
Understand the hardware a local model needsSource and further reading
Hugging Face · Model inference and quantizationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.