Prefill
Also: prompt processing, prompt evaluation
The model reading your question and documents all at once before it starts answering. Long documents mean a longer wait for the first word.
The fuller definition
Prefill is the first phase of running a model, in which it reads the whole prompt and any documents in one parallel pass. It is limited mainly by computing power and determines the wait before the first word appears.
- Prompt
Documents and question
- Parallel read
Mostly compute-bound
- First token
Answer begins
A simplified relationship, not a screenshot or a complete design.
A concrete example
A 40-page document may take several seconds of prefill before the answer starts streaming.
The distinction that matters
Prefill speed and writing speed are different numbers with different bottlenecks.
Related terms
Decode
The model writing its answer one word-piece at a time. How fast this goes depends mostly on memory bandwidth.
Hardware & hostingTime to first token
How long you wait before the first word of the answer appears. It grows with the length of what you gave the model to read.
Hardware & hostingKV cache — Key-value cache
Stored results from part of a model's earlier work. Reusing them can speed up the next step. This cache also takes memory.
Where it is taught
How models learn from examplesSource and further reading
Hugging Face · LLM courseThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.