KV cache — Key-value cache
Stored results from part of a model's earlier work. Reusing them can speed up the next step. This cache also takes memory.
The fuller definition
A key-value cache stores attention-related representations from earlier tokens so an autoregressive model can reuse them while generating later tokens.
- Earlier tokens
Compute keys and values
- Cache
Retain reusable state
- Next token
Reuse prior computation
A simplified relationship, not a screenshot or a complete design.
A concrete example
As a conversation grows, its cached sequence state can consume additional memory.
The distinction that matters
Caching reduces repeated computation but does not create unlimited context or permanent memory.
Related terms
Attention
A way for a model to weigh links between parts of its input. The weights help it process text. They do not show which claims are true.
FoundationsContext window
How much text the model can hold in mind at once: your question, the documents you added, and the conversation so far. Anything outside it is invisible to the model.
Hardware & hostingVRAM — Video random-access memory
The memory on the graphics card. A model has to fit in it to run quickly.
Where it is taught
Understand the hardware a local model needsSource and further reading
Hugging Face · Model inference and quantizationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.