Memory bandwidth
Also: GB/s
How fast data moves out of memory, in gigabytes per second. Think of it as the width of a loading door: wider means the model writes words faster.
The fuller definition
Memory bandwidth is how many gigabytes per second can move between memory and the processor or graphics chip. For local AI it sets how fast a model writes, because every new token requires reading the model's active weights again.
- Memory
Holds the weights
- Door width
GB per second
- Speed
Bandwidth ÷ bytes per token
A simplified relationship, not a screenshot or a complete design.
A concrete example
A card with 896 GB/s writes about twice as fast as one with 448 GB/s on the same model.
The distinction that matters
Capacity decides whether a model fits; bandwidth decides how fast it answers once it does.
Related terms
VRAM — Video random-access memory
The memory on the graphics card. A model has to fit in it to run quickly.
Hardware & hostingUnified memory
One big pool of memory shared by a computer's processor and graphics, as on Apple Silicon Macs. Big models fit; they just run a bit slower than on a fast graphics card.
Training & generationDecode
The model writing its answer one word-piece at a time. How fast this goes depends mostly on memory bandwidth.
Where it is taught
Understand the hardware a local model needsSource and further reading
Hugging Face · Model inference and quantizationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.