GPU offload
Also: offloading, layer offload, CPU+GPU hybrid, n-gpu-layers
How much of the model sits on the graphics card. Whatever does not fit goes to slower system memory, which slows the answer.
The fuller definition
GPU offload is how many of a model's layers are placed on the graphics card. The rest run from system memory on the processor, which lets a model larger than the card run, more slowly.
- On the card
Fast memory
- In system RAM
Slower memory
- Overall speed
Set by the slower part
A simplified relationship, not a screenshot or a complete design.
A concrete example
Setting llama.cpp's -ngl to all puts every layer on the card; a lower number leaves some layers in system RAM.
The distinction that matters
Offloading a few layers costs a little speed; offloading most of a dense model makes it crawl. For mixture-of-experts models, keeping experts in RAM is a useful middle path.
Related terms
VRAM — Video random-access memory
The memory on the graphics card. A model has to fit in it to run quickly.
Hardware & hostingRAM — Random-access memory
A computer's short-term working memory. Programs and data live here while in use.
FoundationsMixture of experts (MoE)
A model built from several specialist sub-networks, with only a few switched on for each piece of text. It saves work without losing size.
Where it is taught
Understand the hardware a local model needsSource and further reading
Hugging Face · Model inference and quantizationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.