Quantization
Also: quantized, quantisation
Storing model numbers with less detail. This can save memory. The effect on quality depends on the model and the task.
Running a model on your own machine means loading all of its numbers into memory, like luggage into an overhead bin. Quantization shrinks the luggage. The graphics card’s memory (VRAM) is the bin that matters for speed.
The fuller definition
Quantization represents model values using fewer numerical levels or fewer bits, often reducing memory and bandwidth needs.
Go deeper
Idealized value storage only, in decimal GB. Metadata, caches, mixed precision, and runtime overhead add to actual requirements. This is not a GPU purchasing calculator.
A concrete example
A compatible runtime loads a lower-precision model variant to reduce its memory footprint.
The distinction that matters
Smaller representations can affect quality and do not make runtime overhead disappear.
Related terms
Model weights
The full set of numbers that make up a trained model. Download the weights and you have the model.
Hardware & hostingVRAM — Video random-access memory
The memory on the graphics card. A model has to fit in it to run quickly.
Hardware & hostingBit depth and precision
How much detail each number in a model is stored with. Fewer bits means a smaller, faster, slightly rougher model.
Where it is taught
Understand the hardware a local model needsSource and further reading
Hugging Face · Model inference and quantizationThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.