Quantization labels (Q4_K_M and others)
Also: Q8_0, Q5_K_M, K-quants, quant label
The codes in model file names, like Q4_K_M. The number is roughly how many bits each setting uses; smaller is lighter and slightly less precise.
The fuller definition
Labels like Q4_K_M and Q8_0 describe how a GGUF model file was quantized. The number is the approximate bits per weight; K marks the K-quant method; S, M, or L give the size variant.
- Q number
Bits per weight
- K
Method
- M
Medium variant
A simplified relationship, not a screenshot or a complete design.
A concrete example
Mistral Small 3.2 24B is 14.33 GB as Q4_K_M and 25.05 GB as Q8_0.
The distinction that matters
Lower numbers mean smaller, faster files with some loss in quality. Q4_K_M is the common starting point.
Related terms
Quantization
Storing model numbers with less detail. This can save memory. The effect on quality depends on the model and the task.
Hardware & hostingGGUF
The file format most local AI apps use for downloaded models.
Hardware & hostingBit depth and precision
How much detail each number in a model is stored with. Fewer bits means a smaller, faster, slightly rougher model.
Where it is taught
How models learn from examplesSource and further reading
Hugging Face · LLM courseThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.