What you will be able to do
- Estimate how much memory a model needs, including the notes it keeps on your document.
- Predict how fast it will write, from one number on a spec sheet.
- Explain why a mixture-of-experts model can be both large and fast.
The organizing idea
Everything in this path comes back to two numbers. Memory capacity, either the graphics card's VRAM or a Mac's unified memory, decides which models fit. Memory bandwidth decides how fast they answer. Every later lesson refers back to this pairing.
The size of the warehouse. The whole model has to fit inside, plus its working notes. If it does not fit, it cannot run on that memory at all.
Measured in gigabytes (GB). Decides which models you can use.The width of the loading door. To write each word, the model carries every needed weight through the door once. A wider door means more words per second.
Measured in gigabytes per second (GB/s). Decides how fast answers appear.Remember this pairing; every later lesson refers back to it. A machine with a huge warehouse and a narrow door runs big models slowly. A machine with a small warehouse and a wide door runs small models very fast.
Formula A: will it fit?
A model is a very large set of learned numbers, its weights. Each weight is stored with a certain number of bits. Quantization stores each weight with fewer bits, which shrinks the file at a small cost in quality.
Model size ≈ parameters × bits ÷ 8, plus the KV cache.
A model is billions of dials. A “24B” model has about 24 billion of them. Each dial holds one number learned in training.
More dials usually means more capability and more memory.How finely each dial is marked. 16 bits is a ruler with 65,536 marks; 4 bits is a ruler with 16. Fewer marks, smaller file, slightly rougher answers.
Shrinking the bits is called quantization.Parameters × bits ÷ 8 = bytes. 24 billion × 4.8 bits ÷ 8 ≈ 14.4 GB. The real Mistral Small 3.2 4-bit file is 14.33 GB.
The formula lands within 1% of the file you would download.The 4.8 figure is the average bits per dial in the common Q4_K_M format, which mixes 4-bit and 6-bit blocks. File size checked against the Hugging Face listing on 16 October 2026.
Worked example. Mistral Small 3.2 has 24.01 billion parameters. At the common 4-bit format (which averages about 4.8 bits per weight): 24.01 × 4.8 ÷ 8 ≈ 14.4 GB. The real file on Hugging Face is 14.33 GB. At 8 bits the formula says 24.0 GB; the real file is 25.05 GB. [source]
The 16-bit figure is the formula (24.01 B × 16 ÷ 8); the 8-bit and 4-bit figures are the real files on Hugging Face. For most legal drafting and extraction, a well-made 4-bit or larger file is the usual starting point; test on your own material before settling.
The second term is the KV cache: as the model reads your document, it keeps a small note for every token on every layer, so it does not have to reread. Its size is 2 × layers × KV heads × head size × bytes per value × tokens. Mistral Small 3.2 has 40 layers, 8 KV heads, and a head size of 128, stored at 2 bytes by default: 163,840 bytes per token. [source] A 32,000-token document (roughly 60 to 80 pages) adds about 5.24 GB. So the honest answer to "will Mistral Small fit on a 16 GB card?" is: the weights do, but not with a long document as well.
Think of a reader who writes a sticky note for every word, on every one of the model’s 40 layers. The pile grows with the document. Per token: 2 × 40 layers × 8 KV heads × 128 dimensions × 2 bytes = 163,840 bytes. Layer and head counts from the model’s config.json. Long documents need room for these notes on top of the model itself.
llama.cpp can store these notes more compactly (its --cache-type-k and --cache-type-v options accept q8_0 and smaller types), which roughly halves the cache at some cost in accuracy. [source]
Formula B: how fast will it write?
Running a model has two phases. Prefill reads your prompt and documents in one parallel pass; it is limited mostly by computing power and sets the wait before the first word. Decode writes the answer one token at a time, and for every token the hardware has to read the needed weights from memory again. That makes writing speed a memory problem.
The model reads your whole prompt and documents at once, in parallel. Limited mostly by computing power. Measured as prompt tokens per second.
A 30-page document might take seconds before the first word appears: the time to first token.The model writes one token at a time, and for each one it must read the needed weights from memory again. Limited mostly by bandwidth.
This is the speed you feel while the answer streams. The speed formula predicts this number.Two different bottlenecks. A fast processor helps reading; a wide memory door helps writing.
Decode speed ≈ memory bandwidth ÷ bytes read per token. For a dense model, bytes per token is roughly the whole file.
Worked example. An NVIDIA GeForce RTX 5090 has 1,792 GB/s of bandwidth. Qwen3.6-27B at 4-bit is 16.82 GB. 1,792 ÷ 16.82 ≈ 107 tokens per second. Treat this as a ceiling: real engines typically reach some fraction of it. Lesson 9.9 explains how to measure the gap on your own setup.
Mixture of experts: large and fast at once
A mixture-of-experts model is split into many specialist sub-networks. A router sends each token to only a few of them, so only the active parameters are read per token. Qwen3.6-35B-A3B has 35 billion parameters in total and 3 billion active: 256 experts, of which 8 routed and 1 shared answer each token. [source] gpt-oss-20b has 21 billion total and 3.6 billion active. [source]
Qwen3.6-35B-A3B has 256 experts. All 35 billion parameters must sit in memory, so capacity is set by the total.
Capacity uses the total parameters.For each token, a router picks 8 experts plus 1 shared one: about 3 billion parameters. Only those are read through the door.
Speed uses the active parameters.This is why a 35B mixture-of-experts model can answer faster than a 14B dense one, but still needs memory for 35B. Expert counts from the Qwen3.6-35B-A3B model card.
So for MoE models, use total parameters in Formula A and active parameters in Formula B. gpt-oss-20b's file is 12.11 GB, of which roughly 3.6 ÷ 20.9 is read per token, about 2.1 GB. On a card with 448 GB/s: 448 ÷ 2.1 ≈ 213 tokens per second, as a ceiling. The real figure is lower because some weights (attention, embeddings) are read for every token regardless, but MoE models do run far faster than their size suggests.
When it does not fit: offloading
If a model is bigger than the card, engines can split it: some layers on the card, the rest in system RAM, read by the processor. llama.cpp calls this CPU+GPU hybrid inference, and its -ngl option sets how many layers go to the card. [source] [source] This is offloading.
Layers stored in VRAM are read through a very wide door: 448 to 1,792 GB/s on current cards.
Layers that do not fit go to system memory, read by the processor through a narrower door: about 90 to 100 GB/s on a typical dual-channel DDR5 desktop.
Derived: DDR5-6000 × 2 channels × 8 bytes ≈ 96 GB/s.Splitting a model works, but the slow part sets the pace. Offloading a few layers costs a little speed; offloading most of a dense model makes it crawl. Mixture-of-experts models are the exception worth knowing: keep the always-used parts on the card and the experts in RAM.
Speed falls toward the slower memory. For MoE models there is a useful middle path: keep the always-used parts on the card and the experts in RAM. llama.cpp's --cpu-moe option does exactly that. [source]
Self-check
- Write Formula A and Formula B in words.
- A 14-billion-parameter dense model at 4.8 bits per weight: about how large is the file?
- A machine has 614 GB/s of bandwidth. About how fast can it write with a 16.82 GB dense model?
- For a mixture-of-experts model, which parameter count goes into each formula?
- Why might a model that fits on a card still fail with a long document?
Check your understanding
- Size ≈ parameters × bits ÷ 8, plus the KV cache. Writing speed ≈ bandwidth ÷ bytes read per token.
- 14 × 4.8 ÷ 8 ≈ 8.4 GB.
- 614 ÷ 16.82 ≈ 36 tokens per second, as a ceiling.
- Total parameters for size; active parameters for speed.
- The KV cache grows with document length and needs memory on top of the weights.