Choosing a model: families, sizes, formats, and licences

What you will be able to do

• Choose a model that fits your memory with room for a document.
• Read a model card for the facts that matter: size, active parameters, context length, licence.
• Pick a file format and quantization level.

The current families

These are open-weight models: the trained weights are published and can be downloaded and run locally. Qwen, Gemma, gpt-oss, and the Mistral models in the table are released under Apache 2.0; Llama 3.3 uses Meta's own community licence and requires accepting terms before download. [source] [source] Licences matter for firm use: read them before deploying, not after.

Formats and quantization labels

GGUF is the single-file format used by llama.cpp, Ollama, and LM Studio. Within a GGUF repository you will see names like Q4_K_M or Q8_0. These are quantization labels: the number is the approximate bits per weight, and the suffix describes the method. Q4_K_M is the common starting point; Q8_0 is close to the original quality at twice the size. Some newer models ship in MXFP4, a 4-bit format; gpt-oss is published that way. MLX on a Mac uses its own format, and the original weights on Hugging Face are usually in safetensors.

LM Studio stores downloaded models under ~/.lmstudio/models/, organized by publisher and model name. [source]

Reading a model card

A model card is the model's specification sheet. Look for: total parameters; active parameters if it is mixture-of-experts (Qwen3.6-35B-A3B: 35 billion total, 3 billion activated) [source]; context length; recommended settings; and the licence. For fit calculations, the config file gives layers, KV heads, and head size; for Mistral Small 3.2 those are 40, 8, and 128. [source]

Picking a model for your memory

1. Take your usable memory: the card's VRAM, or about three-quarters of a Mac's unified memory to leave room for macOS and apps.
2. Subtract the KV cache for the longest document you plan to use (lesson 9.2), plus a gigabyte or two for the engine.
3. Choose the largest model whose 4-bit file fits in what is left. If a mixture-of-experts model fits, try it: it will be much faster.
4. Test two candidates on the same sample task from your own work, and keep the better one.

Self-check

1. What does Q4_K_M tell you about a file?
2. Which licence does Llama 3.3 use, and what is different about getting it?
3. On a 16 GB card, with 5 GB set aside for a long document, roughly how large a 4-bit file can you use?
4. Where do you find a model's layer count and KV-head count?