Understand the hardware a local model needs

Give each component a job

Storage keeps files and model weights when they are not in use. System RAM holds working data for the computer. A CPU performs general processing. A GPU can accelerate suitable parallel operations and has its own memory, usually called VRAM. Actual performance depends on software support and how the workload is placed.

Neural-network systems perform operations on arrays of numbers called tensors. PyTorch's introductory material demonstrates tensor operations on different devices. Having a GPU is useful only when the chosen runtime can use it for the work. [source]

Estimate the weights, then add the rest

A rough dense-model weight estimate is parameter count multiplied by bits per parameter, divided by eight. For an illustrative 8-billion-parameter model at 16 bits, this is about 16 billion bytes, or 14.9 GiB. At a nominal 4 bits, the raw arithmetic gives 4 billion bytes, or 3.7 GiB.

These are weight-only estimates, not promised download sizes or total memory requirements. Quantized formats add metadata or other overhead; some components may use different precision. Runtime buffers and context-related memory need additional capacity. Longer configured context can increase memory use. [source] [source]

A model file fitting on the disk tells you almost nothing about whether the chosen configuration will fit in VRAM or run at a useful speed.

Build a workload description

Before shopping, write down: model and format; intended context; typical input length; response length; simultaneous users; acceptable delay; and task-quality criteria. If the job is reviewing short letters, a configuration for serving several users with very long records may solve a different problem.

Distinguish time to first response from generation speed and total completion time. A fast stream of tokens can follow a long wait while the input is processed. Keep a cold-start measurement and a warmed-up measurement separate.

Investigate multi-GPU claims carefully

Two cards do not automatically behave like one larger card. A runtime must support a suitable placement or parallelism arrangement. The model, interconnect, PCIe layout, memory split, and communication overhead matter. Check the actual runtime's documentation and benchmark the intended setup before treating combined memory as a purchasing guarantee.

The same caution applies to unified memory, system-RAM offloading, and advertised accelerator performance. Compare the complete workload, not a single number on a specification sheet.