Setting it up: three tracks, easy and expert routes

What you will be able to do

• Install a local model on your platform by the easy route, and by the expert route.
• Know where each tool stores its model files.
• Confirm the model is running on the graphics card, not the processor.

Before any track

Install the latest graphics driver from the card maker. Ollama documents its Windows requirements as NVIDIA driver 551.61 or newer, or an AMD ROCm v7 or Vulkan-capable Radeon driver. [source] Commands below are copied from each project's official documentation as of 16 October 2026; check the linked page if one fails, because these projects change monthly.

Track 1: Windows with an NVIDIA card

Easy route: LM Studio. Download it from lmstudio.ai and install. Open the Discover tab, search for a model from lesson 9.7, and choose a Q4_K_M file. Load it from the top bar and set GPU offload to maximum. LM Studio's command-line tool confirms what is loaded: [source]

lms ps

Or Ollama. In PowerShell: [source]

irm https://ollama.com/install.ps1 | iex
ollama run gemma4

Then, in a second window, check where the model landed. The PROCESSOR column should read 100% GPU; a split such as 48%/52% CPU/GPU means part of it spilled into system RAM. [source]

ollama ps

Expert route: llama.cpp. Install with winget, or download the Windows CUDA build (for example llama-b11687-bin-win-cuda-13.4-x64.zip) from the releases page. [source] [source]

winget install llama.cpp

List the devices llama.cpp can see, then run a model from Hugging Face with every layer on the card. -hf takes a repository and optional quantization tag; -ngl sets layers on the GPU and accepts all. [source]

llama-cli --list-devices
llama-cli -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M -ngl all

For a browser chat window, run the server instead. It listens on 127.0.0.1, port 9931, by default, so open http://127.0.0.1:9931. [source]

llama-server -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M -ngl all

Track 2: Windows with an AMD card

Easy route: Ollama. The same install command works. Ollama supports Radeon cards through ROCm v7 on the cards it lists for Windows (RX 7000 series and Radeon PRO W7000), and through Vulkan, which is enabled by default, for others. [source] [source] Check placement with ollama ps as above.

Expert route: llama.cpp. Download the Vulkan build (llama-b11687-bin-win-vulkan-x64.zip) or the ROCm build (llama-b11687-bin-win-rocm-10.0-x64.zip) from the releases page. [source] Run the same --list-devices and -ngl all commands. To build from source instead, the build guide gives the Vulkan route: [source]

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Track 3: Mac with Apple Silicon

Easy route: LM Studio. It requires Apple Silicon and macOS 14 or newer. [source] Install, choose a model in Discover (MLX or GGUF), and load it. On a Mac the GPU uses unified memory, so there is no offload slider to worry about.

Expert route: llama.cpp or MLX. Install llama.cpp with Homebrew; Metal acceleration is on by default. [source] [source]

brew install llama.cpp
llama-cli -hf lmstudio-community/Qwen3.5-9B-GGUF:Q4_K_M

Or use Apple's MLX: [source]

pip install mlx-lm
mlx_lm.chat

Where the files live

• Ollama: C:\Users\%username%\.ollama\models on Windows, ~/.ollama/models on macOS. [source]
• LM Studio: ~/.lmstudio/models/, by publisher and model. [source]
• llama.cpp: its cache, set by the LLAMA_CACHE environment variable. [source]

Adding a browser front-end

Open WebUI installs with Python 3.11 and serves on http://localhost:8080. [source]

pip install open-webui
open-webui serve

Self-check

1. Which command shows whether an Ollama model is fully on the GPU, and what should it say?
2. What does -ngl all do in llama.cpp?
3. Which llama.cpp build would you download for an AMD card on Windows?
4. Where does Ollama keep its models on Windows?