Run a local model and verify the actual setup

Keep the first experiment small

A local runtime loads model weights and performs inference on hardware you control. Ollama provides model management and a local interface; llama.cpp provides another inference implementation with its own supported formats and configuration. They are examples of runtimes, not guarantees that every model and GPU combination is supported. [source] [source]

Use a public or fictional prompt first. Keep document conversion, remote connectors, and agent tools out of this experiment so that you can tell what is being tested.

A step-by-step first run

1. Open the official Ollama quickstart linked below. Follow the installation instructions for your operating system.
2. Choose a downloadable local model from its model library that is compatible with your hardware. Check the model's documentation and licence. Avoid cloud-labelled choices for this exercise.
3. Open your terminal and run the command below, replacing MODEL_TAG with the exact tag you selected.
4. Wait for the download and model load. Ask the fictional prompt below.
5. In a second terminal, inspect the model list and running model information.

ollama run MODEL_TAG

Fictional source A01: “The proposed date is 20 March, subject to inspection.” Summarize only what this sentence establishes. Do not say the inspection occurred.

ollama ls
ollama ps

The CLI reference documents these commands. Use the running-model information to inspect placement and configured context; do not infer GPU use solely from owning a GPU. [source]

Test local operation rather than the app's appearance

After the model has downloaded, use the documented setting to disable cloud features where appropriate, restart the application, and repeat the small task with network access disconnected or restricted. Ollama's FAQ describes local-only configuration. Test the actual pipeline if you later add OCR, embeddings, or a user interface. [source]

An offline inference test establishes something useful about that run. It does not prove that every component or future configuration avoids network access, or that local storage and user access are appropriately protected.

Record the experiment

Write down model tag, runtime version, operating system, hardware, context setting, prompt, output, startup delay, and review result. Retain the exact prompt as a baseline. Add a negation and an ambiguous date before trying a longer document.

If loading fails, check the precise error, free memory, supported hardware, and model format. If it runs slowly, inspect placement and workload size before assuming a hardware fault. Do not use commands that expose a local model server to the internet as a troubleshooting shortcut.