Run Open-Source LLMs Locally: The 2026 Guide
From zero to a private, offline LLM stack on your own hardware: Ollama, Open WebUI, quantization, and the models worth running in 2026.
Note
Before you start
- A Mac, Windows PC, or Linux machine with 16GB+ RAM (an NVIDIA GPU helps but isn't required for 8B models)
- A terminal and roughly 1.5GB of free disk
- Docker only if you want the optional Open WebUI step to run in a container
Jump to section
- 1
Install Ollama
Install Ollama from ollama.com or brew install ollama. It ships with a model runtime and CLI.
- 2
Pull a model
Run `ollama pull qwen3:8b` — a strong 8B parameter model for daily questions. For bigger GPUs, `ollama pull llama3.3:70b`.
- 3
Serve the API
Run `ollama serve` to expose OpenAI-compatible endpoint at localhost:11434/v1.
- 4
Add Open WebUI
Spin up Open WebUI with `docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui` for a ChatGPT-grade chat UI.
- 5
Verify & tune
Open http://localhost:3000, create an account, and check model swap followed by GPU utilization with `ollama ps`.
Running state-of-the-art open models on your own laptop is a 25-minute project in 2026 — and a growing privacy default. This guide runs the complete stack locally with zero cloud accounts.
Why go local
Local inference removes per-token cost, keeps prompts and data off third-party servers, and gives you freedom to swap any open-weight model. For teams handling sensitive code or research notes, it is the only sane baseline.
Hardware budget
An M-series Mac or any 16GB-VRAM GPU runs an 8B model comfortably at 40-60 tok/s; 70B-class models want 32GB+ or quantization to 4-bit.
Pick the tier first
Choose a model tier before touching the CLI. The table below is the whole decision: pick a chip size that matches your RAM, then copy the exact pull command.
Choosing a model
| Model | Params | Quantized size | RAM needed | Best for |
|---|---|---|---|---|
| Qwen3 8B | 8B | ~5 GB | 8-10 GB | Daily Q&A, editing, cheap agents |
| Llama 3.3 | 70B | ~40 GB | 32-40 GB | Deep reasoning, long documents |
| DeepSeek-R1 distill 32B | 32B | ~19 GB | 20-24 GB | Math and code on mid-range GPUs |
To switch models later, just pull a new one — the runtime keeps them resident and swaps on request:
# install the runtime
curl -fsSL https://ollama.com/install.sh | sh
# pull the model that matches your hardware budget
ollama pull qwen3:8b
# serve the OpenAI-compatible endpoint
ollama serveConnect a chat UI
Open WebUI turns the bare endpoint into a ChatGPT-grade interface — conversations, histories, and model switching. Run it in Docker:
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
--name open-webui \
ghcr.io/open-webui/open-webuiThen open http://localhost:3000, create a local account, and confirm the model loads.
Verify the GPU actually loaded
After startup, run ollama ps. It lists loaded models and their memory footprint. If your model shows as 100% CPU, your engine is not using the GPU — check your driver and set the OLLAMA_GPU_LAYERS flag before troubleshooting anything else.
Next steps
Point your agent tools at http://localhost:11434/v1 (OpenAI-compatible), and pin favorites in Open WebUI so the whole team shares one local endpoint.
Questions, answered first
How much VRAM do I actually need?
You need roughly one gigabyte of VRAM per billion parameters at 4-bit. An 8B model fits comfortably in 8-10GB; a 70B model needs 32-40GB or heavy quantization.
Is local inference slower than the hosted APIs?
For an 8B model on an Apple M-series or modern GPU you get 40-60 tokens per second, which is faster than many hosted plans. Speed is the reason small local models win for daily, repetitive tasks.
Can tools like Cursor or other agent frameworks use my local model?
Yes. Ollama exposes an OpenAI-compatible API at localhost:11434/v1, so any tool with a custom OpenAI endpoint setting can point at it — including our agent guide.
You did it
- You can run `ollama list` and see at least one model
- `curl localhost:11434/v1/models` returns a JSON list
- Open WebUI shows your model and responds to a chat
- You have tried a second model and switched mid-conversation