Understanding Your First Local AI Stack, Part 4: Installing the Stack

The Decision That Comes First

#

Every local AI tutorial opens with either “install Ollama” or “here’s a compose file,” and treats the choice as taste. It isn’t. It is determined almost entirely by one question:

Can the thing you’re about to run reach your GPU?

Containers get access to hardware only when the host explicitly hands it over. On Linux with an NVIDIA card there is a supported mechanism for that. On an Apple Silicon Mac there is not — Docker Desktop cannot pass the GPU through at all, because Apple’s virtualisation framework exposes no GPU interface to the VM that Docker runs inside.

That is not an Ollama bug and no flag fixes it. Put Ollama in a container on a Mac and it will run — silently, correctly, and entirely on the CPU.

Recall from Part 2 that generation speed is bounded by memory bandwidth. A Mac’s unified memory has bandwidth in the hundreds of GB/s; its CPU cores reading through the same memory without Metal do not get you the same throughput. Containerising Ollama on a Mac doesn’t cost you a little performance. It costs you most of it.

So the rule that resolves every arrangement below:

Installing Ollama Natively

#

On Linux:

1

curl -fsSL https://ollama.com/install.sh | sh

Worth knowing what that does before you pipe a script into a shell: it drops a binary in /usr/local/bin, creates a dedicated ollama system user, and installs and enables a systemd unit running as that user. It is a system service from the first second, started on boot, not something tied to your login session.

On macOS you download the application. Metal acceleration is automatic — no drivers, no configuration, nothing to enable.

Verify it is actually up, from the outside rather than the CLI:

1

curl http://localhost:11434/api/tags

An empty models array is a success. You have a running server with nothing pulled yet. Then:

1

ollama pull gemma4:12b

GPU Passthrough on Linux

#

If you’re containerising Ollama on Linux with an NVIDIA card, this is the part that people skip and then spend an evening debugging.

First, prove the host can see the GPU. If this fails, nothing downstream can possibly work:

1

nvidia-smi

No output means a driver problem, and the container toolkit will not rescue you. Fix the host first.

Then install the toolkit and point Docker at it:

1
2
3

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The middle command edits /etc/docker/daemon.json to register NVIDIA’s runtime. The restart is not optional — Docker reads that file at start.

Now prove a container can see the GPU, before involving Ollama at all:

1

docker run --rm --gpus=all ubuntu nvidia-smi

Same table as on the host means passthrough works. This two-step verification — host first, then a throwaway container — is worth doing every time, because it splits “my GPU isn’t working” into two questions with different answers.

Only then:

1
2
3
4
5

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

How the Two Containers Find Each Other

#

This is where most people’s first attempt dies, and it always dies the same way: localhost inside a container means the container itself. Open WebUI looking for Ollama on http://localhost:11434 is looking inside its own network namespace, finding nothing, and showing you an empty model dropdown.

There are three arrangements, and they differ only in what address the frontend uses.

graph TD
    subgraph A["Both native"]
      A1["Open WebUI"] -->|"localhost:11434"| A2["Ollama"]
    end
    subgraph B["Frontend in Docker, runtime native"]
      B1["Open WebUI container"] -->|"host.docker.internal:11434"| B2["Ollama on host"]
    end
    subgraph C["Both in Compose"]
      C1["open-webui service"] -->|"http://ollama:11434"| C2["ollama service"]
    end

Arrangement B — frontend containerised, Ollama native. This is the right answer on Apple Silicon, and a good one on Linux too: Ollama gets unmediated GPU access, and the web app stays disposable.

1
2
3
4
5

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

The --add-host flag is doing the real work. On Docker Desktop host.docker.internal already resolves; on Linux it does not exist until you create it, and host-gateway is the magic value that resolves to the host from inside the container.

Arrangement C — both in Compose. Containers on a user-defined network resolve each other by service name. http://ollama:11434 works because Docker runs a DNS resolver on that network, and ollama is the service’s name in the compose file. No IP addresses, no --add-host, no host networking.

A Compose File Worth Keeping

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped
    # Remove this whole block on a machine without an NVIDIA GPU
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama:
  open-webui:

Four things in there are deliberate:

Ollama publishes no ports. There is no ports: block on the runtime. It is reachable by open-webui over the internal network and by nothing else. Given that Ollama’s API has no authentication, not publishing it is the single most valuable line in this file — or rather, the most valuable line that isn’t there.

OLLAMA_BASE_URL uses the service name. This is the arrangement-C wiring from above, stated explicitly rather than relying on a default.

depends_on controls start order, not readiness. It waits for the container to start, not for the server inside it to answer. Open WebUI starting slightly before Ollama is ready is harmless — it just means the model list is briefly empty. Don’t read more into the flag than it does.

Both volumes are named. Which brings us to the part people only think about once.

Where Your Data Actually Lives

#

The Linux model path trips people up because it depends on which user runs the server. The install script’s systemd unit runs as the ollama user, so models land in /usr/share/ollama/.ollama/models. Run the binary yourself and they land in ~/.ollama/models instead. Same software, two locations, and “where did my models go” is usually this.

Point OLLAMA_MODELS at a bigger disk if you need to — on Linux, make sure the ollama user can actually write there:

1

sudo chown -R ollama:ollama /mnt/big-disk/models

That asymmetry is Part 3’s argument showing up in your filesystem. The model is stateless and replaceable; the state lives in the frontend. Your backup strategy should follow the state, not the gigabytes. If you already run a homelab with real backups, this volume belongs in that rotation.

The Binding Question

#

Part 3 deferred this deliberately, so here it is.

By default Ollama binds to 127.0.0.1:11434 — reachable only from the machine it runs on. To change that under systemd:

1

sudo systemctl edit ollama.service

1
2

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

1
2

sudo systemctl daemon-reload
sudo systemctl restart ollama

Now think hard about whether you want this.

0.0.0.0 means every interface — Wi-Fi, Ethernet, VPN, everything. And Ollama has no authentication of any kind. There is no API key, no user model, no rate limit. Anything that can open a TCP connection to that port can list your models, run inference on your GPU, load a 40 GB model to exhaust your VRAM, or pull arbitrary models onto your disk.

The safe options, in the order I’d reach for them:

Private network. Put the machine on a tailnet and bind to that interface only. I’ve written up reaching Ollama across cities over Tailscale — the runtime never touches a public interface, and the network layer does the authentication.

SSH tunnel, for a one-off from a machine you can already log into. Port forwarding over SSH gets you a local port without binding to anything.

Expose the frontend instead. Open WebUI does have accounts and sessions. If you want browser access from your phone, put a reverse proxy and TLS in front of Open WebUI on port 3000 and leave Ollama on localhost. You are then exposing the layer that was built to be exposed.

Firewall the port if you must bind widely — scope the rule to a single interface or subnet rather than trusting the bind address alone. My iptables introduction covers the mechanics.

The pattern here is the one from Part 1: each layer does one job. Authentication is not the runtime’s job, so don’t put the runtime where it needs to do it.

When It Doesn’t Work

#

Almost every installation failure maps to a layer, exactly as promised in Part 1.

That last row deserves its own warning. docker compose down stops containers; docker compose down -v also removes the volumes, and the flag is one keystroke away from routine.

1
2

docker compose down      # safe
docker compose down -v   # deletes your conversations

What You’ve Learned

#

• Native versus Docker is decided by GPU access, not preference. Apple Silicon cannot pass a GPU into a container at all, so Ollama in Docker on a Mac is CPU-only and dramatically slower.
• The runtime goes where the GPU is; the frontend can live anywhere.
• On Linux, verify GPU passthrough in two steps — nvidia-smi on the host, then in a throwaway container — before blaming Ollama.
• localhost inside a container is the container. Use host.docker.internal with --add-host on Linux, or a Compose service name.
• Don’t publish Ollama’s port. It has no authentication, so the network boundary has to come from somewhere else: a tailnet, an SSH tunnel, or a reverse proxy in front of the frontend rather than the runtime.
• Model storage on Linux depends on which user runs the server, which explains most “my models disappeared” reports.
• The 50 MB Open WebUI volume matters more than the 50 GB Ollama one. Weights are re-downloadable; conversations are not.

Next in the Series

#

Part 5: Where to Go Next. The stack is running, so what do you actually build on it? Embeddings and what a vector really is, how RAG works and the specific ways it goes wrong, why “chat with your documents” is a retrieval problem rather than an AI one, and where agents fit — plus an honest look at which of these is worth your time on local hardware.