Understanding Your First Local AI Stack, Part 1: The Big Picture

The Naming Problem

#

Spend twenty minutes reading about local AI and you will collect a pile of nouns:

Ollama. Qwen. Llama. Gemma. GGUF. llama.cpp. Open WebUI. LM Studio. vLLM. Hugging Face.

They get introduced in the same breath, usually in the same sentence, which strongly implies they are alternatives to each other. They are not. Comparing Ollama to Qwen is like comparing PostgreSQL to a CSV file, or nginx to HTTP. Related, adjacent, and absolutely not substitutes.

Here is the actual taxonomy.

Six different categories. Once you have this table, the ecosystem stops being a fog of brand names and becomes a stack with layers you can point at.

What You Are Actually Building

#

Five components, each with exactly one job.

graph TD
    A["Browser"] --> B["Open WebUI
frontend"]
    B -->|"HTTP + JSON"| C["Ollama
runtime and server"]
    C -->|"loads into memory"| D["Model weights
e.g. Qwen3 8B, GGUF"]
    C --> E["Inference engine
ggml / llama.cpp"]
    E --> F["GPU VRAM
or system RAM + CPU"]
    D -.-> E

Notice that the intelligence is not a box on this diagram. It is a relationship between two boxes: a file full of numbers, and an engine that knows how to multiply them. Neither one alone is “the AI.” The weights without an engine are an inert 5 GB blob. The engine without weights is a very fast way to compute nothing.

That distinction matters more than it sounds like it should, and it is the thing that makes the rest of the stack make sense.

The Restaurant Analogy, and Where It Breaks

#

The standard analogy goes like this:

It is useful for one specific insight: you do not walk into the kitchen and hand your order to the chef directly. The waiter exists because taking orders and cooking food are genuinely different jobs, done by different people, on different schedules.

Now here is the part analogies never tell you — where it falls apart.

The chef has no memory. None. Every single time you send a message, the chef is handed a fresh transcript of the entire conversation so far and asked to write the next line, having never seen any of it before. There is no continuity, no “as I was saying.” Your chat history is not stored in the model; it is re-sent, in full, on every request.

This is why context windows exist, why long conversations get slower and eventually get truncated, and why “the model forgot what I said” is a storage question rather than a memory question. Hold onto that — it explains a surprising amount of local AI behaviour later in the series.

If you write backend services, the mental model you already have is closer: the model is a stateless function. Conversation state lives entirely in the caller.

Following a Prompt Through the Stack

#

You type “Explain Docker like I’m five” and press Send. Here is what actually happens.

sequenceDiagram
    participant U as You
    participant W as Open WebUI
    participant O as Ollama
    participant G as GPU

    U->>W: Type message, press Send
    W->>W: Append to conversation history
    W->>O: POST /api/chat with full history
    O->>O: Load model into VRAM if not resident
    O->>G: Run inference
    G-->>O: Token
    O-->>W: Stream token
    W-->>U: Render token
    Note over G,U: Repeat per token until stop
    O->>O: Keep model resident, then unload on idle

Three details in there are worth pulling out, because they explain most beginner confusion:

The full history goes over the wire every time. Not a delta. Not a session ID. The whole conversation, re-serialised, on every message. Turn 40 of a chat sends dramatically more data than turn 1.

Tokens stream back one at a time. The response is not computed and then delivered; it is generated left to right and pushed to your browser as it goes. That is why text appears to be typed rather than pasted.

The model stays loaded after you stop talking. Ollama keeps the weights resident in VRAM for an idle period (five minutes by default) so your next message does not pay the load cost again. This is why your first message is slow, the next ten are fast, and the one you send after lunch is slow again. Not a bug. Not a memory leak. A cache with a TTL.

Why Not Just One Application?

#

LM Studio bundles all of this into a single desktop app, and for many people that is genuinely the right answer. So why does the split-component version exist?

Because Ollama’s HTTP API is a contract, and contracts are what make parts replaceable.

graph LR
    A["Open WebUI"] --> API["Ollama HTTP API"]
    B["Your own Go service"] --> API
    C["A VS Code extension"] --> API
    D["A shell script + curl"] --> API
    API --> M1["Qwen3"]
    API --> M2["Gemma 3"]
    API --> M3["Llama 3.x"]

Everything on the left can change without touching anything on the right, and vice versa. Swap the model, keep the interface. Swap the interface, keep the model. Point four different tools at the same runtime and let them share one loaded copy of the weights instead of each hoarding their own.

Ollama also exposes an OpenAI-compatible endpoint alongside its native one, which means a pile of existing tooling written against that API will talk to your local machine with a changed base URL and nothing else. That is not a coincidence; it is the entire strategic point of having a contract at the boundary.

That same HTTP contract is also why “local” does not have to mean “on the machine in front of you.” Ollama does not care whether the caller is on the same box or a different one — it just answers requests on a port. I used exactly that to run models on my beefier home PC and query them from a laptop in another city; see Accessing Ollama Remotely with Tailscale if you want the networking side of this stack.

The same reason your web app does not let the browser query the database directly. One job per layer, talking over a defined interface.

Why Run Models Locally At All

#

Honest version, including the parts the enthusiast blogs skip.

Privacy. Prompts never leave your machine. For anything touching client code, personal notes, or internal documents, this is not a nice-to-have — it is often the only reason the tool is usable at all.

Offline capability. Once the weights are on disk, an internet connection is optional. Aeroplanes, trains, bad hotel wifi, ISP outages.

No per-token billing. No usage meter, no rate limits, no surprise invoice for a runaway loop. You pay in electricity and in the hardware you already bought.

Control. You choose the model, the quantisation, the system prompt, the sampling parameters. Nothing gets deprecated out from under you or silently swapped for a cheaper variant.

Understanding. You get to see the whole machine. Which, if you have read this far, is presumably the actual reason you are here.

The Payoff: Debugging by Layer

#

Here is the practical dividend of all this architecture talk. When something breaks, the symptom tells you which layer to inspect, and you stop reinstalling things at random hoping the universe takes pity.

Every one of those maps to a box in the first diagram. That is what an architecture is for.

What You’ve Learned

#

• The ecosystem contains at least six distinct categories of thing — models, formats, engines, runtimes, frontends, registries — and most confusion comes from treating them as competitors.
• “The AI” is not a single component. It is weights plus an engine to execute them.
• The model is stateless. Your entire conversation is re-sent on every message.
• Components are split because the HTTP API between them is a contract, which is what makes any of it swappable.
• Local inference buys privacy, predictability and control. It does not usually buy speed.
• Symptoms map to layers, which turns debugging into a lookup rather than a ritual.

Next in the Series

#

Part 2: Ollama and the Model Layer. We go one level down: what Ollama actually does when you run ollama pull, what a GGUF file contains, what quantisation trades away, and how to work out in advance whether a given model will fit in your VRAM — before you download 9 GB to find out it doesn’t.