What changes when your LLM runs on the user's machine

Building against a hosted model API, you design for latency and rate limits. Building against a local model via Ollama, the constraints flip: latency is often lower and free, but you're now designing for wildly variable hardware, models that may not be pulled yet, and zero guarantee of availability if the user's machine is under load.

The frontend has to do more work as a result. Streaming becomes non-negotiable rather than a nice-to-have, since local inference on modest hardware can be slow enough that a non-streaming UI feels broken. You also need honest, specific error states — 'model not found, run ollama pull llama3.2' is a real state your UI has to handle, not an edge case.

The payoff is a genuinely different privacy story: nothing leaves the device, which changes what you're allowed to promise the user and what you don't have to build (no server-side data handling for the conversation itself). That tradeoff is worth designing for deliberately, not backing into.