The Case Against Always-On Internet AI

The privacy case for local AI is well-rehearsed. Less discussed are the mundane operational reasons that make local inference the better default for a significant portion of everyday AI work — reasons that have nothing to do with ideology.

Latency is the first one. For short prompts — autocomplete, single-turn questions, command generation — a local 7B or 14B model responds faster than a cloud API once you account for network round-trip. An Ollama request to a warm model on Apple Silicon returns a first token in under 200ms on a local network. A cloud API call adds 300–600ms of overhead before the first token regardless of how fast the model generates. For interactive use where you are iterating quickly, this is perceptible.

Reliability is the second. Cloud AI APIs have outages. Rate limits are imposed without warning when provider infrastructure is under load. GPT-4 has had multiple multi-hour outages. Anthropic's API has gone down during peak usage. If your workflow depends on a cloud API and it is unavailable, your workflow stops. A local model does not have an SLA, but it also does not have an operations team that can take it down.

Cost is the third. For high-volume use — processing large documents, running evaluations, iterating on prompts during development — cloud API costs accumulate quickly. A developer running a test suite against a cloud API can easily spend $50–100 in an afternoon of iteration. The same work against a local model costs electricity. The per-token cost of local inference at scale is close to zero.

None of these are arguments against cloud AI in every context. There are tasks where the absolute capability of the frontier matters, where the speed of the latest model release matters, where the specific features of a hosted API matter. Use the right tool. But the assumption that cloud is the default and local is the special case gets the economics backwards for a large fraction of real workloads.

If you are doing document processing, code generation, summarisation, or any high-volume structured task, the question to ask is not 'why would I use a local model?' It is 'why am I paying per token for this when I do not need to?'