Inference Strategy Is a Portfolio, Not a Slogan

A company may control data location, model weights, serving infrastructure, routing policy, or only the application contract. These are different forms of control with different costs. Running open weights locally does not create resilience if one machine, one operator, or one undocumented build remains a single point of failure.

A useful strategy names which property each workload requires: regulated data handling, offline continuity, predictable latency, frontier reasoning quality, or price stability. The deployment then follows the requirement rather than an ideology. Keeping a workload from leaving the building is a requirement. Owning inference, as a slogan, is not.

Models should be addressed through task classes and typed request contracts rather than scattered vendor names. The seam records model identity, parameters, cost, latency, and policy decisions while preserving a stable interface for callers.

Replaceability can then be tested. Route a representative workload to a second provider or local server and compare accepted results. If the product requires edits across prompts, tools, and UI to change the model, the gateway is cosmetic. The vendor has been wrapped, and the vendor remains in every caller.

Fallback sounds safe until it silently moves sensitive data or lowers quality. Each route should declare whether fallback is permitted, which destinations are eligible, what evidence must be recorded, and whether the task should stop instead.

Failing closed is appropriate when data classification or required capability cannot be preserved. Degraded service may be appropriate for low-risk drafts. The distinction belongs in policy before an outage. An incident chat is a poor place to invent data-handling rules.

Local inference adds model packaging, hardware capacity, upgrades, observability, warmup, concurrency, and incident response. Hosted inference adds vendor, pricing, policy, and regional dependencies. Neither side is free.

The decision should compare total cost per accepted task, recovery time, privacy boundary, and replacement effort. Ownership is valuable when it improves those properties for a defined workload. Moving the server so that ownership can be claimed is a different project.