The situation: your legal team, your regulator, or your own risk assessment has concluded that customer records, case files, patient notes or contract text cannot be sent to a third-party AI API — full stop. That does not mean you forgo AI. It means the model comes to the data instead of the data going to the model. This article covers the three sovereignty tiers we design for, the hardware and model choices that actually work in 2026, and the trade-offs nobody puts in the sales deck.
Why "just use the cloud API" stops being an answer
For most businesses, a well-configured cloud AI service with a data processing agreement is fine — we build on Azure daily and say so openly. But three groups regularly cannot accept it:
- European-regulated organizations. GDPR itself does not forbid cloud AI, but transfer-risk assessments after Schrems II, sectoral rules (banking secrecy, health records, legal privilege) and the EU AI Act's transparency duties make "where exactly does this prompt go, and who can be compelled to read it?" a board-level question. For some data categories the only clean answer is: it never leaves infrastructure we control.
- Government and public sector. Classified or restricted material, citizen registries, procurement records. Many states now publish explicit rules that such data may only be processed in accredited national or on-premise environments. A foreign-operated inference endpoint — however encrypted — is often disqualified by policy before any technical discussion starts.
- Companies with commercial secrets. Unreleased designs, M&A documents, pricing models. No regulation forces them, but they reasonably decide the marginal quality of a frontier cloud model is not worth wondering what happens to their prompts.
The three sovereignty tiers
| Tier | Where inference runs | Who it fits |
|---|---|---|
| 1. Region-pinned cloud | Public cloud, locked to an EU (or national) region, customer-managed keys, no abuse-monitoring retention | Most GDPR-conscious businesses; fastest and cheapest sovereignty step |
| 2. Private-cloud enclave | Dedicated GPU capacity in a sovereign cloud or local datacenter, models you chose, network-isolated | Banks, insurers, healthcare groups; data stays in-country and in-tenant |
| 3. Fully on-premise | Your own servers, your own rack, air-gappable | Government, defense-adjacent, legal privilege, anyone whose answer to "can it leave the building?" is no |
The engineering effort rises steeply from tier 1 to tier 3 — and so does the degree of control. The right question is never "which is most secure?" but "which is the least heavy tier that satisfies our actual obligation?"
What local models can genuinely do in 2026
The gap between open-weight and frontier models has narrowed to the point where, for scoped enterprise tasks, a well-served local model is not a compromise:
- Document intelligence — summarizing case files, extracting fields from contracts, classifying correspondence. Open-weight models in the 8–70B range handle this excellently, especially with retrieval grounding.
- Internal assistants — a chatbot over your policies, procedures and knowledge base, with every query and every document staying inside your network.
- Multilingual work — modern open models handle Vietnamese, German, French and other non-English languages far better than the previous generation; for niche domains, a LoRA fine-tune on your own corpus closes the rest of the gap.
- Structured output — JSON extraction, form filling, report drafting against a template. Constrained decoding on a local server is actually more reliable than prompting a remote API.
What they still do worse: open-ended frontier reasoning, very long multi-step agentic work, and staying current on world knowledge. Honest architecture puts those tasks — if they exist and the data allows — on a cloud tier, and keeps the sensitive work local. That split-brain design is the pattern we deploy most.
The hardware math
The sticker shock is smaller than most CTOs expect. A single server with two 48 GB GPUs runs a quantized 70B model with solid throughput for a few hundred employees' document workloads. A 8–14B model — enough for retrieval-grounded Q&A and extraction — runs on one workstation-class GPU. Concretely:
- Pilot: one GPU workstation, a vLLM or llama.cpp server, retrieval over your document store. Proves value in weeks.
- Production: two inference servers behind a load balancer for redundancy, a GPU-less embedding+retrieval node, monitoring, and an offline model-update procedure (new weights arrive on disk, are evaluated against your test set, then promoted).
- What to avoid: buying hardware before measuring your real token throughput. Most "we need a GPU cluster" conversations end with two servers after the workload is actually profiled.
The governance point people miss: a local model does not just protect data in — it protects the audit trail. Every prompt, every retrieved document, every answer can be logged into your own SIEM, retained under your own schedule, and produced for a regulator without asking a vendor for logs they may not keep. For EU AI Act documentation duties, owning the full inference log is a quiet superpower.
What you give up — said plainly
- Model pace. You will run this quarter's open-weight models, not this week's frontier release. For scoped tasks this rarely matters; for open-ended assistants it sometimes does.
- Ops responsibility. GPU drivers, model updates, capacity planning are now your problem (or your partner's). Tier 1 and 2 outsource most of this; tier 3 does not.
- Elasticity. A traffic spike cannot be absorbed by someone else's datacenter. Size for peak or queue gracefully.
A deployment pattern that works
- Classify data first. Most organizations discover only 10–30% of their AI use cases actually touch data that requires tier 2 or 3. Route the rest to region-pinned cloud and spend the sovereignty budget where it counts.
- Start with retrieval, not fine-tuning. A local model with good retrieval over your documents beats a fine-tuned model with no retrieval, and it is auditable — you can show exactly which sources produced an answer.
- Build the evaluation set before choosing a model. Fifty real tasks from your own workload, scored blind. The "best" model on public leaderboards is frequently not the best on your case files.
- Design the update path on day one. A sovereign AI system that still runs its launch-day model two years later is a slowly failing system. Weights, prompts and retrieval indexes all need a tested promotion procedure.
Where MERSO IT fits
We build on both sides of this line: Azure-based AI where cloud is acceptable, and local open-weight deployments — vLLM serving, retrieval pipelines, evaluation harnesses, SIEM-integrated logging — where it is not. Because we work in both worlds, we will tell you honestly which tier your workload actually needs, rather than selling you the heaviest one.