Running AI without the cloud: what air-gapped LLMs actually require

TL;DR: Most "on-prem AI" pitches actually mean a private cloud with a managed connection back to a vendor for updates and telemetry. True air-gapping removes that connection entirely — no NAT, no DNS to an external host, no certificate chain reaching outside the perimeter, and no route by which a single packet can leave. That single difference changes almost everything about how a model gets deployed, updated, and trusted.

Air-gapped is an architecture, not a setting

It's tempting to treat "air-gapped" as a stronger firewall rule. It isn't. A genuinely air-gapped deployment has no NAT, no DNS resolution to external hostnames, no public certificate authority chain, and no network path — physical or logical — by which a packet can leave the enclave. Every dependency the system needs at runtime — the model weights, the inference engine, the vector store, the UI — has to already be inside the perimeter before it's ever sealed. You can't retrofit air-gapping onto a system that assumed connectivity; you have to design for its absence from day one.

This matters because a lot of "secure AI" marketing describes an isolated or private-cloud deployment — still reachable for vendor updates, telemetry, or remote support — rather than a true air gap. For classified workloads, the distinction isn't academic: it's the difference between a system that meets the compliance bar and one that doesn't, because an isolated system still carries an entire category of network-boundary and monitoring requirements that an air-gapped one simply removes by not having a boundary to defend in the first place.

The update problem nobody likes to talk about

A connected AI system gets better every time its vendor ships a new model. An air-gapped one doesn't — until someone physically carries an update across the gap. In practice this means packaging models, container images, and configuration as signed tarballs: manifests, images, and any orchestration templates bundled together, cryptographically signed, and walked across the perimeter on approved media. The registry on the inside stays frozen between updates rather than pulling live. Cadence is deliberately slower than a connected system would tolerate — monthly for less sensitive domains, often quarterly for the most sensitive defence workloads — because every update is a controlled event, not a background process.

That trade-off is the whole point. A slower update cycle in exchange for a system that cannot be reached, tampered with, or exfiltrated from over a network is, for a defence workload, not a compromise — it's the requirement.

What actually has to run locally

Running a modern language model on-premises means the full stack has to fit inside the enclave: the model weights themselves (typically open-weight models sized and quantized to fit available hardware, from a single workstation up to a joint-command-scale deployment), the inference serving layer, any retrieval or document store the model draws context from, and the authentication layer gating who can query it. None of these can quietly depend on an external API, an update server, or a hosted vector database — each one is a thread back to the outside world if it does, and the whole point of the exercise is that there shouldn't be one.

Why a human still has to be in the loop

An air-gapped model can't phone home for a second opinion, a live correction, or a safety patch mid-session. That makes the evidence trail behind every AI output non-negotiable: a verdict that can't be traced back to a specific source document, sensor track, or clause isn't one an operator can safely act on. In practice, that means every AI-generated recommendation in a defence workflow needs to resolve to a verbatim citation, and a trained human needs the ability to confirm or override it, with the override itself logged. The model narrows the search space and does the reading no one has time to do manually; the human keeps the authority.

What this means operationally

• Procurement has to account for a slower, deliberate update cadence rather than assuming continuous improvement.
• Hardware sizing has to happen up front — there's no elastic cloud to burst into if a model needs more compute than expected.
• Every integration (document stores, sensor feeds, dashboards) has to be vetted for hidden outbound calls before it's allowed inside the perimeter.
• Audit and override logging aren't optional add-ons; they're what makes an air-gapped system's output trustworthy in the first place.

Frequently asked questions

What does air-gapped AI actually mean?

An AI system with no network route to the outside world at all — no NAT, no DNS to external hosts, no public CA chain, no path by which a packet can leave the enclave. It's an architecture decision, not a firewall setting.

Is air-gapped the same as an isolated or private-cloud deployment?

No. An isolated deployment can still have a managed connection back to a vendor for updates or telemetry. Air-gapped removes that connection entirely — every runtime dependency has to already be inside the enclave.

How do you update an air-gapped AI system?

Via signed tarballs — models, images, and manifests — physically carried across the gap and verified against a signature before deployment, on a deliberately slower cadence than a connected system.