Guide to Air-Gapped and On-Premise LLM Deployments

Last reviewed: July 2026.

An air-gapped LLM deployment runs with no routable external network connection at all, not even occasional calls to a hosted model API. This is a meaningfully harder engineering problem than standard on-premise deployment, since every dependency, model weights, container images, drivers, updates, monitoring tooling, has to be acquired, verified, and operated without live internet access. It's justified for specific regulatory, contractual, or security requirements, not a default posture for on-premise AI, and it requires a complete offline operating model, not just a disconnected server room.

Quick answer: An air-gapped LLM deployment needs a defined isolation tier, a reference architecture separating a connected staging zone from the isolated production zone, an offline software supply chain with internal registries and mirrors, artefact validation with signing and chain of custody, security and identity controls that don't depend on external services, local observability, and a governed process for updates, evaluation, rollback, and disaster recovery.

Deployment Taxonomy: On-Premise, Disconnected, and Air-Gapped Aren't the Same

These terms get used loosely, but they describe meaningfully different architectures with different operational burdens:

A strict air-gapped deployment has no routable external network connection at all. Related architectures such as restricted, disconnected, or isolated-enclave environments may still permit controlled transfer gateways, one-way communication, or scheduled maintenance access; those are not equivalent to a continuous air gap. Define the required isolation tier explicitly, and in writing, before designing the platform, since the engineering effort scales sharply at each tier.

Do You Actually Need a Strict Air Gap?

Confirm the actual regulatory, contractual, or security requirement before committing to the strictest and most operationally demanding tier:

Choose the least operationally restrictive architecture that still genuinely satisfies the requirement. A connected private-cloud or on-premise posture with strong access controls meets most regulatory needs with far less ongoing burden than a strict air gap; reserve the strictest tier for requirements that specifically demand it. See our HIPAA and GDPR compliant AI agents guide for how common regulatory requirements map to deployment models.

Reference Architecture: The Two-Zone Model

A workable air-gapped platform separates a connected zone, where artefacts are acquired and verified, from the isolated zone, where they're actually run:

• Connected staging zone. Artefact acquisition, licence and version recording, checksum and signature verification, SBOM generation, malware and vulnerability scanning, and testing all happen here, with live internet access.
• Controlled transfer mechanism. Physical media, a one-way data diode, or a formally governed transfer gateway moves verified artefacts from the staging zone into the isolated zone, never the reverse for anything except explicitly approved exports.
• Isolated production zone. The air-gapped platform itself: internal container and model registry, the model-serving layer, an internal API gateway, and the enterprise applications and RAG services that consume it, all operating without external connectivity.

This staging-versus-production split mirrors the pattern documented in NVIDIA's NIM air-gap deployment guidance: prepare and package the required model and runtime assets in a connected environment, transfer them through an approved process, and then run without access to remote registries such as NGC or Hugging Face Hub.

The full flow looks like this: approved connected staging zone, artefact acquisition and verification, malware scanning and SBOM and signature verification, controlled transfer mechanism, internal container and model registry, the air-gapped platform (Kubernetes or bare metal), the model server, an internal API gateway, and finally the enterprise applications and RAG services that call it.

Hardware and Model-Capacity Planning

Size the isolated platform against actual model and traffic requirements, not a generic reference configuration:

• GPU type, quantity, and memory. Driven by model size, quantization level, and target context length.
• CPU, RAM, and local NVMe capacity. Needed for the serving stack, registries, vector database, and log storage, not just the model itself.
• Quantization and context length. Directly trade off against hardware footprint and achievable throughput.
• Concurrent requests, throughput, and latency targets. Determine whether a single-node or clustered serving layer is required.
• High availability and spare hardware. Air-gapped environments can't lean on cloud elasticity, so spare capacity has to be provisioned and physically present in advance.
• Power, cooling, and replacement lead times. Component failures can't be resolved by spinning up a cloud instance; plan lead times for physical replacement parts accordingly.

See our GPU clustering guide for the clustering and scaling considerations that carry over to an isolated deployment, with the added constraint that no elastic cloud capacity is available as a fallback.

The Offline Software Supply Chain

This is the largest and most commonly underestimated part of an air-gapped platform. Every one of the following needs an offline acquisition and update path: model weights, tokenizers, container images, GPU drivers, CUDA libraries, inference engine binaries, Python packages, operating system packages, Helm charts, Kubernetes operators, security signatures, SBOMs, and evaluation datasets. Red Hat's OpenShift disconnected-environment guidance follows the same principle for cluster software: installation content and dependencies are mirrored and made available locally rather than pulled from public registries, since a disconnected cluster cannot reach them directly.

Internal Registries and Package Mirrors

An offline platform stays maintainable only if every artefact type has a corresponding internal source:

• Private OCI container registry for serving and supporting container images.
• Internal model registry for model weights and versioned checkpoints.
• Python package mirror for the serving stack's dependencies.
• Operating system repository mirror for base image and host packages.
• Helm chart repository for Kubernetes deployment manifests.
• Driver and firmware archive for GPU and hardware updates.
• Approved artefact catalogue recording what's been validated and cleared for use, distinct from what merely exists in a mirror.

Artefact Validation and Chain of Custody

Every artefact that crosses into the isolated zone should pass through the same controlled sequence: acquire it from an approved source, record its version and licence, verify its checksum and signature, generate or collect its SBOM, scan it for malware and known vulnerabilities, test it in the connected staging environment, approve it for transfer, record the custody and the approving operator, import it into the isolated registry, and verify it again after transfer. Skipping steps to save time is how an unvalidated or tampered artefact ends up running inside an environment specifically designed to keep untrusted content out.

Removable-Media and Physical Transfer Risk

Physical media is a common transfer mechanism for air-gapped environments, but it's also a documented infiltration path. CISA's Cybersecurity Performance Goals warn that adversaries can introduce malware into disconnected or air-gapped systems through removable media such as USB drives, which is why a disconnected network doesn't stay meaningfully isolated if untracked physical media can cross the boundary freely. Practical controls include using dedicated, approved media rather than general-purpose drives, tracking media by serial number, preferring read-only transfer where possible, scanning for malware at both the sending and receiving boundary, using write blocking, sanitizing media after use, logging chain of custody for every transfer, and prohibiting personal USB devices entirely.

Security Architecture

An isolated platform still needs a full security architecture, arguably a more disciplined one, since it can't rely on a cloud provider's managed security tooling:

• Network segmentation and host hardening.
• Secure boot and disk encryption.
• Secrets management, RBAC, and service accounts scoped to least privilege.
• Audit logging and admission controls at the platform layer.
• Workload signing and model integrity verification before execution.
• Physical security and insider-risk controls around the hardware itself.
• Device control, including disabling or physically removing unnecessary wireless interfaces on sensitive equipment. NIST SP 800-209 provides a relevant control example for preventing wireless transmission capabilities in air-gapped storage equipment specifically; map broader platform controls to your organization's applicable security framework and system classification rather than treating this one publication as a general-purpose air-gapped architecture standard.

Identity and Access Management Without External Providers

Air-gapped systems can't assume live access to a cloud identity provider. Practical alternatives include a local identity provider or a replicated directory, an offline certificate authority, local multi-factor authentication, defined break-glass accounts for emergency access, privileged-session logging, and periodic manual account reconciliation to catch drift that automated deprovisioning would normally handle.

Local Observability and Monitoring

Cloud observability services aren't reachable, so monitoring has to run entirely inside the isolated zone: a local stack built from components such as Prometheus, Grafana, an OpenTelemetry collector, a local log store, a security information and event management tool, and offline-accessible dashboards. Track GPU utilization and memory, queue depth, tokens per second, time to first token, request failures, model load failures, disk utilization, registry health, temperature and power, and security events. Without this instrumentation running locally, a failure inside the isolated zone can go undetected far longer than it would in a connected environment with external alerting.

Data Ingress and Egress Controls

Isolating the model doesn't automatically make its inputs and outputs safe. A complete design covers approved source systems for data entering the environment, batch import processes, document sanitization, classification labels, data loss prevention checks, prompt and output logging, a defined export approval process, one-way gateways where exports are permitted at all, human review of sensitive outputs, and output redaction. The air gap protects the boundary; it doesn't substitute for controlling what crosses it in either direction.

RAG and Supporting Infrastructure

An air-gapped enterprise LLM system is rarely just the model. It typically needs local, offline-capable versions of document ingestion, OCR, an embedding model, a vector database, a reranker, object storage, a metadata database, and a retrieval API, all operating under the same isolation constraints as the model itself. See our RAG vs. fine-tuning guide for the retrieval architecture this depends on, our vector database comparison guide for selecting a vector store that can actually run self-hosted inside an isolated environment, and our prompt caching guide for the caching layer options available when the serving stack itself is self-hosted.

Updates, Evaluation, and Rollback

Treat every model, driver, runtime, or prompt-template change as a full lifecycle, not a drop-in replacement: monitor vendor and security advisories from the connected staging zone, acquire approved updates there, verify licences, signatures, and checksums, scan and test the update, run model-quality and security evaluations against it, transfer it through the approved channel, deploy it first to a non-production isolated environment, validate compatibility, promote to production, and preserve the previous model and runtime version so a rollback is possible if the update underperforms. Every such change can affect behavior in ways worth testing deliberately: answer quality, hallucination rate, refusal behavior, latency, throughput, context handling, safety controls, domain-specific task performance, retrieval quality, and access-control enforcement should all be part of the regression check before a change reaches production.

Backup and Disaster Recovery

Cover model registry backup, configuration backup, vector index backup, prompt-template versioning, secrets recovery, offline restoration media, a spare-node strategy, a defined recovery time objective and recovery point objective, and periodic restore tests. A backup strategy that's never been tested against an actual restore is not a verified recovery capability; schedule the test, don't just schedule the backup.

Cost Model

Air-gapping shifts cost from raw infrastructure into ongoing operations:

Air-gapping usually increases total operating cost through duplicated registries, manual artefact transfer, offline patching, additional security validation, local observability, specialist staffing, and slower update cycles. Evaluate the cost difference as total cost of ownership, not server expenditure alone; see our AI infrastructure maintenance costs guide for the broader maintenance-cost framework this fits into, and our open source vs. commercial LLM cost guide for how self-hosted model economics compare more generally.

The Air-Gap Readiness Formula

Air-gap readiness = offline model rights + offline dependencies + local observability + a controlled update path + local security operations + recovery capability

Treat this as a completeness check, not a scoring exercise: if any one component is missing, the deployment is not operationally complete, regardless of how well the others are implemented. A platform with a working offline supply chain but no tested recovery capability, for example, is not actually air-gap ready; it's an isolated system with an unaddressed operational gap.

Model Selection for Air-Gapped Environments

Air-gapped environments require models whose weights, licence terms, serving software, dependencies, and updates can all be obtained and operated entirely offline. These may be open-weight models or commercially licensed models specifically designed for private deployment; the relevant requirement is offline deployment rights and offline availability of every dependency, not simply whether the model is described as open source. A model can have open weights but still carry licence, redistribution, commercial-use, or support restrictions that make it unsuitable for a given air-gapped deployment. Confirm licence terms explicitly before committing to a model for an isolated environment. Available serving stacks include NVIDIA NIM, vLLM, TensorRT-LLM, TGI, and llama.cpp, with Ollama suited to limited or non-enterprise cases; select based on your hardware, the specific model's support, required concurrency, quantization needs, observability integration, and licence and support requirements, not a universal ranking.

A Practical Deployment Workflow

Verify the specific regulatory, contractual, or security requirement against the deployment taxonomy above, rather than defaulting to the strictest tier.

Confirm licence terms, offline availability, and hardware fit before committing.

Stand up internal registries and mirrors before the isolated zone needs its first artefact.

Define validation, signing, and logging for every artefact crossing into the isolated zone.

None of these can depend on a reachable external service.

Verify recovery capability works before it's needed under pressure, not after.

Air-Gap Readiness Checklist

• Is the required isolation tier documented and confirmed against an actual requirement, not assumed?
• Are model licence terms and offline deployment rights confirmed for every model in use?
• Do internal registries and mirrors exist for containers, models, packages, OS repositories, and Helm charts?
• Is there a documented artefact validation and chain-of-custody process, including signing and scanning?
• Are removable-media controls, tracking, scanning, write blocking, and sanitization defined and enforced?
• Do identity, secrets, and access management work without any external identity provider?
• Is local observability covering GPU, latency, throughput, and security events actually running, not just planned?
• Is there a tested update, evaluation, and rollback process, not just a manual patching habit?
• Have backup and disaster recovery been tested with an actual restore, not only a backup job?
• Has the full total cost of ownership been budgeted, not just hardware procurement?

When Not to Use a Strict Air Gap

A strict air gap isn't the default secure posture; it's justified by a specific requirement. Reconsider it when the actual regulatory or contractual need can be met by a connected private-cloud or on-premise deployment with strong access controls, when the organization lacks the specialist staffing to operate an offline supply chain and local security operations sustainably, or when the operational cost of manual patching and validation would meaningfully slow response to a genuine security vulnerability. A connected posture that's actually well-operated can be more secure in practice than a strict air gap that's under-resourced to maintain.

Frequently Asked Questions

What's the difference between on-premise and air-gapped LLM deployment?

Connected on-premise keeps infrastructure in your own data center but often still has some network path to external services. A strict air gap removes all routable external connectivity, a stricter requirement that needs an offline supply chain, manual update processes, and local monitoring.

Can I use a commercial LLM API in an air-gapped environment?

Generally not, since most commercial APIs assume network connectivity. Air-gapped deployments typically require a self-hosted open-weight or commercially licensed model that can run entirely within the isolated environment, with confirmed offline deployment rights.

How do model updates work without internet access?

Through a defined manual transfer process, approved removable media, a controlled one-way transfer mechanism, or a formally governed transfer gateway, with signature verification, scanning, and staging-environment testing before anything enters the isolated zone.

Is air-gapping always necessary for regulated AI workloads?

Not always. Many regulatory requirements can be met with a connected on-premise or private-cloud deployment with strong access controls, which carries far less operational burden than a strict air gap. Confirm the specific requirement against the deployment taxonomy before committing to the stricter architecture.

What's the biggest operational risk in an air-gapped deployment?

An incomplete offline supply chain or an untested recovery process, since both failures surface only when they're needed most: during a security patch under time pressure, or during an actual disaster recovery event.

Does an air gap protect the data going into and out of the model?

Not by itself. The air gap protects the network boundary; data ingress and egress still need their own sanitization, classification, logging, and export-approval controls.

Methodology and sources: Architecture and operational patterns in this guide reflect publicly documented approaches from NVIDIA's NIM air-gap deployment documentation, Red Hat's disconnected-environment installation guidance, CISA's cybersecurity performance goals, and NIST's storage infrastructure security guidelines, current as of the review date above. Verify current specifics against each source and against your chosen model-serving stack's own documentation before implementation, since supported configurations and guidance evolve over time.

Our team designs AI infrastructure to the actual isolation requirement your organization has, including the full offline supply chain and operations model, not a default assumption about what "secure" deployment means.