A definitive enterprise blueprint for running AI inside your perimeter: who can ask, how agents act, which model responds, what knowledge grounds the answer, where inference runs, and what evidence proves the result was governed.
These are the questions procurement, security architecture, and platform teams ask first when evaluating on-prem, air-gapped, and governed private AI deployments.
It is the blueprint for running enterprise AI inside your controlled perimeter: identity, orchestration, model routing, private retrieval, inference, approvals, and audit evidence work together so requests become governed answers without sending prompts, documents, embeddings, or logs to an uncontrolled external data plane.
Use on-premises, private cloud, or air-gapped AI when sensitive data, regulatory duties, operational resilience, latency, data sovereignty, or procurement policy require customer-controlled infrastructure and a verifiable audit trail.
The model runtime is only one layer. A production enterprise platform also needs SSO, RBAC, policy-based model routing, private RAG, tool controls, human approval gates, observability, SIEM export, model lifecycle management, and evidence that auditors can inspect.
A production on-prem LLM architecture is not a single box labeled "model." It is a governed path. The platform must know who is asking, which policy applies, which agents and tools may act, which model is permitted, what private knowledge can be used, where inference runs, and which evidence is retained.
The production data plane should keep prompts, documents, embeddings, model outputs, logs, and approval records inside the enterprise boundary unless a route is explicitly approved, policy-gated, and auditable.
These layers are intentionally separable. Security teams can review identity and network controls, platform teams can operate inference and retrieval, governance teams can inspect policy and approval behavior, and business owners can evaluate answer quality by workflow.
Direct answer: Every request should start with enterprise identity, not a separate AI user store. SAML or OIDC login, group membership, role-based access control, and environment scope decide which agents, tools, models, and knowledge sources the user can reach.
The identity layer is the first control point because every downstream action inherits the actor, team, purpose, data classification, and approval policy. In regulated deployments, this layer should integrate with Entra ID, Okta, Keycloak, LDAP, privileged access management, and the organization's existing joiner-mover-leaver process.
Direct answer: It is the control layer that turns a user request into a governed plan: which specialist agents run, which tools they may call, which data they can retrieve, and where a human must approve the workflow before action is taken.
For simple chat, orchestration can be lightweight. For enterprise work, the orchestrator must coordinate agents, internal tools, retrieval, validation, and escalation. It should enforce least privilege at runtime, separate read-only analysis from write actions, and record every step for operations and audit.
Direct answer: Policy-based model routing selects the right model for each task based on sensitivity, domain, residency, latency, cost, quality, and approval status. High-risk workloads can be pinned to specific local models; external model APIs can be disabled entirely.
Routing is not a convenience feature. It is a governance control. A legal review task, HR policy answer, code analysis request, and public marketing draft may all need different models and logging policies. The router should make those decisions from inspectable policy instead of ad hoc prompt logic.
Direct answer: Private RAG retrieves relevant internal knowledge from customer-controlled stores, generates embeddings locally, enforces document permissions, and returns source-grounded context to the model without copying the corpus to a vendor cloud.
Secure on-prem RAG needs more than a vector database. It needs ingestion controls, source-system permissions, index lifecycle management, citation capture, prompt-injection filtering, and deletion workflows for retention or right-to-erasure requests. The same identity decision that allowed the request must constrain retrieval.
Direct answer: Enterprises should treat LLM inference as an operated service: approved models, repeatable serving stacks, capacity planning, batching, queue management, GPU monitoring, fallback policy, and a release process for model changes.
The runtime can use open-weight models, licensed commercial models, or smaller domain models, served through stacks such as vLLM, NVIDIA Triton, TGI, llama.cpp, or Ollama depending on workload. The architecture should keep workflows portable so model swaps do not rewrite the agent layer.
Direct answer: It should capture the actor, request, policy decisions, retrieved sources, model route, prompts, tool calls, human approvals, response, evaluation signals, and retention metadata in a searchable record that can stream to the enterprise SIEM.
Auditability is the difference between a useful demo and a defensible production system. Security, compliance, and model-risk teams need to reconstruct what happened, why it happened, who approved it, which data influenced it, and whether the same workflow still behaves correctly after model or corpus changes.
The right deployment model is a risk decision before it is an infrastructure decision. A bank, public-sector agency, manufacturer, hospital group, or energy operator may all want private enterprise AI, but their acceptable network paths, update process, and operating model can be very different.
Best for: Defense, intelligence, OT, critical infrastructure, and highly restricted networks.
Trade-off: Maximum isolation and strongest egress control, but updates, model refreshes, vulnerability management, and support access need offline artifact processes.
Best for: Regulated enterprises that want local data control, predictable operating cost, and integration with existing data-center controls.
Trade-off: Strong control and procurement clarity, but capacity planning and GPU operations must be owned by the enterprise or a managed operations partner.
Best for: Teams that need cloud elasticity but still require single-tenant networking, customer-managed identity, region pinning, and private data-plane controls.
Trade-off: Faster infrastructure scaling than data-center hardware, but still depends on cloud-provider services and must be assessed against residency and third-party risk policies.
Best for: Enterprises that want local handling for sensitive work and explicitly approved cloud model access for low-risk workloads.
Trade-off: Can improve model choice and burst capacity, but only works if the router, network policy, logging, and data classification rules make egress explicit and auditable.
Choose on-prem or air-gapped AI when data control, regulatory evidence, network isolation, model sovereignty, or high-volume cost predictability matters more than the speed of a managed public cloud API. Choose hybrid when workloads can be clearly classified and external routes can be governed. Choose cloud-only when risk is low and managed-service speed is the primary constraint.
| Criterion | On-prem or air-gapped | Hybrid private AI | Cloud-only AI |
|---|---|---|---|
| Data sensitivity | Choose on-prem or air-gapped when prompts, documents, embeddings, outputs, or logs include regulated, classified, trade-secret, or customer-confidential data. | Hybrid can work when low-risk content is clearly separable from restricted data and the router can enforce that boundary. | Cloud-only is usually simplest for public, low-risk, or already cloud-native workloads. |
| Regulatory and audit pressure | Best when auditors need direct evidence of identity, policy enforcement, data lineage, model route, approval, and retention inside customer-controlled systems. | Possible when audit records show exactly which requests used external services and why policy allowed them. | Works when cloud contractual controls and logs satisfy your risk team, regulator, and customer commitments. |
| Network isolation | Required for disconnected sites, restricted networks, OT-adjacent environments, and teams that must disable outbound internet. | Works for segmented environments where outbound paths are narrow, approved, inspected, and logged. | Best for internet-connected teams that prioritize managed service speed over isolation. |
| Operating cost profile | Favors predictable high-volume usage where platform, hardware, and operations can be amortized across many workflows. | Useful when utilization is uneven or burst capacity is needed before long-term capacity is proven. | Often cheapest to start, but token, storage, connector, and audit-export costs should be modeled at production volume. |
| Model control | Best when teams need approved local models, model version pinning, rollback, fine-tuned domain models, or no vendor-managed inference dependency. | Works when policy decides which models can handle which tasks and external model use is optional. | Best when a managed model catalog is more important than local model sovereignty. |
Sizing should start from the workflow, not the model leaderboard. A secure on-prem RAG workflow includes identity checks, retrieval, reranking, context assembly, inference, tool calls, policy checks, evaluation signals, and log export. Each step affects latency and capacity.
| Scale | Typical usage profile | Infrastructure shape | Sizing note |
|---|---|---|---|
| Pilot or controlled rollout | 25 to 100 named users; 2 to 10 concurrent conversations; a few governed workflows. | One small GPU node or a private cloud GPU instance for inference, plus CPU nodes for orchestration, retrieval, and observability. | Benchmark the real workflow, not only raw tokens per second. Include retrieval, reranking, tool calls, approvals, and log export in the test. |
| Department production | 100 to 1,000 users; 10 to 75 concurrent sessions; multiple teams and knowledge domains. | Separate runtime pools for chat, RAG-heavy workloads, and batch jobs; redundant vector/search services; central logging and metrics. | Use routing policy to keep small tasks on smaller models and reserve larger models for work that needs deeper reasoning or domain quality. |
| Enterprise platform | Thousands of users; 75+ concurrent sessions; many agents, tools, environments, and regulated workflows. | Multiple GPU pools, queue management, model registry, staging and production environments, HA control plane, SIEM integration, backup and DR. | Capacity planning should include peak concurrency, context length, output length, retrieval fanout, batch jobs, model refresh windows, and failover targets. |
| Air-gapped or restricted network | Often fewer users but higher assurance requirements, slower change windows, and stricter artifact review. | Local model registry, offline update channel, internal package repository, local telemetry store, and internal-only documentation and support runbooks. | Plan the update process as carefully as inference. Model files, containers, vulnerability fixes, and evaluation artifacts all need controlled import. |
"Best server" is a workload question (for the desk-side tier, see our DGX Spark and Mac Studio comparison). The table below maps the common GPU classes to the open-weight model sizes they serve well and the agent workloads they fit. Two formulas do most of the sizing work before any benchmark: model memory ≈ parameters × bits per weight ÷ 8 (a 70B model at INT4 ≈ 35 GB of weights), and KV-cache ≈ 2 × layers × KV heads × head dimension × active tokens × bytes per value — which is why long-context agent sessions, not model size, exhaust VRAM first. CPU sizing follows the agent layer, not the model: orchestration, retrieval, reranking and policy checks are CPU-bound, and a practical starting ratio is 2–4 CPU cores and 8–16 GB RAM per concurrent agent session on the orchestration tier, scaled by measured trace data.
| Server / GPU class | Model classes served | Best fit | Sizing note |
|---|---|---|---|
| Desk-side 128 GB unified-memory systems (NVIDIA DGX Spark / GB10, Apple Mac Studio, AMD Ryzen AI Max) | Up to ~70B dense or larger MoE models at 4-bit for one user at a time | Model evaluation, prompt and agent development, air-gapped demos | An evaluation tier, not a serving tier: memory bandwidth, not capacity, caps multi-user throughput. |
| NVIDIA L4 / RTX PRO 6000 Blackwell (96 GB) nodes | SLMs and 1B–14B open-weight models at FP8/INT4; 30B–70B at 4-bit on the 96 GB card | Routing tiers, classification, extraction, high-volume light agent steps | Cheapest tokens in the fleet; let the router send the easy majority of requests here. |
| Single L40S (48 GB) server | 7B–14B at FP8/INT8, or 30B-class at INT4 | Pilot deployments, department RAG assistants, single-team agent workloads | The usual entry point for a governed pilot: one inference node plus CPU orchestration. |
| Single H100 80 GB / H200 141 GB server | 70B-class at INT4/AWQ on one H100, or at FP8 on one H200; 8B–34B at full precision with high throughput | Department-to-enterprise production serving via vLLM continuous batching | Measure real tokens/s on your prompt mix — batch throughput, not single-stream speed, sets capacity. |
| 8× H200 / B200 or AMD MI300X-class node | 70B at FP8 with headroom, 100B+ dense and large MoE models across cards via tensor or expert parallelism | Enterprise platforms consolidating many teams; frontier open-weight quality on-prem | Plan a second node before you need it: staging, evaluation, failover, and refresh windows all consume capacity. |
platform + infrastructure amortization + operations + model lifecycle + retrieval lifecycle + governance evidence + support + workflow buildout
| Cost category | Include | Common miss |
|---|---|---|
| Infrastructure and accelerators | GPU or accelerator nodes, CPU orchestration nodes, storage, networking, backup, power, cooling, maintenance, and refresh cycles. | Teams often budget for GPUs but not high-availability control-plane nodes, storage growth, spare capacity, or hardware support. |
| Model operations | Model selection, approval, quantization, benchmarking, security review, version pinning, rollback, and refresh. | Model refresh and evaluation maintenance are recurring costs, not one-time implementation tasks. |
| Retrieval and corpus operations | Connectors, ingestion pipelines, document permissions, embeddings, index rebuilds, reranking, citation quality, and corpus drift monitoring. | Private RAG cost grows with document churn, permission complexity, and quality requirements. |
| Governance and compliance | AI inventory, risk classification, human oversight, retention, audit evidence, model-risk review, and control mapping. | Audit-ready evidence requires structured logs, decision receipts, and review workflows from day one. |
| Security and platform operations | Secrets management, IAM integration, network policy, vulnerability management, SIEM export, incident response, and admin training. | Air-gapped support needs offline artifact handling, local package mirrors, and maintenance-window coordination. |
| User adoption and workflow buildout | Agent design, tool integration, acceptance testing, prompt and policy iteration, change management, and support. | The largest ROI usually comes from governed workflows, not from exposing a raw chat box. |
Azure OpenAI private endpoints, AWS Bedrock private networking, and similar managed cloud patterns can be good choices for many workloads. They are not the same as a customer-controlled on-prem or air-gapped data plane. The meaningful comparison is not "cloud bad, on-prem good." It is which control plane can prove the requirements your risk team actually has.
| Dimension | Governed private AI | Managed cloud AI | Generic on-prem LLM hosting |
|---|---|---|---|
| Data plane location | Prompts, documents, embeddings, outputs, and logs remain inside the customer perimeter or private tenancy. | Traffic can use private networking, but inference and service operations still run in the cloud-provider or model-provider environment. | May keep inference local but often lacks governed retrieval, audit, and policy layers. |
| Model control | Approved local models, licensed models, and optional external models can be routed by policy. | Model choice is tied to the provider catalog and regional availability. | Usually supports a local model server, but workflow portability and model governance vary widely. |
| Governance enforcement | Identity, router policy, tool permissions, approval gates, and audit capture are enforced during execution. | Governance often depends on cloud IAM, service logs, and application code around the model API. | Frequently policy-by-documentation rather than policy enforced by the runtime. |
| Private RAG | Retrieval, embeddings, permissions, source citations, and index lifecycle stay under customer control. | Can be private-networked, but content may still be processed by managed services outside the enterprise data center. | Often has a vector store but weak permission inheritance, citation evidence, or prompt-injection controls. |
| Auditability | Every request can produce a decision receipt: actor, sources, route, tools, approvals, answer, and evaluation signals. | Logs exist, but reconstructing a regulated workflow may require joining cloud, app, model, and security logs. | System logs may exist without AI-specific traces or evidence suitable for auditors. |
| Cost predictability | Higher setup effort, but predictable platform and infrastructure planning for heavy usage. | Fastest start, but production token, storage, egress, and monitoring costs can scale with usage. | Local inference cost can be predictable, but missing governance layers create hidden implementation work. |
A local LLM endpoint does not solve identity, retrieval permissions, tool safety, approval, observability, or audit requirements.
Design the control plane first: identity, policy, router, RAG, tools, approvals, telemetry, and evidence.
Answers degrade when indexes are stale, permissions change, source documents move, or old policies remain embedded in retrieval results.
Track source freshness, rebuild cadence, document deletion workflows, evaluation sets, and retrieval quality metrics.
Synthetic tokens-per-second tests miss retrieval latency, reranking, tool calls, long context, approvals, and batch jobs.
Benchmark representative workflows end to end, then reserve capacity for peak concurrency and maintenance windows.
Plugins, telemetry, package updates, model downloads, or fallback APIs can create data paths that security teams did not approve.
Make all egress explicit, deny by default, log every route, and support fully offline update processes for restricted environments.
Sparse logs cannot prove what happened; excessive raw prompt logging can create unnecessary sensitive-data exposure.
Define retention, redaction, access control, and audit fields before production. Log enough to reconstruct decisions safely.
High-impact workflows can move from recommendation to action without the review, accountability, or segregation of duties required by policy.
Classify workflows by risk and require human approval for external actions, irreversible writes, regulated outputs, and exception cases.
The safest rollout starts narrow and evidence-rich: one or two sensitive workflows, real identity permissions, real retrieval sources, real audit events, and a clear route to production operations. A generic chatbot pilot rarely exposes the controls that matter later.
Inventory the first workflows, owners, data sources, user groups, model sensitivity, expected actions, regulatory obligations, and required evidence. Separate low-risk assistants from regulated decision-support workflows.
Decide whether the first deployment is air-gapped, on-premises, private cloud, or hybrid. Document which network paths are allowed, which are blocked, and who can approve exceptions.
Integrate SSO, groups, role mapping, data-source permissions, admin roles, and environment separation before broad user access. AI permissions should inherit enterprise access controls.
Connect approved sources, generate embeddings locally, preserve document permissions, capture citations, define deletion and rebuild procedures, and test retrieval against known answers.
Create a model inventory with owner, version, license, benchmark results, risk status, deployment location, and permitted workload classes. Configure routing rules that enforce those decisions.
Record prompts, retrieval, tool calls, model routes, responses, approvals, and evaluation signals. Export structured events to the SIEM and define retention, redaction, and access policy.
Load test the complete workflow, not only inference. Include concurrent users, retrieval fanout, context length, tool latency, batch jobs, failover, and maintenance windows.
Set cadence for model refresh, vulnerability review, index rebuilds, evaluation updates, policy review, incident drills, and audit evidence sampling.
This architecture aligns with well-known AI risk, LLM security, and inference operations references while adding the governed multi-agent and strict perimeter controls needed for regulated enterprise AI.
Use SAML or OIDC for authentication, map identity-provider groups to platform roles, and enforce permissions across agents, tools, models, retrieval sources, admin actions, and environments. The audit trail should record the authenticated actor and the policy decision for each request. In VDF AI, Entra ID sign-in and group-to-role mapping are built in; Okta and Keycloak connect through an SSO-aware reverse proxy.
The platform runs without outbound internet. Models, containers, patches, documentation, and evaluation artifacts are imported through an inspected offline process. Inference, retrieval, embeddings, logs, and admin actions stay inside the disconnected environment.
The AI platform should export structured events for prompts, retrieval, tool calls, model routes, approvals, responses, failures, and policy decisions. Security teams can retain, search, alert, and investigate those events in their existing SIEM.
No. Enterprise search finds documents. Private RAG retrieves permitted context, passes it to the model, captures citations, enforces source permissions, manages embeddings and indexes, and records the evidence used to produce an answer.
Yes, if policy allows it. A hybrid architecture can route low-risk workloads to approved external models while pinning sensitive workloads to local models. The important requirement is that egress is explicit, policy-gated, and logged.
Match the server class to the model tier the router will actually use: L4/RTX-class nodes for small routed models, a single L40S (48 GB) server for 7B–14B pilot workloads, A100/H100 80 GB servers for 70B-class production serving with vLLM, and 4×–8× H100/H200 nodes when many teams or 100B+ open-weight models consolidate onto one platform. Buy for the sustained utilization floor and let routing absorb peaks — an idle GPU fleet is how the on-prem cost case dies.
Use the two-formula starting point, then benchmark: GPU memory ≈ model parameters × bits per weight ÷ 8 plus KV-cache (≈ 2 × layers × KV heads × head dim × active tokens × bytes), and CPU/VM capacity ≈ 2–4 cores with 8–16 GB RAM per concurrent agent session for orchestration, retrieval and policy checks. Then replace the estimate with a benchmark of the full governed workflow — identity, retrieval, reranking, inference, tool calls, and log export — because agent platforms bottleneck on the workflow, not on raw tokens per second.
Start from representative workflows and measure end-to-end latency, not only model tokens per second. Include concurrency, context length, output length, retrieval fanout, reranking, tool latency, batch jobs, and failover targets before buying production hardware.
At minimum: actor, role, timestamp, request purpose, policy decisions, data sources retrieved, model and version, prompt or redacted prompt, tool calls, approval actions, response, evaluation signals, errors, and retention metadata.
Private cloud networking can reduce exposure to the public internet, but the model service remains cloud-managed. A true on-prem or air-gapped architecture keeps the production data plane, model runtime, retrieval indexes, logs, and governance evidence under customer-controlled infrastructure.
VDF AI can map the layers to your identity provider, network zones, model runtime, private RAG sources, approval policy, SIEM, and audit evidence requirements. Start with the Trust Center or evaluate vendors with the enterprise AI agent RFP checklist.