A definitive enterprise blueprint for running AI inside your perimeter: who can ask, how agents act, which model responds, what knowledge grounds the answer, where inference runs, and what evidence proves the result was governed.
These are the questions procurement, security architecture, and platform teams ask first when evaluating on-prem, air-gapped, and governed private AI deployments.
It is the blueprint for running enterprise AI inside your controlled perimeter: identity, orchestration, model routing, private retrieval, inference, approvals, and audit evidence work together so requests become governed answers without sending prompts, documents, embeddings, or logs to an uncontrolled external data plane.
Use on-premises, private cloud, or air-gapped AI when sensitive data, regulatory duties, operational resilience, latency, data sovereignty, or procurement policy require customer-controlled infrastructure and a verifiable audit trail.
The model runtime is only one layer. A production enterprise platform also needs SSO, RBAC, policy-based model routing, private RAG, tool controls, human approval gates, observability, SIEM export, model lifecycle management, and evidence that auditors can inspect.
A production on-prem LLM architecture is not a single box labeled "model." It is a governed path. The platform must know who is asking, which policy applies, which agents and tools may act, which model is permitted, what private knowledge can be used, where inference runs, and which evidence is retained.
The production data plane should keep prompts, documents, embeddings, model outputs, logs, and approval records inside the enterprise boundary unless a route is explicitly approved, policy-gated, and auditable.
These layers are intentionally separable. Security teams can review identity and network controls, platform teams can operate inference and retrieval, governance teams can inspect policy and approval behavior, and business owners can evaluate answer quality by workflow.
Direct answer: Every request should start with enterprise identity, not a separate AI user store. SAML or OIDC login, group membership, role-based access control, and environment scope decide which agents, tools, models, and knowledge sources the user can reach.
The identity layer is the first control point because every downstream action inherits the actor, team, purpose, data classification, and approval policy. In regulated deployments, this layer should integrate with Entra ID, Okta, Keycloak, LDAP, privileged access management, and the organization's existing joiner-mover-leaver process.
Direct answer: It is the control layer that turns a user request into a governed plan: which specialist agents run, which tools they may call, which data they can retrieve, and where a human must approve the workflow before action is taken.
For simple chat, orchestration can be lightweight. For enterprise work, the orchestrator must coordinate agents, internal tools, retrieval, validation, and escalation. It should enforce least privilege at runtime, separate read-only analysis from write actions, and record every step for operations and audit.
Direct answer: Policy-based model routing selects the right model for each task based on sensitivity, domain, residency, latency, cost, quality, and approval status. High-risk workloads can be pinned to specific local models; external model APIs can be disabled entirely.
Routing is not a convenience feature. It is a governance control. A legal review task, HR policy answer, code analysis request, and public marketing draft may all need different models and logging policies. The router should make those decisions from inspectable policy instead of ad hoc prompt logic.
Direct answer: Private RAG retrieves relevant internal knowledge from customer-controlled stores, generates embeddings locally, enforces document permissions, and returns source-grounded context to the model without copying the corpus to a vendor cloud.
Secure on-prem RAG needs more than a vector database. It needs ingestion controls, source-system permissions, index lifecycle management, citation capture, prompt-injection filtering, and deletion workflows for retention or right-to-erasure requests. The same identity decision that allowed the request must constrain retrieval.
Direct answer: Enterprises should treat LLM inference as an operated service: approved models, repeatable serving stacks, capacity planning, batching, queue management, GPU monitoring, fallback policy, and a release process for model changes.
The runtime can use open-weight models, licensed commercial models, or smaller domain models, served through stacks such as vLLM, NVIDIA Triton, TGI, llama.cpp, or Ollama depending on workload. The architecture should keep workflows portable so model swaps do not rewrite the agent layer.
Direct answer: It should capture the actor, request, policy decisions, retrieved sources, model route, prompts, tool calls, human approvals, response, evaluation signals, and retention metadata in a searchable record that can stream to the enterprise SIEM.
Auditability is the difference between a useful demo and a defensible production system. Security, compliance, and model-risk teams need to reconstruct what happened, why it happened, who approved it, which data influenced it, and whether the same workflow still behaves correctly after model or corpus changes.
The right deployment model is a risk decision before it is an infrastructure decision. A bank, public-sector agency, manufacturer, hospital group, or energy operator may all want private enterprise AI, but their acceptable network paths, update process, and operating model can be very different.
Best for: Defense, intelligence, OT, critical infrastructure, and highly restricted networks.
Trade-off: Maximum isolation and strongest egress control, but updates, model refreshes, vulnerability management, and support access need offline artifact processes.
Best for: Regulated enterprises that want local data control, predictable operating cost, and integration with existing data-center controls.
Trade-off: Strong control and procurement clarity, but capacity planning and GPU operations must be owned by the enterprise or a managed operations partner.
Best for: Teams that need cloud elasticity but still require single-tenant networking, customer-managed identity, region pinning, and private data-plane controls.
Trade-off: Faster infrastructure scaling than data-center hardware, but still depends on cloud-provider services and must be assessed against residency and third-party risk policies.
Best for: Enterprises that want local handling for sensitive work and explicitly approved cloud model access for low-risk workloads.
Trade-off: Can improve model choice and burst capacity, but only works if the router, network policy, logging, and data classification rules make egress explicit and auditable.
Choose on-prem or air-gapped AI when data control, regulatory evidence, network isolation, model sovereignty, or high-volume cost predictability matters more than the speed of a managed public cloud API. Choose hybrid when workloads can be clearly classified and external routes can be governed. Choose cloud-only when risk is low and managed-service speed is the primary constraint.
| Criterion | On-prem or air-gapped | Hybrid private AI | Cloud-only AI |
|---|---|---|---|
| Data sensitivity | Choose on-prem or air-gapped when prompts, documents, embeddings, outputs, or logs include regulated, classified, trade-secret, or customer-confidential data. | Hybrid can work when low-risk content is clearly separable from restricted data and the router can enforce that boundary. | Cloud-only is usually simplest for public, low-risk, or already cloud-native workloads. |
| Regulatory and audit pressure | Best when auditors need direct evidence of identity, policy enforcement, data lineage, model route, approval, and retention inside customer-controlled systems. | Possible when audit records show exactly which requests used external services and why policy allowed them. | Works when cloud contractual controls and logs satisfy your risk team, regulator, and customer commitments. |
| Network isolation | Required for disconnected sites, restricted networks, OT-adjacent environments, and teams that must disable outbound internet. | Works for segmented environments where outbound paths are narrow, approved, inspected, and logged. | Best for internet-connected teams that prioritize managed service speed over isolation. |
| Operating cost profile | Favors predictable high-volume usage where platform, hardware, and operations can be amortized across many workflows. | Useful when utilization is uneven or burst capacity is needed before long-term capacity is proven. | Often cheapest to start, but token, storage, connector, and audit-export costs should be modeled at production volume. |
| Model control | Best when teams need approved local models, model version pinning, rollback, fine-tuned domain models, or no vendor-managed inference dependency. | Works when policy decides which models can handle which tasks and external model use is optional. | Best when a managed model catalog is more important than local model sovereignty. |
Sizing should start from the workflow, not the model leaderboard. A secure on-prem RAG workflow includes identity checks, retrieval, reranking, context assembly, inference, tool calls, policy checks, evaluation signals, and log export. Each step affects latency and capacity.
| Scale | Typical usage profile | Infrastructure shape | Sizing note |
|---|---|---|---|
| Pilot or controlled rollout | 25 to 100 named users; 2 to 10 concurrent conversations; a few governed workflows. | One small GPU node or a private cloud GPU instance for inference, plus CPU nodes for orchestration, retrieval, and observability. | Benchmark the real workflow, not only raw tokens per second. Include retrieval, reranking, tool calls, approvals, and log export in the test. |
| Department production | 100 to 1,000 users; 10 to 75 concurrent sessions; multiple teams and knowledge domains. | Separate runtime pools for chat, RAG-heavy workloads, and batch jobs; redundant vector/search services; central logging and metrics. | Use routing policy to keep small tasks on smaller models and reserve larger models for work that needs deeper reasoning or domain quality. |
| Enterprise platform | Thousands of users; 75+ concurrent sessions; many agents, tools, environments, and regulated workflows. | Multiple GPU pools, queue management, model registry, staging and production environments, HA control plane, SIEM integration, backup and DR. | Capacity planning should include peak concurrency, context length, output length, retrieval fanout, batch jobs, model refresh windows, and failover targets. |
| Air-gapped or restricted network | Often fewer users but higher assurance requirements, slower change windows, and stricter artifact review. | Local model registry, offline update channel, internal package repository, local telemetry store, and internal-only documentation and support runbooks. | Plan the update process as carefully as inference. Model files, containers, vulnerability fixes, and evaluation artifacts all need controlled import. |
platform + infrastructure amortization + operations + model lifecycle + retrieval lifecycle + governance evidence + support + workflow buildout
| Cost category | Include | Common miss |
|---|---|---|
| Infrastructure and accelerators | GPU or accelerator nodes, CPU orchestration nodes, storage, networking, backup, power, cooling, maintenance, and refresh cycles. | Teams often budget for GPUs but not high-availability control-plane nodes, storage growth, spare capacity, or hardware support. |
| Model operations | Model selection, approval, quantization, benchmarking, security review, version pinning, rollback, and refresh. | Model refresh and evaluation maintenance are recurring costs, not one-time implementation tasks. |
| Retrieval and corpus operations | Connectors, ingestion pipelines, document permissions, embeddings, index rebuilds, reranking, citation quality, and corpus drift monitoring. | Private RAG cost grows with document churn, permission complexity, and quality requirements. |
| Governance and compliance | AI inventory, risk classification, human oversight, retention, audit evidence, model-risk review, and control mapping. | Audit-ready evidence requires structured logs, decision receipts, and review workflows from day one. |
| Security and platform operations | Secrets management, IAM integration, network policy, vulnerability management, SIEM export, incident response, and admin training. | Air-gapped support needs offline artifact handling, local package mirrors, and maintenance-window coordination. |
| User adoption and workflow buildout | Agent design, tool integration, acceptance testing, prompt and policy iteration, change management, and support. | The largest ROI usually comes from governed workflows, not from exposing a raw chat box. |
Azure OpenAI private endpoints, AWS Bedrock private networking, and similar managed cloud patterns can be good choices for many workloads. They are not the same as a customer-controlled on-prem or air-gapped data plane. The meaningful comparison is not "cloud bad, on-prem good." It is which control plane can prove the requirements your risk team actually has.
| Dimension | Governed private AI | Managed cloud AI | Generic on-prem LLM hosting |
|---|---|---|---|
| Data plane location | Prompts, documents, embeddings, outputs, and logs remain inside the customer perimeter or private tenancy. | Traffic can use private networking, but inference and service operations still run in the cloud-provider or model-provider environment. | May keep inference local but often lacks governed retrieval, audit, and policy layers. |
| Model control | Approved local models, licensed models, and optional external models can be routed by policy. | Model choice is tied to the provider catalog and regional availability. | Usually supports a local model server, but workflow portability and model governance vary widely. |
| Governance enforcement | Identity, router policy, tool permissions, approval gates, and audit capture are enforced during execution. | Governance often depends on cloud IAM, service logs, and application code around the model API. | Frequently policy-by-documentation rather than policy enforced by the runtime. |
| Private RAG | Retrieval, embeddings, permissions, source citations, and index lifecycle stay under customer control. | Can be private-networked, but content may still be processed by managed services outside the enterprise data center. | Often has a vector store but weak permission inheritance, citation evidence, or prompt-injection controls. |
| Auditability | Every request can produce a decision receipt: actor, sources, route, tools, approvals, answer, and evaluation signals. | Logs exist, but reconstructing a regulated workflow may require joining cloud, app, model, and security logs. | System logs may exist without AI-specific traces or evidence suitable for auditors. |
| Cost predictability | Higher setup effort, but predictable platform and infrastructure planning for heavy usage. | Fastest start, but production token, storage, egress, and monitoring costs can scale with usage. | Local inference cost can be predictable, but missing governance layers create hidden implementation work. |
A local LLM endpoint does not solve identity, retrieval permissions, tool safety, approval, observability, or audit requirements.
Design the control plane first: identity, policy, router, RAG, tools, approvals, telemetry, and evidence.
Answers degrade when indexes are stale, permissions change, source documents move, or old policies remain embedded in retrieval results.
Track source freshness, rebuild cadence, document deletion workflows, evaluation sets, and retrieval quality metrics.
Synthetic tokens-per-second tests miss retrieval latency, reranking, tool calls, long context, approvals, and batch jobs.
Benchmark representative workflows end to end, then reserve capacity for peak concurrency and maintenance windows.
Plugins, telemetry, package updates, model downloads, or fallback APIs can create data paths that security teams did not approve.
Make all egress explicit, deny by default, log every route, and support fully offline update processes for restricted environments.
Sparse logs cannot prove what happened; excessive raw prompt logging can create unnecessary sensitive-data exposure.
Define retention, redaction, access control, and audit fields before production. Log enough to reconstruct decisions safely.
High-impact workflows can move from recommendation to action without the review, accountability, or segregation of duties required by policy.
Classify workflows by risk and require human approval for external actions, irreversible writes, regulated outputs, and exception cases.
The safest rollout starts narrow and evidence-rich: one or two sensitive workflows, real identity permissions, real retrieval sources, real audit events, and a clear route to production operations. A generic chatbot pilot rarely exposes the controls that matter later.
Inventory the first workflows, owners, data sources, user groups, model sensitivity, expected actions, regulatory obligations, and required evidence. Separate low-risk assistants from regulated decision-support workflows.
Decide whether the first deployment is air-gapped, on-premises, private cloud, or hybrid. Document which network paths are allowed, which are blocked, and who can approve exceptions.
Integrate SSO, groups, role mapping, data-source permissions, admin roles, and environment separation before broad user access. AI permissions should inherit enterprise access controls.
Connect approved sources, generate embeddings locally, preserve document permissions, capture citations, define deletion and rebuild procedures, and test retrieval against known answers.
Create a model inventory with owner, version, license, benchmark results, risk status, deployment location, and permitted workload classes. Configure routing rules that enforce those decisions.
Record prompts, retrieval, tool calls, model routes, responses, approvals, and evaluation signals. Export structured events to the SIEM and define retention, redaction, and access policy.
Load test the complete workflow, not only inference. Include concurrent users, retrieval fanout, context length, tool latency, batch jobs, failover, and maintenance windows.
Set cadence for model refresh, vulnerability review, index rebuilds, evaluation updates, policy review, incident drills, and audit evidence sampling.
This architecture aligns with well-known AI risk, LLM security, and inference operations references while adding the governed multi-agent and strict perimeter controls needed for regulated enterprise AI.
Use SAML or OIDC for authentication, map identity-provider groups to platform roles, and enforce permissions across agents, tools, models, retrieval sources, admin actions, and environments. The audit trail should record the authenticated actor and the policy decision for each request.
The platform runs without outbound internet. Models, containers, patches, documentation, and evaluation artifacts are imported through an inspected offline process. Inference, retrieval, embeddings, logs, and admin actions stay inside the disconnected environment.
The AI platform should export structured events for prompts, retrieval, tool calls, model routes, approvals, responses, failures, and policy decisions. Security teams can retain, search, alert, and investigate those events in their existing SIEM.
No. Enterprise search finds documents. Private RAG retrieves permitted context, passes it to the model, captures citations, enforces source permissions, manages embeddings and indexes, and records the evidence used to produce an answer.
Yes, if policy allows it. A hybrid architecture can route low-risk workloads to approved external models while pinning sensitive workloads to local models. The important requirement is that egress is explicit, policy-gated, and logged.
Start from representative workflows and measure end-to-end latency, not only model tokens per second. Include concurrency, context length, output length, retrieval fanout, reranking, tool latency, batch jobs, and failover targets before buying production hardware.
At minimum: actor, role, timestamp, request purpose, policy decisions, data sources retrieved, model and version, prompt or redacted prompt, tool calls, approval actions, response, evaluation signals, errors, and retention metadata.
Private cloud networking can reduce exposure to the public internet, but the model service remains cloud-managed. A true on-prem or air-gapped architecture keeps the production data plane, model runtime, retrieval indexes, logs, and governance evidence under customer-controlled infrastructure.
VDF AI can map the layers to your identity provider, network zones, model runtime, private RAG sources, approval policy, SIEM, and audit evidence requirements. Start with the Trust Center or evaluate vendors with the enterprise AI agent RFP checklist.