Agentic AI Architecture: The Components of an Enterprise Agent System and an On-Premises Reference Design

Agentic AI architecture is the design of the system that lets AI agents plan and act inside an enterprise: how requests arrive, how an orchestrator divides work between agents, how each agent's runtime calls models, tools and knowledge, and how policy, identity, approvals and audit wrap every step. Its components are channels, orchestration, the agent runtime, a model gateway with serving engines, tools reached through MCP or A2A, permission-aware retrieval, a governance plane, observability and audit, and a security boundary, organised into a control plane and a data plane.

This article covers the components and how they connect. Two neighbouring subjects have their own guides: reusable agent behaviours such as supervisor–worker or reflection are explained in the field guide to agentic design patterns, and the GPUs, network zones and sizing underneath are in the infrastructure blueprint for on-premises AI.

The layers and components of an agentic AI architecture

Whoever builds an enterprise agent system, the same ten parts appear. The table lists them roughly in the order a request meets them.

Orchestration and the agent runtime

The two are often bundled but answer different questions. The orchestrator owns the workflow: it splits a request into tasks, assigns each to an agent, runs some in parallel, enforces budgets and inserts approval or verification steps, as the explainer on agent orchestration describes. The runtime executes one agent’s loop of reasoning, tool calls and observations and keeps its run state; long-term memory is a separate store with its own access rules, covered in the explainer on agent memory. The explainer on agent runtimes and the definition of an agent harness show where frameworks stop and operational software takes over.

The model layer

Agents should never call model providers directly. A gateway between agents and models applies policy (which models may see which data classes), routes each call, fails over when an endpoint is down, enforces quotas and logs the decision; serving engines such as vLLM or SGLang sit behind it on your GPUs. The gateway is also where an air-gap rule becomes enforceable, because it is the only path to any model. The AI gateway page describes this layer for on-premises estates.

Tools and protocols

Three integration styles coexist. The Model Context Protocol connects agents to tools, data and prompts; its specification revision dated 28 July 2026 defines stdio and Streamable HTTP transports and bases authorisation for HTTP servers on OAuth 2.1. The Agent2Agent (A2A) protocol, which has a stable 1.0 specification, lets agents built by different teams or on different frameworks delegate tasks to one another. Plain HTTP tools with typed schemas cover the rest. Both protocols now sit in the Agentic AI Foundation, hosted by the Linux Foundation. Put an MCP gateway in front of tool servers so grants, rate limits and logging live in one place; the MCP explainer and the explainer on agent-to-agent communication cover the protocols. OWASP’s Top 10 for Agentic Applications names insecure inter-agent communication as ASI07, so authenticate every hop.

Knowledge and retrieval

In an agentic system, retrieval is a tool the agent calls when it needs evidence, not a fixed first step. The retrieval service must receive the identity of the person the agent serves and return only passages that person may open, with document references. Hybrid keyword and vector search helps with codes and identifiers that embeddings blur. See the enterprise RAG page for the permission model and the patterns for grounding agents in retrieval for agent-side designs.

The governance plane

Governance belongs in a plane the agents cannot modify, enforced at runtime. The May 2026 joint guidance from six national cyber agencies recommends continuous runtime verification of identity and authorisation “using a centralised policy decision point for each request”. The same plane holds the registry. Gartner predicted on 28 April 2026 that an average global Fortune 500 enterprise will have over 150,000 agents in use by 2028, up from fewer than 15 in 2025, and named a centralised agent inventory as one of its six steps against sprawl. The guide to AI agent management platforms covers registries, and the guide to agent permissions covers entitlements.

Observability, audit and the security boundary

Traces explain behaviour; the audit trail proves it. Keep them apart: traces for engineers, sampled and short-lived, and an append-only audit store for evidence, retained under your policy, as the guide to agent audit trails sets out. The security boundary contains failure: code runs in sandboxes such as gVisor, Kata Containers or Firecracker, egress passes one proxy, and each agent tier sits in its own network segment.

Control plane and data plane

The control plane holds definitions, prompts, policies, users, the registry and the model catalogue; the data plane holds the runtime, model serving, tools, retrieval and logs. In an on-premises design both sit inside your perimeter. In hybrid offers, the location of the control plane decides who can change what your agents do, which is why the comparison of deployment models treats it as a separate placement decision.

Following one request through the architecture

The walkthrough below follows a request to assess a flagged payment exception, submitted from a case-management application by an operations analyst. The same path applies to requests from schedules or events.

1. Channels. The request arrives through the internal API with the analyst’s session. Recorded: request ID, caller and time.
2. Identity. The governance plane confirms who is asking and resolves roles from directory groups. Recorded: the roles and entitlements in force.
3. Orchestration. The orchestrator loads the current workflow version and plans the tasks. Recorded: workflow version and plan.
4. Policy. The governance plane confirms that these agents may handle this data class. Recorded: the decision and its reason.
5. Retrieval. An investigation agent fetches the payment history and the relevant procedure, filtered by the analyst’s permissions. Recorded: documents and passages used.
6. Model call. The gateway sends the agent’s reasoning step to an approved local model. Recorded: model, version and routing reason.
7. Tool request. The agent asks to call the sanctions-screening tool, and permissions are checked at that moment. Recorded: tool, arguments, and allow or deny with the reason.
8. Tool execution. The tool runs with a short-lived, scoped credential inside the security boundary. Recorded: result and duration.
9. Verification. A second agent drafts a resolution, and a verification step checks the evidence against the procedure. Recorded: the verification outcome.
10. Approval. Releasing the payment is consequential, so the run pauses for a named approver. Recorded: approver, decision and elapsed time.
11. Action. The approved action executes once, protected by an idempotency key. Recorded: the outcome.
12. Answer and seal. The answer returns to the analyst, and the run is sealed in the audit store. Recorded: trace ID and a hash of the record.

Steps 4, 7 and 10 are where governance acts rather than observes. If any of them lives only in a prompt, the architecture has a gap. The production mechanics behind each row, from timeouts to retention, are in the deployment checklist.

What the public cloud reference architectures assume

All three hyperscalers now publish agentic reference architectures, and they agree on the logical layers.

AWS. The Prescriptive Guidance paper on governing and architecting agentic AI at scale, first published on 23 April 2026, describes an applications layer, an agents layer and three core services: model access, tools and knowledge bases, with observability, security and discoverability spanning all of them. The agents layer is built on Amazon Bedrock AgentCore: Runtime, Memory, Identity, Gateway interceptors for MCP tool calls, a Cedar-based Policy engine and Evaluations, plus a central agent registry tied to MCP gateway access controls.

Microsoft. The Azure Architecture Center’s baseline reference architecture for Microsoft Foundry chat runs Foundry Agent Service in its standard setup. Agent state and chat history go to Azure Cosmos DB, uploaded files to Azure Storage and indexes to Azure AI Search. The customer supplies those resources, “but Foundry Agent Service manages them exclusively”. Outbound traffic from agents is routed through Azure Firewall. A separate guide, dated February 2026, catalogues orchestration patterns: sequential, concurrent, group chat, handoff and magentic.

Google Cloud. The multi-agent reference architecture, first published in September 2025 and given production governance guidance on 9 October 2026, uses a coordinator agent and subagents that talk over A2A, runtimes on Cloud Run, GKE or Agent Runtime in Gemini Enterprise Agent Platform (which Google presented in April 2026 as the evolution of Vertex AI), MCP for tools, Model Armor to screen prompts and responses, and Agent Gateway, Agent Identity and Agent Registry for governance.

Where they differ from an on-premises design is in what they take for granted as a managed service.

None of this makes the cloud designs wrong. They assume the provider’s operators, regions and support model sit inside your trust boundary, which is a decision about deployment more than about architecture.

An on-premises reference design for a regulated enterprise

The same layers, mapped to software you can run yourself. Licences were checked against each project’s repository on 6 November 2026.

The hardware beneath this table (GPU classes, network zones, storage tiers and sizing) is covered by the infrastructure blueprint linked at the top.

Illustrative component list: a self-hosted agent stack for a bank

This list is illustrative and does not describe any customer’s system. It shows how the layers might be filled for a bank automating payment exceptions, KYC refreshes and complaint triage without leaving its own data centres.

• Channels: the case-management and operations applications call agents through an internal API; staff use a chat front end behind single sign-on.
• Orchestration: one versioned workflow per process, each with approval steps before any write and a verification step before any customer-facing output.
• Agents: an intake agent that classifies cases, a research agent with read-only tools that gathers evidence, and a drafting agent that proposes the resolution.
• Models: two or three open-weight models of different sizes on internal GPUs, with routine classification routed to the smallest; no external endpoints are configured.
• Tools: MCP servers for the KYC platform, CRM, a core-banking read replica, sanctions screening and ticketing; the only write tools create notes and proposed adjustments below a value cap.
• Knowledge: policy manuals, procedures and product terms indexed with their access groups; customer documents retrieved only within the case owner’s permissions.
• Governance: a registry entry per agent with owner, risk class and version; approval roles mapped from directory groups; a kill switch drilled on a schedule.
• Evidence: a trace per run, and append-only audit records retained as the bank’s DORA and EU AI Act policies require, from which internal audit assembles evidence packs.

The banking solutions page covers the regulatory side for banks.

Where design patterns fit

Patterns such as planner–executor, supervisor and worker, reflection and routing are behaviours you implement inside the orchestration and runtime layers; they do not add layers. The field guide linked above walks through them with examples, and the platform-level patterns post shows how gateways, approval gates and audit planes compose.

Anti-patterns to avoid

1. Controls written only into prompts. Enforce them in the gateway, the tool layer and the orchestrator instead.
2. One agent holding every tool. Split agents by function so each has a narrow grant set and a small blast radius.
3. A shared service account for all agents. Actions cannot be attributed, and one leaked credential opens every system.
4. Retrieval under the agent’s identity. The agent then shows users documents they could never open themselves.
5. Direct model calls that bypass the gateway. Policy, failover, cost tracking and air-gap rules stop applying to that path.
6. Logs the agent can edit or delete. Evidence must sit where no agent has write access.
7. Tools and models fetched at run time. Pin versions and hashes, and pull from internal registries.
8. A vendor control plane that sees production prompts. Your most sensitive data then leaves the design you just drew.
9. Treating tool output or retrieved text as instructions. OWASP lists agent goal hijack as ASI01 for this reason.
10. Evaluating once, at launch. Every change of model, prompt, tool or data source needs the test set again.

How VDF AI fits

VDF AI supplies the orchestration, agent runtime, model gateway and governance layers of this design and runs wherever the rest of it runs: VDF AI Cloud, your own data centre, a private cloud or VPC, a sovereign cloud, or a fully air-gapped network. VDF AI Networks is the orchestration layer: a visual canvas with fourteen or more node types, including Agent and Multi-Agent nodes (the latter with a planner, a decomposition strategy and resource policies), Tool, MCP Action, Data Source and Webhook nodes, a Human Approval node and a Verification node. Networks start manually, on a schedule, from a webhook or on an event.

VDF AI Agents, the Agent Hub, is where individual agents are built and run: any LLM or SLM per agent, tools from an MCP-powered registry, knowledge sources such as databases, Confluence or Jira, and a per-agent audit log and execution trace. VDF AI Router is the model gateway, returning every routing decision with its reason and candidate list and failing over across an ordered list of candidates.

The governance plane registers every agent, Network and model with an owner, purpose, risk class and version, enforces identity and entitlements from your directory per agent and per tool, and binds the prompt, sources, model, tools and outcome of each run into a hashed decision receipt. Networks, agents, intent templates and tools can each be executed through an API, so other systems call the architecture as a service rather than through a screen.