An AI gateway, also called an LLM gateway, is the proxy between applications and model providers that routes each request, fails over when a provider breaks, enforces token and spend limits and logs usage. Comparing gateways comes down to where each one runs, what it enforces before a call is made, and whether it also governs MCP tool traffic.
What to compare in an AI gateway
Most gateways now share a baseline: an OpenAI-compatible endpoint, many providers behind it, retries and fallbacks, and request logs with token counts. The differences that decide a purchase sit elsewhere.
- Where it runs. On the vendor’s network, in your own cloud account, or on-premises with no outside dependency. This settles whether prompts ever leave your perimeter.
- What it enforces before the call. Token and spend limits, model allowlists, content guardrails and PII redaction only count as controls if they run in the request path.
- Licence and tiers. Several open-source gateways keep specific features, such as semantic caching or cost-based rate limiting, in paid editions.
- Tool traffic. Agents call tools as well as models. Some gateways now proxy or generate MCP servers; others handle model calls only.
- Evidence. What a log records per call, and whether it is enough to reconstruct why a given model answered.
If you are new to the category, start with what an on-premises AI gateway does and come back for the comparison.
AI gateways compared at a glance
Each entry below reflects the vendor’s own documentation or repository, verified September 2026. A dash means the feature was not documented on the pages we checked, not that it is absent.
Where each gateway runs and how it is licensed
| Gateway | Hosting | Source and licence | MCP and tool traffic |
|---|---|---|---|
| Kong AI Gateway | Self-hosted Kong Gateway, or Kong Konnect (managed) | Kong Gateway is Apache 2.0; key AI plugins need AI Gateway Enterprise | AI MCP Proxy: REST-to-MCP conversion and MCP proxying with per-tool ACLs (Enterprise) |
| LiteLLM | Self-hosted (Docker, Terraform for AWS and GCP) or hosted | MIT; enterprise features such as SSO under a commercial licence | MCP gateway |
| Portkey AI Gateway | Self-hosted (Docker, Node.js, Cloudflare Workers), Portkey Cloud, or enterprise private cloud | MIT; some features hosted or enterprise only | MCP gateway with authentication and access control |
| Agent Router (formerly Envoy AI Gateway) | Kubernetes, local, or hosted by Tetrate | Apache 2.0 | One filtered MCP tool catalogue per agent |
| TrueFoundry AI Gateway | SaaS, VPC, on-premises or air-gapped | Vendor product | MCP gateway |
| Cloudflare AI Gateway | Managed, on Cloudflare’s network | Managed service | – |
| Vercel AI Gateway | Managed, callable from any infrastructure | Managed service | – |
| Azure API Management AI gateway | Azure-managed; capabilities vary by tier | Managed service | Exposes REST APIs as MCP servers; passthrough to existing MCP servers |
| Amazon Bedrock AgentCore Gateway | Fully managed on AWS | Managed service | Converts APIs, Lambda functions and services into MCP tools |
| AWS Multi-Provider Generative AI Gateway | Deployed in your AWS account on ECS or EKS | Reference architecture built on LiteLLM | As LiteLLM |
What each gateway enforces
| Gateway | Routing and fallback | Limits and spend | Caching | Guardrails |
|---|---|---|---|---|
| Kong AI Gateway | Load balancing and failover across providers | Token- or cost-based limits per consumer, model or provider (Enterprise) | Semantic cache (Enterprise) | Prompt guards, PII sanitiser, AWS, Azure and GCP guardrail integrations |
| LiteLLM | Retries, fallbacks, load balancing | Spend tracking, budgets, rate limits, virtual keys | Yes | Yes |
| Portkey AI Gateway | Retries, fallbacks, load balancing | – | Simple (open source); semantic (hosted and enterprise) | 50+ guardrails |
| Agent Router | Provider fallback, model name virtualisation | Token limits per team, app or model | – | – |
| TrueFoundry AI Gateway | Latency-based routing and fallback | Cost- or token-based quotas | – | PII filtering, toxicity detection |
| Cloudflare AI Gateway | Retries, model fallback, dynamic routing | Rate limits, spend limits, cost analytics | Identical-request caching | Guardrails, data loss prevention |
| Vercel AI Gateway | Provider ordering, provider and model fallbacks | Budgets per team, project, key or member | Configurable | – |
| Azure API Management AI gateway | Weighted and priority load balancing, circuit breaker | Token limits and quotas per consumer | Semantic caching with Azure Managed Redis | Azure AI Content Safety policy |
| Amazon Bedrock AgentCore Gateway | Model-based routing across providers | – | – | – |
| AWS Multi-Provider Generative AI Gateway | Via LiteLLM | Via LiteLLM | Prompt caching with ElastiCache | Amazon Bedrock guardrails |
Open-source and self-hostable gateways
Kong AI Gateway
Kong extends its API gateway to model and tool traffic. Its documentation describes one consistent API across providers including OpenAI, Anthropic, Azure AI, Amazon Bedrock and Gemini, and the AI Proxy plugin also lists self-hosted backends such as Ollama and vLLM. The Kong Gateway repository is Apache 2.0 and calls itself an API, LLM and MCP gateway. The plugins an enterprise usually wants most are marked “AI Gateway Enterprise” on Kong’s plugin pages: AI Semantic Cache, AI Rate Limiting Advanced, which limits by total, prompt or completion tokens or by cost, and AI MCP Proxy, which needs Gateway 3.12 or later and converts REST APIs into MCP tools with per-tool ACLs. Guardrail plugins include prompt guards, a semantic prompt guard, a PII sanitiser and integrations with AWS Guardrails, Azure Content Safety, GCP Model Armor and Lakera. It runs self-hosted or in Kong’s managed Konnect platform.
LiteLLM
LiteLLM is a Python SDK plus a proxy server that exposes more than 100 LLM APIs in the OpenAI format. The proxy handles retries, fallbacks and load balancing, virtual keys, spend tracking per project and user, budgets and rate limits, guardrails, caching and an MCP gateway, with logging callbacks to tools such as Langfuse and MLflow. The core is MIT licensed, and enterprise features such as SSO sit under a separate commercial licence. It runs from Docker, from Terraform modules for AWS and GCP, or as a hosted service. AWS’s own multi-provider gateway guidance is built on it.
Portkey AI Gateway
Portkey’s open-source gateway is MIT licensed and routes to more than 1,600 models across 45+ providers, including Ollama for local models. It offers automatic retries of up to five attempts, fallbacks, load balancing, more than 50 guardrails, role-based access control and key management. Simple caching is in the open-source edition; semantic caching is listed for the hosted and enterprise versions. It also describes an MCP gateway with authentication, access control and observability. You can self-host with Docker, Node.js or Cloudflare Workers, use Portkey Cloud, or buy the enterprise edition for private cloud on AWS, Azure, GCP, OpenShift or Kubernetes.
Agent Router, formerly Envoy AI Gateway
Envoy AI Gateway has been renamed Agent Router and is now an Agentic AI Foundation project. The project says the code and maintainers are unchanged, and container images, Helm charts, CRDs and the aigw CLI keep their names. It runs on Envoy Proxy and Envoy Gateway and offers one OpenAI-compatible API for hosted providers, self-hosted inference and MCP servers. Features include provider fallback, model name virtualisation, token limits per team, app or model, centralised credentials, OpenTelemetry telemetry and a single MCP tool catalogue filtered per agent. It is Apache 2.0 and runs locally, on Kubernetes, or as Tetrate’s hosted Agent Router Service. Teams that already operate Envoy will find it the most familiar option.
TrueFoundry AI Gateway
TrueFoundry sells its gateway as part of a broader platform. The product page lists access to more than 1,600 models, latency-based routing with fallback, cost- or token-based quotas scoped with metadata filters, guardrails including PII filtering and toxicity detection, an MCP gateway and request-level observability. It is one of the few commercial gateways whose page names SaaS, VPC, on-premises and air-gapped deployment.
Managed gateways from cloud and platform providers
Cloudflare AI Gateway
Cloudflare runs its gateway on its own network. The features page lists caching of identical requests, rate limiting, spend limits by model, provider or custom metadata, dynamic routing by segment, geography or A/B test, guardrails for harmful content, data loss prevention scanning for PII and financial data, token-based authentication, stored provider keys, analytics and logging, and custom costs for negotiated rates. Request retry and model fallback are also documented. It suits teams already on Cloudflare; prompts and responses pass through Cloudflare’s infrastructure.
Vercel AI Gateway
Vercel’s gateway is a managed service that any server, cloud or local environment can call with an API key; Vercel deployments can use OIDC instead. It accepts the AI SDK, OpenAI Chat Completions and Responses, and Anthropic Messages formats. It routes across providers and fallback models and records status, provider, latency, token usage, cost and every routing attempt. Budgets can be set per team, project, API key or member, but the documentation, updated 14 September 2026, notes that budgets cover spend billed through Vercel’s system credentials, that bring-your-own-key spend is metered separately, and that budgets behave as soft caps.
Azure API Management AI gateway
In Azure the AI gateway is a set of capabilities in API Management rather than a separate product, and Microsoft notes that availability varies by service tier. It governs OpenAI Chat Completions and Responses, Anthropic Messages on v2 tiers and Google Vertex AI APIs, with a unified OpenAI-compatible model API in preview. The llm-token-limit policy sets tokens-per-minute limits or quotas per consumer, semantic caching uses Azure Managed Redis, and a content safety policy calls Azure AI Content Safety. Backends support round-robin, weighted, priority and session-aware load balancing with a circuit breaker. Prompts and completions can be logged to Azure Monitor. On the tool side it can expose existing REST APIs as MCP servers, pass traffic through to MCP servers and manage A2A agent APIs.
AWS options
AWS offers two different routes. Amazon Bedrock AgentCore Gateway is a fully managed entry point for agent traffic: it converts APIs, Lambda functions and services into MCP tools, fronts other agents, and routes inference across model providers through a model-based routing endpoint, with inbound and outbound authentication and semantic tool search. The Guidance for Multi-Provider Generative AI Gateway on AWS is a reference architecture you deploy yourself: LiteLLM on ECS or EKS behind AWS WAF and a load balancer, with ElastiCache for settings and prompt caching, RDS for virtual keys, Secrets Manager for provider credentials, Bedrock for guardrails and logs written to S3.
Self-hosted and on-premises versus managed gateways
This is the decision that narrows the list fastest, and it turns on the data path more than on features.
| Consideration | Managed gateway | Self-hosted or on-premises gateway |
|---|---|---|
| Prompt and response path | Through the vendor’s network | Stays on infrastructure you run |
| Air-gapped operation | Not possible | Possible when every model behind it is local |
| Patching and scaling | Done by the vendor | Done by your platform team |
| Provider credentials | Held by the vendor unless you bring your own keys | Held in your own secret store |
| Logs and audit | Vendor console and exports | Straight into your SIEM and retention policy |
| Feature access | Usually the full feature set | Some features gated behind enterprise tiers |
Two points are easy to miss. A self-hosted gateway is not automatically private: if every model behind it is a public API, prompts still leave the building, so gateway placement only pays off alongside local models or approved private endpoints. And logs are sensitive data in their own right, because a gateway that stores prompts and completions holds a copy of everything users asked. Decide retention before you switch logging on. Our guide to air-gapped deployments covers what a fully disconnected setup requires.
How to choose an LLM gateway
- Fix the data path first. Decide which workloads may send prompts to a third-party network and which may not. That single decision removes half the options for regulated teams.
- List controls that must be enforced, not observed. Token limits, spend caps, model allowlists and content checks should block a request before it reaches a model. A dashboard that shows an overspend afterwards is reporting, not control.
- Read the tier labels. For each open-source candidate, confirm whether the features you need, such as semantic caching, cost-based limits, SSO or MCP support, are in the free edition.
- Decide whether the gateway also governs tools. If agents will call MCP servers, choose between a gateway that handles both kinds of traffic and a separate tool registry with its own grants and audit.
- Look at how routing decisions are made. Static rule tables and provider ordering are easy to reason about but drift as models and prices change. Our explainer on how LLM routing works and the cost, latency and quality formula set out the trade-offs.
- Be deliberate about caching. Semantic caching saves money and can serve stale or cross-tenant answers; see the semantic caching trade-offs before enabling it for regulated workloads.
- Model the spend. Estimate what routing, caching and limits would save on your actual traffic with the AI savings calculator before committing to a licence.
How VDF AI fits
VDF AI’s gateway is VDF AI Router, a Docker-packaged service that runs on your VMs, Kubernetes or bare metal and exposes a REST API and a Python SDK. It reaches local models through Ollama and cloud models through OpenRouter. Policy is the first layer of every decision: regulated domains are restricted to approved models, workloads can be pinned to a named model, and organisation-wide allow and deny lists apply before any call. Air-gap mode disables external APIs entirely. Each decision returns the selected model, a readable reason, up to five ordered failover candidates and per-model scores, and the SEEMR learning engine refines routing from observed quality, latency, failures and energy, with a configurable share of traffic, around two per cent, sent to a challenger model.
Budgets and rate limits sit in the request path. The Router does not publish semantic caching, PII redaction or prompt-injection filtering, so teams that need those should add dedicated controls or pick a gateway above that documents them. Tool traffic is handled by a separate component: the MCP gateway grants tools per role and writes tool calls to the same audit trail as routing decisions. It is a commercial product, and on-premises deployments are licensed as an annual capacity pool.
Sources
- Kong AI Gateway documentation
- Kong AI Semantic Cache plugin
- Kong AI Rate Limiting Advanced plugin
- Kong AI MCP Proxy plugin
- Kong Gateway repository
- LiteLLM repository
- LiteLLM proxy documentation
- Portkey AI Gateway repository
- Agent Router, formerly Envoy AI Gateway
- Agent Router repository
- TrueFoundry AI Gateway
- Cloudflare AI Gateway features
- Vercel AI Gateway documentation
- Azure API Management AI gateway capabilities
- Amazon Bedrock AgentCore Gateway
- AWS Multi-Provider Generative AI Gateway guidance