AI Gateway
One Control Point for Every Model Call
An AI gateway is a single control point between your applications and the language models they call: it selects an approved model for each request, enforces policy before the call is made, applies rate limits and budgets, and records every decision for audit and cost accounting. VDF AI runs that control point inside your perimeter, on your hardware, with external endpoints optional and air-gap mode available.
Local open-weight models on your GPUs, approved external endpoints when policy allows.
One REST endpoint your existing applications can point at today.
Bring your model list and your policy constraints. We route against both.
How model traffic gets out of hand
- Each team wired its own provider SDK, so nobody can say which models the company uses
- The model name is hard-coded in a dozen repositories and changing it is a release
- Spend arrives as one invoice a month after the decisions that caused it
- A regulated workload reached a model nobody approved, and the log cannot prove otherwise
What the gateway changes
- One endpoint in front of every model: VDF AI Router
- Policy evaluated before routing, so approval cannot be optimised away
- Budgets and rate limits applied in the request path, per workload
- The decision, the candidates and the scores recorded for every call: how routing decisions are scored
- Tool calls governed alongside model calls through the MCP tool registry
Provider SDKs in Every Repository
or One Governed Gateway
Direct SDK calls
- Model choice frozen at the moment someone wrote the code
- Keys distributed to every application that needs one
- No shared view of spend until the invoice
- An outage at one provider stops the feature
- Approval lists live in a policy document, not the request path
- Audit means grepping application logs
AI gateway
- Model chosen per request against quality, cost, latency and sensitivity
- Credentials held once, at the gateway
- Cost and energy accounted per call, per workload
- Ordered failover to the next capable model
- Allow and deny lists enforced before any call is made
- One audit trail covering every model decision
Decide. Enforce.
Account.
Decide per request, not per repository
Each call is scored against the models available to it — local open-weight models on your own GPUs and any external endpoints policy permits — and returns an ordered list, so a failure falls through to the next capable model instead of the feature going down.
Enforce policy before the call
Mark a domain as regulated and only approved models are ever considered. Pin a workload to a named model, keep organisation-wide allow and deny lists, and disable external endpoints entirely in air-gap mode. Policy is the first layer of the pipeline, so cost optimisation cannot overrule it.
Account for what it cost
Rate limits and budgets apply in the request path, per workload. Every call is written with the model chosen, the candidates rejected, the scores behind the decision and the energy used, which is what turns an AI bill into a reviewable line item.
Three ways to put a gateway in front of your models
| Capability | Hosted AI gateway | Proxy you build | VDF AI Router |
|---|---|---|---|
| Prompts traverse a third party | Yes, by design | No | No |
| Runs air-gapped | No | If you build it | Yes, external APIs disabled |
| Policy evaluated before routing | Varies | Your code | First layer of the pipeline |
| Routing improves as models change | Vendor-managed | Manual rule table | Learned from outcomes |
| Per-call audit with rejected candidates | Rarely | Your code | Standard |
| Engineering cost to keep it running | Low | Continuous | Low, capacity-licensed |
Eight questions to ask before you buy an AI gateway
Gateway buyers are usually a platform team and a risk function at the same table. These are the questions that separate a routing library from a control point. The scoring method behind our answers is set out in the cost, latency and quality formula.
- Where does the gateway itself run?A gateway hosted by the vendor sees every prompt you send through it. If the control point is outside your perimeter, so is the traffic it inspects.
- Can policy override the routing decision?Cost optimisation must never move a regulated workload to an unapproved model. Ask to see the order of evaluation, not the marketing diagram.
- What happens when the chosen model is down?A single model choice is a single point of failure. A gateway should return an ordered fallback list, not one bet.
- Which models can it actually reach?Local open-weight models served on your own GPUs, and approved external endpoints. If the answer is one vendor catalogue, it is a client library, not a gateway.
- Is every call recorded with its reason?For an audit you need the request, the model chosen, the candidates rejected and why — not a latency chart.
- How are spend and rate limits enforced?Budgets and rate limits belong in the request path, applied before the call is made, not reconciled from an invoice a month later.
- Does it work with no internet at all?Air-gapped deployment is a yes or a no. Ask whether external calls are disabled by configuration or by architecture.
- What does integration cost your teams?If every application has to adopt a new SDK, adoption stalls. A REST endpoint your existing clients can point at is the difference between a pilot and a platform.
The gateway by deployment mode, and what sits around it
The decision layer is the same everywhere. What changes is which endpoints it may reach and where the models themselves run.
- On-Premises LLMArchitecture, controls and trade-offs for this mode.
- Self-Hosted LLMArchitecture, controls and trade-offs for this mode.
- Sovereign LLMArchitecture, controls and trade-offs for this mode.
- Air-Gapped LLMArchitecture, controls and trade-offs for this mode.
- VDF AI RouterThe gateway itself: routing pipeline, learning engine and API.
- MCP gateway and tool registryThe same governance applied to the tools agents call.
- AI governance platformWhere gateway decisions become audit evidence.
- AI savings calculatorModel the spend before you commit to a routing policy.
Questions about AI gateways
What is an AI gateway?
An AI gateway is a single control point between your applications and the language models they call. It selects an approved model for each request, enforces policy before the call is made, applies rate limits and budgets, returns a fallback order when a model fails, and records every call for audit and cost accounting. It is the equivalent of an API gateway for model traffic, and the place where model choice stops being hard-coded in application code.
What is the difference between an AI gateway and an API gateway?
An API gateway routes a request to the one service that can answer it, and its decisions are deterministic: path, method, version. An AI gateway chooses between many models that can all answer, trading quality, cost, latency, sensitivity and energy against each other, and that choice changes as models and prices change. The two also fail differently — an API returns an error, a model returns a plausible answer at the wrong quality or the wrong price. They coexist: the API gateway governs service traffic, the AI gateway governs model traffic.
Is an AI gateway the same thing as an LLM gateway?
In practice the two terms describe the same layer, and vendors use them interchangeably. "LLM gateway" emphasises language-model traffic specifically; "AI gateway" is the broader term now used to cover model calls, tool calls over MCP and agent-to-agent traffic. At VDF AI, that layer is VDF AI Router, and it governs model traffic for chat, agents and any internal application that points at it.
Can an AI gateway run on-premises or air-gapped?
Yes. VDF AI Router is a Docker-packaged service that runs on your VMs, Kubernetes or bare metal, next to the GPUs serving your models. In air-gap mode external APIs are disabled entirely and routing is restricted to local models, so no bytes leave the network. This is the deployment most regulated buyers need and the one cloud-hosted gateways cannot offer, because their control plane is the product.
How does an AI gateway reduce inference cost?
Most spend goes on flagship models answering questions a smaller model would answer just as well. A gateway scores each request and sends it to the cheapest model that still meets the quality bar, keeping expensive models for the work that earns them. Typical VDF AI deployments see a 40 to 60 per cent reduction in inference cost, with budgets and rate limits enforced in the request path so a runaway job cannot spend a quarter of the budget overnight.
Does the gateway govern MCP tools as well as models?
Model traffic and tool traffic are governed as one path, but by two components: VDF AI Router decides which model answers, and the MCP tool registry decides which tools an agent may call, with grants assigned per role by an administrator. Both write to the same audit trail, so a reviewer can reconstruct which model answered a request and which tools it used.
Calculate your AI infrastructure savings
Model the cost and energy impact of running AI on-prem versus cloud-only — then see the benchmark data behind the numbers.