Self-Hosted LLM
An enterprise LLM deployment is the infrastructure for running large language models — open-weight models like Llama, Mistral, and Qwen served through engines like vLLM and Ollama — as a production service for your organization, installed and operated by your own team — in your data center, private cloud, or VPC — instead of consumed as a vendor-managed SaaS, giving you control over the stack, the models, and the upgrade cadence.
Self-hosting LLMs went mainstream through Ollama on laptops; the enterprise version is the same idea with different failure modes: concurrency, VRAM budgeting, model governance, and someone on call. The winning pattern is a routed fleet — several small models plus one large — behind a single API your applications never have to change.
Self-hosting an LLM went mainstream through Ollama on laptops, and that origin story shapes expectations unhelpfully. A single-user model on a workstation is genuinely trivial. The enterprise version is a different system with different failure modes: concurrency, VRAM budgeting, queueing behaviour under load, model version governance, and somebody carrying a pager.
The winning architecture is a routed fleet behind one stable API. Several small models handle the routine majority, one larger model handles the rest, and applications call a single endpoint that never changes when you swap what is behind it. This matters more for self-hosting than for any other deployment mode, because model freedom is the reason you self-hosted — and freedom is worthless if every model change breaks a dozen integrations.
The honest trade-off is CVE ownership. Choosing open-source components means you inherit their vulnerability stream: the serving engine, the CUDA stack, the model gateway, the Python dependency tree. That is manageable with a platform team and genuinely unmanageable without one, which is the real dividing line between self-hosting succeeding and quietly becoming shadow infrastructure.
Why teams run their LLM deployment self-hosted
Built for technical evaluators and platform engineers who want deployment control without vendor lock-in.
You control the stack, not the vendor
A self-hosted LLM deployment runs where you decide — bare metal, private cloud, or an isolated VPC. You choose the models, the upgrade windows, and the integrations, instead of inheriting whatever the SaaS vendor ships next quarter.
Open-source engines, enterprise wrapper
The building blocks — Ollama, vLLM, llama.cpp, open-weight models — are mature. What separates a production LLM deployment from a weekend project is the layer above them: access control, audit, observability, and lifecycle management.
No per-seat or per-token meter
Self-hosting replaces usage-metered pricing with infrastructure you already budget for. Teams that rolled out a metered LLM deployment to thousands of employees routinely find self-hosting cheaper within the first year.
Core capabilities of an enterprise LLM deployment
Open-weight model serving
Serve Llama, Mistral, Qwen, and domain models on your own GPUs with vLLM-class throughput — models you possess, not endpoints you rent.
LLM routing
Route each request to the cheapest capable model instead of sending everything to the largest one — the single biggest lever on inference cost.
Fine-tuning on your data
Adapt open-weight models to your terminology and tasks with data that never leaves your environment.
Evaluation and benchmarking
Measure model quality on your actual workloads with audit-grade reports before and after every model change.
What a self-hosted deployment changes
Decide the ops model up front: DIY assembly from open-source parts maximizes flexibility but you own every CVE; a supported self-hosted platform gives you the control without the 2 a.m. pager.
The LLM deployment should be deployable with your standard tooling — Docker Compose for pilots, Kubernetes with Helm for production — and upgradeable without data migration surprises.
Model flexibility is the point: the stack should serve open-weight models locally and route to any API you explicitly allow, so no single model vendor becomes load-bearing.
The self-hosted LLM deployment stack
Self-hosted LLM, layer by layer — with the reason each choice holds up under this deployment mode.
| Layer | Typical choice | Why, here |
|---|---|---|
| Serving engine | vLLM for production concurrency; Ollama for developer machines | They solve different problems. Ollama's ergonomics do not translate to multi-user serving, and conflating the two is the most common sizing mistake. |
| Gateway | An OpenAI-compatible API layer in front of the fleet | Compatibility is what preserves model freedom: applications keep working when you change the model behind the endpoint. |
| Model management | Versioned weights in local object storage | Knowing exactly which model version served a given response is a governance requirement long before it is an engineering convenience. |
| Orchestration | Kubernetes with Helm for production; Compose for pilots | The deployment should be boring — your standard tooling, your standard upgrade path, no bespoke operational model. |
| Dependency hygiene | Pinned images with a scheduled CVE review | This is the recurring cost of self-hosting. Budget the time explicitly or it gets skipped until an audit finds it. |
Sizing a self-hosted LLM deployment
| Profile | Scale | Hardware | What actually binds |
|---|---|---|---|
| Developer/pilot | A handful of users, experimentation | A single workstation-class GPU | Comfortable and misleading — nothing here predicts behaviour at fifty concurrent users. |
| Team service | 100–500 users behind one API | 2 GPUs with continuous batching enabled | Batching is the difference between a shared service and a queue people give up on. |
| Platform | Multiple applications and teams as consumers | 4–12 GPUs, autoscaled, with quota per consumer | Without per-consumer quotas, one badly behaved batch job starves every interactive user. |
Regulations that point to self-hosted
Vendor risk
Takes the SaaS processor behind your LLM deployment off the vendor-risk register entirely.
GDPR
You are sole controller and processor for everything the LLM deployment touches — no international transfer analysis.
SOC 2 / ISO 27001
A self-hosted LLM deployment inherits your existing certified controls and evidence.
IP protection
Proprietary code and documents stay inside the LLM deployment you operate, never training someone else’s model service.
When self-hosted is the right call — and when it isn’t
Choose self-hosted when
- Your team already operates containerized services and wants the LLM deployment to be one more well-behaved workload.
- You need to swap models freely — open-weight today, a different engine next quarter — without renegotiating a contract.
- Procurement or security has rejected SaaS AI tools and you need an equivalent capability inside your own environment.
Consider another mode when
- Nobody owns operations → a self-hosted LLM deployment without an owner becomes shadow infrastructure; consider a supported on-premises deployment with vendor SLAs.
- Your driver is national jurisdiction or classified data → the sovereign and air-gapped variants of this LLM deployment address those specifically.
Same capability, different deployment mode:
LLM: Self-Hosted vs the alternatives
| Deployment mode | Typical owner | What you gain — and give up |
|---|---|---|
| Self-Hosted (this page) | Platform Engineering Lead | Complete stack and model freedom with no usage meter — in exchange for your team owning operations, CVE response, and the upgrade cadence. |
| On-Premises | CTO / Head of Infrastructure | Maximum physical control and the strongest economics at steady volume — in exchange for owning the hardware, the capacity plan, and the upgrade cycle. |
| Air-Gapped | CISO / Classified Program Lead | Structural security no contract can match — in exchange for moving every model, index, and software update through an offline bundle process. |
| Sovereign | CIO / Chief Data Officer (public sector & regulated EU) | Legal control that survives foreign disclosure orders and sanctions — in exchange for in-country hosting constraints and heavier procurement diligence. |
| Private | CISO / Data Protection Officer | The fastest route to confidential AI — in exchange for a boundary defined by tenancy and contract rather than by a building you own. |
How to deploy a self-hosted LLM deployment
- 01
Name the owner before the architecture
Decide which team operates this and carries the pager. Self-hosted infrastructure without a named owner becomes shadow infrastructure, and that outcome is a people problem no architecture will fix.
- 02
Put the compatible gateway in first
Stand up the API layer before any application integrates. If teams call the serving engine directly, model freedom disappears the moment a model name is hard-coded in production code.
- 03
Benchmark concurrency, not single-request speed
Measure throughput and latency at your target concurrent user count. Single-request benchmarks on an idle GPU are reassuring and tell you almost nothing about production behaviour.
- 04
Add routing before the second application
Introduce policy-based routing early so cheap requests reach small models by default. Retrofitting this after several integrations exist means renegotiating with each of their owners.
- 05
Schedule the maintenance you have taken on
Put CVE review, dependency updates, and model refresh on a calendar with an owner. The reason self-hosted deployments decay is not technical difficulty — it is that nobody scheduled the upkeep.
Where self-hosted LLM deployment projects fail
Sizing from a single-user benchmark
Numbers from Ollama on a workstation do not survive contact with concurrency. KV cache growth means the tenth simultaneous user behaves nothing like the first.
Applications calling models directly
Hard-coded model names spread across services turn every model change into a coordinated release. The freedom that justified self-hosting is lost by the third integration.
Unowned dependency drift
Without a scheduled review, the serving stack accumulates known vulnerabilities. This is usually discovered by a security audit rather than by the team running it.
How to evaluate a self-hosted LLM deployment
Which open-weight models does the stack serve today, and how fast can you adopt new ones?
Is there a routing layer, or does every request pay flagship-model prices?
What GPU footprint does your workload actually need once routing and quantization are applied?
How are model updates tested — is there an evaluation harness with your data?
Can inference logs feed your observability and audit stack?
Self-hosting converts an LLM deployment from an opex meter into a fixed platform cost: typical enterprises replace per-seat licenses at 500+ users with a flat deployment that costs less than a third as much at scale.
A self-hosted LLM deployment, on the VDF AI platform
VDF AI ships the serving, routing, fine-tuning, and evaluation layers as one platform — the Self-Evolving Model Router picks the cheapest capable model per request, on your hardware.
Self-Hosted LLM questions, answered
What is a self-hosted LLM deployment?
An enterprise LLM deployment is the infrastructure for running large language models — open-weight models like Llama, Mistral, and Qwen served through engines like vLLM and Ollama — as a production service for your organization, installed and operated by your own team — in your data center, private cloud, or VPC — instead of consumed as a vendor-managed SaaS, giving you control over the stack, the models, and the upgrade cadence.
Why do enterprises choose a self-hosted LLM deployment over a cloud service?
A self-hosted LLM deployment runs where you decide — bare metal, private cloud, or an isolated VPC. You choose the models, the upgrade windows, and the integrations, instead of inheriting whatever the SaaS vendor ships next quarter. Self-hosting converts an LLM deployment from an opex meter into a fixed platform cost: typical enterprises replace per-seat licenses at 500+ users with a flat deployment that costs less than a third as much at scale.
Which regulations drive self-hosted LLM deployment adoption?
The most common drivers are Vendor risk, GDPR, SOC 2 / ISO 27001, IP protection. Vendor risk: Takes the SaaS processor behind your LLM deployment off the vendor-risk register entirely.
Can VDF AI run as a self-hosted LLM deployment?
Yes. VDF AI ships the serving, routing, fine-tuning, and evaluation layers as one platform — the Self-Evolving Model Router picks the cheapest capable model per request, on your hardware. VDF AI provides the OpenAI-compatible gateway, routing, and model lifecycle management above your chosen serving engine, so you keep the open-source flexibility without hand-building the governance and upgrade layer that turns it into a platform.
What is the difference between Ollama and vLLM for self-hosting an LLM?
They target different situations. Ollama optimises for a single user on one machine with excellent ergonomics; vLLM optimises for many concurrent users with continuous batching and high GPU utilisation. Using Ollama for a multi-user service is the most common self-hosting mistake, because it works well enough in testing to hide the problem until real load arrives.
What does it actually cost to operate a self-hosted LLM?
Hardware plus a real share of a platform engineer. The infrastructure cost is predictable and usually favourable against per-token pricing at volume; the cost people omit is ongoing operations — CVE review across the serving and CUDA stack, dependency updates, capacity monitoring, and on-call. Teams with an existing platform function absorb this easily, and teams without one usually should not self-host.
How do you avoid vendor lock-in when self-hosting models?
Put an OpenAI-compatible gateway in front of the fleet before any application integrates, and never let services call a model by name directly. That single boundary is what lets you swap open-weight models, change serving engines, or add a hosted model later without touching application code — which is the freedom self-hosting was supposed to buy.
How many concurrent users can one GPU serve?
It depends far more on context length than on user count, because the KV cache grows with both. A mid-size quantized model on a 48 GB card with continuous batching typically handles a few dozen concurrent short-form requests comfortably, while the same setup handles only a handful of long-document requests. Benchmark at your real context sizes.
Related guides and resources
Calculate your AI infrastructure savings
Model the cost and energy impact of running AI on-prem versus cloud-only — then see the benchmark data behind the numbers.