Self-Hosted RAG
A RAG (retrieval-augmented generation) system grounds LLM answers in your own documents — indexing them into a vector store, retrieving the relevant passages per question, and generating cited answers instead of hallucinations, installed and operated by your own team — in your data center, private cloud, or VPC — instead of consumed as a vendor-managed SaaS, giving you control over the stack, the models, and the upgrade cadence.
Every component of a RAG pipeline has an excellent open-source option — embedders, vector stores, rerankers — which is exactly why self-hosted RAG projects sprawl. The discipline that matters is treating retrieval quality as a measured product, not a pipeline that "works": golden-question sets, per-corpus evaluation, and permission enforcement from day one.
Every component of a RAG pipeline has an excellent open-source option — embedders, vector stores, rerankers, chunkers, evaluation harnesses — and that abundance is precisely why self-hosted RAG projects sprawl. It is possible to spend a quarter selecting components and still not have answered a single user question, because component choice feels like progress and is easy to keep doing.
The discipline that separates working systems from perpetual pipelines is treating retrieval quality as a measured product. That means a golden-question set built from real user questions, per-corpus scoring, and a rule that no component is swapped without a before-and-after number. Teams that adopt this find most of their quality problems are in chunking and metadata rather than in the model or the vector database they spent weeks choosing.
The second discipline is enforcing permissions from the first commit. Retrofitting access control into a working RAG pipeline is disproportionately painful, because it touches ingestion, storage schema, and query path simultaneously. Self-hosting gives you the freedom to design this properly; it also gives you nobody to blame if you do not.
Why teams run their RAG system self-hosted
Built for technical evaluators and platform engineers who want deployment control without vendor lock-in.
You control the stack, not the vendor
A self-hosted RAG system runs where you decide — bare metal, private cloud, or an isolated VPC. You choose the models, the upgrade windows, and the integrations, instead of inheriting whatever the SaaS vendor ships next quarter.
Open-source engines, enterprise wrapper
The building blocks — Ollama, vLLM, llama.cpp, open-weight models — are mature. What separates a production RAG system from a weekend project is the layer above them: access control, audit, observability, and lifecycle management.
No per-seat or per-token meter
Self-hosting replaces usage-metered pricing with infrastructure you already budget for. Teams that rolled out a metered RAG system to thousands of employees routinely find self-hosting cheaper within the first year.
Core capabilities of an enterprise RAG system
Document ingestion & chunking
Index wikis, policies, contracts, and tickets with structure-aware chunking so retrieval returns answers, not fragments.
Hybrid retrieval
Combine vector similarity with keyword and metadata filters — the difference between demo-grade and production-grade accuracy.
Cited, source-backed answers
Every answer links to the source passages, so users can verify and auditors can trace.
Access-aware retrieval
Retrieval respects document permissions per user — the answer engine never becomes a permissions bypass.
What a self-hosted deployment changes
Decide the ops model up front: DIY assembly from open-source parts maximizes flexibility but you own every CVE; a supported self-hosted platform gives you the control without the 2 a.m. pager.
The RAG system should be deployable with your standard tooling — Docker Compose for pilots, Kubernetes with Helm for production — and upgradeable without data migration surprises.
Model flexibility is the point: the stack should serve open-weight models locally and route to any API you explicitly allow, so no single model vendor becomes load-bearing.
The self-hosted RAG system stack
Self-hosted RAG, layer by layer — with the reason each choice holds up under this deployment mode.
| Layer | Typical choice | Why, here |
|---|---|---|
| Embedding model | An open-weight embedder served locally, version-pinned | Pin it. An unpinned embedder that silently updates invalidates your index and the symptom looks like random quality loss. |
| Vector store | Qdrant or Milvus for scale; pgvector to avoid a new system | For most corpora this choice matters far less than chunking strategy, which is where the quality actually comes from. |
| Reranker | A cross-encoder over the top candidates | Usually the single highest-value addition after basic retrieval works — and frequently skipped in favour of a bigger generator. |
| Chunking | Structure-aware splitting with heading and source metadata | Naive fixed-size chunking severs the context that made a passage answerable. This is the most common root cause of poor retrieval. |
| Permissions | ACLs stored per chunk, filtered at query time | Design this before the first index. Adding it later touches ingestion, schema, and query path at once. |
| Evaluation | Golden questions with retrieval and answer scoring | Without it, component swaps are guesses and you cannot tell an improvement from a regression. |
Sizing a self-hosted RAG system
| Profile | Scale | Hardware | What actually binds |
|---|---|---|---|
| Single team | 50–200 users, one corpus under 1M chunks | pgvector on an existing database, embeddings on a shared GPU | Adding a dedicated vector database at this size buys complexity rather than performance. |
| Multi-corpus | Several source systems, 5M+ chunks | Dedicated vector-store node, separate embedding capacity | Reranking cost scales with candidate count and becomes a real latency contributor here. |
| Shared retrieval service | Chat and agents both consuming retrieval | Clustered store with query quotas per consumer | Agents issue far more retrieval calls per unit of work than humans, and will dominate the load. |
Regulations that point to self-hosted
Vendor risk
Takes the SaaS processor behind your RAG system off the vendor-risk register entirely.
GDPR
You are sole controller and processor for everything the RAG system touches — no international transfer analysis.
SOC 2 / ISO 27001
A self-hosted RAG system inherits your existing certified controls and evidence.
IP protection
Proprietary code and documents stay inside the RAG system you operate, never training someone else’s model service.
When self-hosted is the right call — and when it isn’t
Choose self-hosted when
- Your team already operates containerized services and wants the RAG system to be one more well-behaved workload.
- You need to swap models freely — open-weight today, a different engine next quarter — without renegotiating a contract.
- Procurement or security has rejected SaaS AI tools and you need an equivalent capability inside your own environment.
Consider another mode when
- Nobody owns operations → a self-hosted RAG system without an owner becomes shadow infrastructure; consider a supported on-premises deployment with vendor SLAs.
- Your driver is national jurisdiction or classified data → the sovereign and air-gapped variants of this RAG system address those specifically.
Same capability, different deployment mode:
RAG: Self-Hosted vs the alternatives
| Deployment mode | Typical owner | What you gain — and give up |
|---|---|---|
| Self-Hosted (this page) | Platform Engineering Lead | Complete stack and model freedom with no usage meter — in exchange for your team owning operations, CVE response, and the upgrade cadence. |
| On-Premises | CTO / Head of Infrastructure | Maximum physical control and the strongest economics at steady volume — in exchange for owning the hardware, the capacity plan, and the upgrade cycle. |
| Air-Gapped | CISO / Classified Program Lead | Structural security no contract can match — in exchange for moving every model, index, and software update through an offline bundle process. |
| Sovereign | CIO / Chief Data Officer (public sector & regulated EU) | Legal control that survives foreign disclosure orders and sanctions — in exchange for in-country hosting constraints and heavier procurement diligence. |
How to deploy a self-hosted RAG system
- 01
Collect real questions before choosing components
Gather fifty to a hundred questions people actually ask. This becomes your evaluation set and it will change your component choices — usually toward simpler ones than you expected.
- 02
Build the thinnest end-to-end pipeline that works
Get ingestion, embedding, retrieval, and generation working with default choices before optimising anything. A complete mediocre pipeline tells you where the problems are; a perfect embedder in isolation does not.
- 03
Design chunking against your document structure
Split on headings, sections, and semantic boundaries, carrying source and heading metadata into each chunk. Fixed-size splitting is the default and it is where most retrieval quality is lost.
- 04
Add ACL filtering before opening access
Store permissions per chunk and filter at query time. Doing this before the first real users arrive avoids a rewrite touching ingestion, schema, and query path simultaneously.
- 05
Add a reranker and measure the delta
Introduce cross-encoder reranking and score it against your golden questions. If the number does not move, remove it — that discipline is what stops the pipeline accumulating components nobody can justify.
Where self-hosted RAG system projects fail
Component selection as a substitute for progress
Weeks spent comparing vector databases while no user has asked a question. The choice matters far less than chunking and evaluation, and the comparison never conclusively ends.
Fixed-size chunking
Splitting on character count cuts through tables, procedures, and argument structure, so the retrieved passage is missing the part that made it an answer. It presents as hallucination and is not.
An unpinned embedding model
A silently updated embedder produces vectors incompatible with the existing index. Quality degrades gradually and the cause is extremely hard to identify after the fact.
How to evaluate a self-hosted RAG system
Does retrieval enforce per-user document permissions at query time?
Are answers cited to sources, with retrieval quality measurable on your corpus?
Which embedding models are used, and do they run inside your environment?
How does the pipeline handle updates — re-indexing cadence, deletion propagation?
Can the RAG layer serve multiple agents and applications, not just one chatbot?
Self-hosting converts an RAG system from an opex meter into a fixed platform cost: typical enterprises replace per-seat licenses at 500+ users with a flat deployment that costs less than a third as much at scale.
A self-hosted RAG system, on the VDF AI platform
VDF AI’s private RAG layer indexes your corpus inside your perimeter, enforces document ACLs at query time, and serves cited answers to both chat users and agent workflows.
Self-Hosted RAG questions, answered
What is a self-hosted RAG system?
A RAG (retrieval-augmented generation) system grounds LLM answers in your own documents — indexing them into a vector store, retrieving the relevant passages per question, and generating cited answers instead of hallucinations, installed and operated by your own team — in your data center, private cloud, or VPC — instead of consumed as a vendor-managed SaaS, giving you control over the stack, the models, and the upgrade cadence.
Why do enterprises choose a self-hosted RAG system over a cloud service?
A self-hosted RAG system runs where you decide — bare metal, private cloud, or an isolated VPC. You choose the models, the upgrade windows, and the integrations, instead of inheriting whatever the SaaS vendor ships next quarter. Self-hosting converts an RAG system from an opex meter into a fixed platform cost: typical enterprises replace per-seat licenses at 500+ users with a flat deployment that costs less than a third as much at scale.
Which regulations drive self-hosted RAG system adoption?
The most common drivers are Vendor risk, GDPR, SOC 2 / ISO 27001, IP protection. Vendor risk: Takes the SaaS processor behind your RAG system off the vendor-risk register entirely.
Can VDF AI run as a self-hosted RAG system?
Yes. VDF AI’s private RAG layer indexes your corpus inside your perimeter, enforces document ACLs at query time, and serves cited answers to both chat users and agent workflows. VDF AI provides the chunking, reranking, permission filtering, and evaluation harness as a maintained layer over your chosen open-source components, so you keep the flexibility without rebuilding the parts that actually determine retrieval quality.
What actually improves RAG quality most?
Chunking and reranking, in that order — well ahead of the vector database or the generator model. Structure-aware chunking that preserves headings and section boundaries fixes the majority of retrieval misses, and a cross-encoder reranker over the top candidates usually delivers the next largest gain. Both are cheaper than upgrading the model and are routinely skipped in favour of it.
Do you need a dedicated vector database for self-hosted RAG?
Not below roughly a million chunks. pgvector inside a database you already operate and back up is usually the better engineering choice at that scale, because it removes a system from your estate. Qdrant or Milvus start earning their keep when index size, query concurrency, or filtering complexity outgrow what a single database node handles comfortably.
How do you evaluate a self-hosted RAG system?
Score retrieval and generation separately against a golden-question set drawn from real user questions. Retrieval is measured on whether the correct passage appears in the top results; generation on whether the answer is faithful to it. Combining them into one score is what leaves teams tuning prompts for months against what is actually a chunking problem.
How do you add permissions to a RAG system?
Store access control per chunk at ingestion and filter on it at query time, resolved against live group membership. Do it before the first real users arrive — retrofitting permissions touches ingestion, storage schema, and query path at the same time, and it is one of the more painful migrations in this stack.
Related guides and resources
Evaluate your knowledge stack
Find out how a private RAG and retrieval layer would perform on your data — accuracy, latency, governance, and what to fix before you scale.