How to Measure Private RAG Accuracy: An Enterprise Evaluation Framework

Most private RAG deployments are assessed the same way: someone senior asks a dozen questions they know the answers to, the answers look right, and the project moves to rollout. Six months later a business unit reports that the assistant “gets things wrong sometimes”, and nobody can tell whether the system degraded, the corpus changed, or it was always this accurate and nobody measured.

The fix is not exotic: a fixed evaluation set, a few metrics that separate distinct failure modes, and the discipline of running them on every change. What follows is that practice adapted for regulated, on-premises deployments — where the evaluation data is itself sensitive, and where “how accurate is it” is a question a risk committee will eventually ask in writing.

Two systems, two failure modes

A RAG pipeline fails in two places, and treating it as one system is the root of most wasted optimisation.

Retrieval failure. The passage that answers the question was never put in front of the model. Nothing downstream can recover from this — a well-behaved model will correctly say it does not know, and a poorly-behaved one will invent something. The levers are chunking strategy, the embedding model, metadata filters, the reranker, and whether the document was ingested at all.

Generation failure. The right passage was retrieved and the answer still misrepresents it — an unsupported claim, a wrong number, a citation pointing at something that does not say what the sentence says. The levers here are the model, the prompt, and the citation mechanics.

Measure them separately, because a system can score respectably end-to-end while being badly broken at one layer: a pipeline that retrieves the wrong passage and faithfully summarises it produces a fluent, well-cited, wrong answer, and scores well on any metric that only checks whether the answer matches its sources.

Build the evaluation set from real questions

The evaluation set is the asset. Everything else is arithmetic.

Collect real queries. Take them from what the business actually asks — support tickets, internal help channels, an existing search system’s query log, or a two-week capture from the pilot. Invented questions are systematically easier, because whoever writes them already knows how the corpus is organised.

Label the expected source. For each question, record the document and ideally the passage that should answer it. This label is what makes retrieval measurable — the expensive part, worth doing properly with a subject-matter expert rather than approximately with a script.

Include the questions that should fail. Every set needs unanswerable questions — things the corpus does not cover, and things just outside the user’s permissions. A system that never says “I don’t know” is not being accurate; it is being confident.

Segment by workload. Legal wording lookup, HR policy questions, field procedures and finance queries have different corpora, users and tolerances. One blended set hides the workload that is failing.

Keep it inside the boundary. The set holds real questions and real passages from the organisation’s most sensitive documents. It belongs in the same environment as the pipeline, under the same access controls, versioned like code.

Retrieval metrics that matter

Four numbers, of which one dominates.

• Recall@k — for how many questions does the correct passage appear in the top k results? This is the primary metric, because it is the ceiling on everything downstream. Choose k to match what actually reaches the model after reranking, not an aspirational number.
• Precision@k — how much of what was retrieved is relevant. Low precision wastes context budget and dilutes the model’s attention.
• MRR, and nDCG where several passages are partially relevant — both capture position rather than presence, which is what moves when you add or change a reranker.

Track recall@k per workload and per document source: aggregates conceal the common pattern of excellent retrieval over the well-structured policy repository and poor retrieval over the scanned PDFs nobody wanted to discuss during ingestion.

Generation metrics that matter

• Groundedness (faithfulness). Is every claim in the answer supported by a retrieved passage? This is the metric that maps most directly to the risk a regulated organisation is trying to control.
• Citation correctness. Does each citation point at a source that actually contains the claim? A citation that looks right and is not is worse than no citation, because it defeats the reviewer’s check.
• Answer relevance. Does the answer address the question asked, rather than an adjacent one the retrieved passages happen to cover.
• Refusal correctness. On the unanswerable questions, does the system decline rather than improvise? Report this separately — it is usually the metric that moves most when a model is swapped.

Permission correctness is an accuracy metric

In a multi-tenant or multi-department deployment, an answer that is factually perfect and drawn from a document the user is not entitled to see is a failure — an incident, in fact, not a quality issue. It belongs in the same test run as everything else.

Build a permission test set: for each of several user roles, questions whose answers exist in the corpus but outside that role’s entitlement. The expected result is a refusal or an answer drawn only from permitted sources. Run it on every change to the index, the metadata filters or the access policy, and treat any leak as a blocking failure. It catches the most damaging class of regression, because a retrieval regression produces complaints while a permission regression produces nothing at all until someone notices.

Who judges, and how

Human review is the reference standard and does not scale to every run, so use two tiers. Automated judging handles the full set on every change, using a locally hosted model with a fixed rubric and a pinned judge version — pinning matters, because changing the judge changes the scores without changing the system. Human review then covers a fixed subset scored by a subject-matter expert each cycle, to check that the automated judge still agrees with informed human opinion. When agreement drifts, the rubric or the judge is the problem, not the pipeline.

Keeping the judge local is not a purity argument: automated judging feeds it both the question and the retrieved source passages, so it processes strictly more sensitive material than the pipeline itself.

Run it as a regression suite

The value of the set comes from running it on every change that could plausibly move a number, one change at a time:

• Chunking strategy or size
• Embedding model — requires a full re-index, and is the change most likely to move recall sharply in either direction
• Reranker addition, removal or version
• Retrieval parameters, including k and any similarity thresholds
• Generation model upgrades, which belong under the same model governance process as any other version change
• Prompt edits and corpus refreshes, including scheduled re-ingestion

Record every run against the pipeline version, the model versions and the index build. That record turns “the assistant seems worse this month” into a diagnosable question — and it is what internal audit or a supervisor will ask for when they want to know how performance is monitored over time.

What to avoid

Evaluating only on synthetic questions. They measure the corpus’s organisation, not the system’s behaviour under real queries.

Changing several things at once. A new embedding model, reranker and chunk size shipped together produce one number and no information about which change caused it.

Reporting a single accuracy figure. A risk committee is better served by per-workload recall and groundedness with thresholds attached than by a platform-wide percentage.

Letting the set rot, or ignoring the index build date. Labelled passages move as the corpus changes, and a pipeline that scores well against a stale index is measuring yesterday’s answers — which is why synchronising knowledge sources and evaluating them belong to the same operating routine.

How VDF AI supports RAG evaluation

VDF AI runs the whole loop inside the customer’s environment: the corpus, the index, the models and the evaluation set never leave the boundary, so building a realistic test set does not create a new export path for sensitive content. Retrieval is grounded in the customer’s own sources with citations attached to each claim, which makes groundedness and citation correctness checkable rather than estimated. Access policy is enforced at the data layer, so permission tests exercise the same controls production uses. And for the model-side half of the question — comparing candidate local models on domain-specific scenarios with recorded, audit-ready results — the Model Evaluation Suite provides the benchmarking layer that sits alongside pipeline evaluation.