RAG Evaluation: Metrics, a Practical Framework and Open-Source Tools for Retrieval and Generation

Why RAG evaluation splits into two halves

A wrong answer from a RAG system can have two very different causes. Either the retriever never found the passage that contained the answer, or it found it and the model ignored, misread or embellished it. The fixes are different: the first calls for changes to chunking, embeddings, hybrid search or reranking, and the second for changes to the prompt, the model or the context window. An end-to-end “was the answer right?” score cannot tell you which.

So a useful RAG evaluation framework scores each half on its own, then checks the whole. This guide covers the metrics for each half, how to build the dataset they run on, where model-based judging goes wrong, and the tools that automate the work. Two related guides go deeper on neighbouring questions: our framework for measuring private RAG accuracy focuses on the enterprise programme around evaluation, and the comparison of LLM evaluation tools covers general model and application testing beyond RAG.

RAG evaluation metrics for retrieval

Retrieval metrics compare what the retriever returned with what it should have returned. The classic information-retrieval metrics need labelled relevant passages for each question; the LLM-judged variants can work from a reference answer instead.

A few practical notes:

• Choose k to match production. If the generator receives eight passages, report recall@8, not recall@20.
• Label at the passage level where you can. Document-level labels overstate quality when the right document is retrieved but the wrong section of it.
• Use the non-LLM variants when labels exist. Ragas, for example, documents a non-LLM context precision that compares retrieved chunks with reference contexts using string similarity, and an ID-based version that only checks whether retrieved chunk IDs appear in the reference set (checked October 2026). These are cheap, deterministic and easy to explain.
• Test the filters. Permission filtering and metadata filters are part of retrieval. Include questions asked by users who should not see the answer, and count any leaked passage as a severe failure.

For tuning the retrieval layer itself, the guide to embedding models and rerankers explains which component usually moves which metric.

RAG evaluation metrics for generation

Generation metrics take the retrieved context as given and ask what the model did with it.

Ragas computes faithfulness exactly as described in the first row: it extracts the claims in a response, checks whether the retrieved context supports each one, and divides supported claims by total claims, giving a score from 0 to 1. TruLens frames the same ideas as its RAG triad of context relevance, groundedness and answer relevance (both checked October 2026).

Faithfulness and correctness are different things. An answer can be perfectly faithful to an out-of-date policy and still be wrong. That is a content problem, and it shows up only if your test set includes reference answers checked by someone who knows the current rule.

Citation accuracy deserves its own line in any enterprise report. Readers trust cited answers more, so a confident citation to a passage that does not say what the sentence claims is worse than no citation at all.

Building the evaluation dataset

Metrics are only as good as the questions they run on. A dataset that holds up has these properties:

• Real questions first. Pull them from search logs, help desk tickets, chat transcripts and subject-matter experts. Synthetic questions generated from your documents are useful for coverage but tend to be easier than real ones, because they borrow the document’s own wording.
• Reference answers and sources. For each question, record the expected answer and the passage or passages that support it. Domain experts should write or approve these.
• Hard cases on purpose. Include multi-hop questions that need two documents, questions whose answer changed recently, questions with near-duplicate distractor documents, unanswerable questions, and permission-restricted questions.
• Enough volume per segment. A few hundred questions spread across the main content areas gives more useful signal than thousands of near-identical ones. Report by segment, because averages hide weak areas.
• Version control. Store the dataset with a version number, alongside the corpus snapshot it was built against. When documents change, some reference answers will need updating; track that, or scores will drift for reasons unrelated to the system.

Keep a held-out portion that nobody tunes against. Teams that optimise prompts on the full set end up measuring how well they have memorised it.

LLM-as-judge: what to watch

Most generation metrics, and the LLM-judged retrieval metrics, use a language model to score outputs. That scales, but the judge is a model with its own failure modes.

The best-known study of the approach, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Lianmin Zheng and colleagues (2023), examined position, verbosity and self-enhancement biases, along with limited reasoning ability, and found that strong judges reached over 80% agreement with human preferences, about the level humans reach with each other. That is encouraging for broad comparisons and a warning for fine-grained scores. In practice:

• Calibrate before trusting. Have experts label a sample of answers, run the judge on the same sample, and measure agreement. Repeat whenever you change the judge model or prompt.
• Pin the judge. Record the judge model, version, prompt and temperature with every score. A judge upgrade can shift every number in the report.
• Prefer claim-level checks. Asking a judge “is this claim supported by this passage?” is more reliable than asking for an overall score out of ten.
• Avoid self-judging where possible. Using the same model to generate and to judge invites the self-enhancement bias the paper describes.
• Use small classifiers where they fit. Ragas documents a faithfulness variant that replaces the judge LLM with Vectara’s HHEM-2.1-Open, a small open T5-based hallucination classifier that runs on CPU by default.

The ARES framework (Saad-Falcon and colleagues, 2023) takes a related route: it generates synthetic training data, fine-tunes lightweight judge models for context relevance, answer faithfulness and answer relevance, and uses a few hundred human annotations with prediction-powered inference to put confidence intervals on the results. Its repository is Apache 2.0 licensed, though it has seen little recent activity.

A RAG evaluation framework in five steps

1. Define what good means per use case. A policy assistant for staff needs high faithfulness and correct refusals; a research assistant may trade some precision for recall. Write the thresholds down before you measure.
2. Build and version the dataset as described above, with a held-out split.
3. Score components offline. Run retrieval metrics on every change to chunking, embeddings, indexes or filters. Run generation metrics on every change to prompts or models. Because each run is cheap, it can sit in continuous integration.
4. Check end to end, including permissions. Run the full pipeline as different users and score answer correctness, refusals and leaks.
5. Gate releases and monitor production. Block a release when a metric drops below its threshold or regresses beyond an agreed margin. In production, sample live traffic for judge-based faithfulness scoring and route low scores and user complaints to reviewers, whose labels feed back into the dataset.

The framework is the same whichever tool computes the numbers. What makes it work is the dataset, the thresholds and the habit of running it on every change.

RAG evaluation tools

These are the most widely used options for computing RAG metrics. Licences were read from each project’s repository, and features from its documentation, on 29 October 2026.

When to choose which. Ragas is the most RAG-specific of the libraries and the easiest place to start if you want named retrieval and generation metrics. DeepEval suits teams that want evaluations written as unit tests and run in CI. TruLens fits teams that want evaluation tied to traces of each step. Phoenix adds a UI for traces, datasets and experiments that can run inside your network; its licence lets you self-host freely but not offer it to third parties as a managed service. LangSmith suits teams already using the LangChain ecosystem with an Enterprise agreement. Our broader LLM evaluation tools comparison covers several more, including promptfoo, MLflow and Langfuse.

Running RAG evaluation offline or on-premises

Evaluation data is often more sensitive than production traffic. A test set built from real tickets and expert answers concentrates exactly the material you were careful about when you chose a private RAG deployment. Sending it to a cloud judge creates the export path you avoided.

A fully local evaluation stack is practical:

• Retrieval metrics need no model. Recall@k, precision@k, MRR and nDCG are arithmetic over labels, and the non-LLM context precision variants use string similarity.
• Judges can run locally. Serve an open-weight model with Ollama, vLLM or another OpenAI-compatible server and point DeepEval or TruLens at it, or route Ragas through LiteLLM after confirming your endpoint is supported. Use a different model family from the generator where you can. Our guide to the best local LLMs for RAG helps with choosing.
• Classifiers reduce judge load. A small hallucination classifier such as HHEM can score faithfulness on CPU for routine runs, with a larger judge reserved for disputed cases.
• Store results with the corpus. Keep scores, judge settings and dataset versions in the same controlled environment as the documents, so audit and regression analysis use one source.

Calibrate the local judge against human labels exactly as you would a cloud one; a smaller model is not automatically a worse judge, but you need evidence either way.

How VDF AI fits

VDF AI runs retrieval, models and evaluation inside your own environment: on-premises, in a private cloud, air-gapped, or on VDF AI Cloud where policy allows. With local models configured, documents, indexes, test sets, judge prompts and results stay on infrastructure you control.

Private RAG returns answers with citations to the passages used, and retrieval respects each user’s permissions, so groundedness, citation accuracy and permission tests can be run against the same controls production uses. Agents can call checking tools from the tool registry, such as citation verification and answer confidence scoring, as steps in a workflow. For the model side, the Model Evaluation Suite lets domain experts store test scenarios and compare candidate models, including local ones, with timestamped results for audit. The AI Router can send judge calls and generator calls to different models and logs each one, and the enterprise RAG page describes the retrieval architecture these metrics measure.

Deployment and licensing options are on the pricing page.