How to Evaluate a RAG System: Retrieval Metrics, Faithfulness, Answer Quality

To evaluate a retrieval augmented generation (RAG) system, measure it in three separate layers. First, check whether the retriever found the passages that contain the answer. Second, check whether the model's answer stays within what those passages say. Third, check whether the final answer is correct and actually addresses the question. Each layer has its own metrics, and a single overall score hides which layer failed.

A RAG system answers a question in two stages. A retriever searches a document collection and returns the few passages (chunks) that look most relevant. A language model then writes an answer using those chunks as its evidence. Because there are two stages, there are two places where an answer can go wrong, and the repair for each is completely different.

This article explains how to build a test set for RAG, how to compute the standard retrieval metrics with a worked example, and how to measure faithfulness and answer quality with a language model acting as a grader. It ends with the tools that implement these metrics, the release gates that use them, and a method for tracing a wrong answer to the layer that caused it.

Why RAG evaluation is split into layers

A single score such as "the answer was correct 78 percent of the time" tells you that something is wrong but not where. Suppose the score drops to 70 percent after a change. The cause could be a new chunking strategy that split a key paragraph in half. It could also be a new model that ignores the retrieved text and answers from memory. These two problems need opposite repairs, so the evaluation has to separate them.

The three layers map directly onto the pipeline:

• The retrieval layer checks the ranked list of chunks that came back for each question. Its metrics need to know which chunks are relevant to each question.
• The generation layer checks what the model did with the chunks it received. Its main metric is faithfulness, which asks whether every claim in the answer is supported by the retrieved text.
• The end-to-end layer checks the final answer against what the user needed. Its metrics are answer relevance and answer correctness.

The layers depend on each other in one direction. Good retrieval does not guarantee a good answer, but poor retrieval almost always produces a poor answer. If the passage with the answer never reached the model, no change to the instructions or the model can recover it. That is why the diagnosis at the end of this article always starts with retrieval.

For a broader view of evaluating applications built on language models, including agents and the statistics of pass rates, see LLM Evaluation: How to Test an LLM App Before It Ships. This article stays inside RAG and goes one level deeper on each metric.

The running example: a support docs bot

The examples below use one illustrative system throughout. A software company has a help center with 1,200 articles about billing, accounts, and integrations. The articles are split into about 9,000 chunks of roughly 300 tokens each. A token is a short piece of text, usually a word or part of a word, and it is the unit that models read and write.

When a customer asks a question, the retriever embeds the question as a vector, searches the chunk index, and returns the top 5 chunks. The model then writes an answer from those 5 chunks and cites the articles it used. How such a pipeline is built, including chunking, vector databases, hybrid search, and re-ranking, is covered in System Design for RAG. Here the only question is how to measure it.

Step 1: Build a test set with labeled chunks

Every metric in this article needs a fixed set of test questions (a test set, also called an evaluation dataset). For RAG, each test case has up to three parts: the question, the IDs of the chunks that contain the answer, and a short reference answer written by a person. Retrieval metrics need the chunk labels. Answer correctness needs the reference answer. Faithfulness and answer relevance need neither, which makes them useful when labels are scarce.

Where the questions come from

The best questions come from real users. Pull a sample of questions from search logs, support tickets, or chat transcripts, and remove personal data. Real questions contain the misspellings, vague phrasing, and mixed topics that hand-written questions usually lack.

Sample on purpose rather than at random. A random sample of support questions is dominated by a few common topics, such as password resets. A useful test set covers each product area, includes short and long questions, and includes questions that need two articles to answer. It should also include questions that the help center cannot answer at all, because a good system says "I do not know" to those instead of inventing an answer.

Labeling the relevant chunks

For each question, a person who knows the product finds the chunks that contain the answer and records their IDs. This takes time. In the support bot example, labeling 200 questions at about four minutes each is around 13 hours of work, which is a realistic budget for one engineer over a few days.

Two details make the labels last longer. First, label at the level of the article and the passage, not only the chunk ID, because chunk IDs change every time the chunking strategy changes. Store the source text span so that the label can be mapped onto new chunks automatically. Second, record when a question has more than one relevant chunk, because recall metrics depend on the full set of relevant chunks, not just the first one found.

Synthetic questions and their risks

A faster method is to generate questions with a language model (synthetic generation). You give the model a chunk and ask it to write a question that the chunk answers. The chunk then becomes the labeled relevant chunk without any manual labeling, and hundreds of cases can be produced in an hour.

Synthetic questions have three known weaknesses, so use them as a supplement rather than a replacement:

• They copy the wording of the chunk. A generated question tends to reuse the exact terms in the source text, so keyword search finds it easily. Real users describe their problem in their own words, so retrieval scores on synthetic questions are usually higher than on real ones.
• They have exactly one answer chunk. Real questions often need two or three chunks, and synthetic questions rarely test that.
• They miss the questions that matter most. The model writes questions about what the documents already say, so it never produces the questions the documentation fails to answer.

A practical mix for the support bot is 150 real questions with human labels and 300 synthetic questions, reviewed by a person who deletes the unnatural ones. Report the two groups separately, so that a high score on the synthetic group never hides a low score on the real one.

Step 2: Measure retrieval with ranking metrics

Retrieval metrics compare the ranked list of chunks the retriever returned with the labeled relevant chunks. They need no language model, they run in seconds, and they are deterministic, which makes them the cheapest metrics in the whole evaluation. The value k is the number of top results being scored, usually the same number the system passes to the model.

The five standard metrics are:

• Recall at k is the share of all relevant chunks that appear in the top k results. It answers the question "did we find everything the answer needs?"
• Precision at k is the share of the top k results that are relevant. It answers the question "how much of what we passed to the model was useful?"
• Hit rate at k is the share of questions with at least one relevant chunk in the top k. It is the most forgiving metric.
• Mean reciprocal rank (MRR) looks at the position of the first relevant chunk. If it is at position 1 the score is 1, at position 2 it is 1/2, at position 3 it is 1/3, and if no relevant chunk appears the score is 0. MRR is the average of these scores across questions.
• Normalized discounted cumulative gain (nDCG) rewards relevant chunks for appearing near the top, with a discount that grows with position, and then divides by the best possible score. A value of 1.0 means every relevant chunk appears above every irrelevant one.

A worked example

Take three test questions from the support bot, with k = 5. The numbers are illustrative.

Question 1 is "How do I change the card used for my monthly invoice?" Two chunks are relevant, chunk A and chunk B. The retriever returns, in order, an unrelated chunk, chunk A, and three more unrelated chunks. Chunk B is missing.

• Recall at 5 is 1 relevant chunk found out of 2, which is 0.50.
• Precision at 5 is 1 relevant chunk out of 5 returned, which is 0.20.
• The first relevant chunk is at position 2, so the reciprocal rank is 0.50.
• For nDCG, the gain of a relevant chunk at position p is 1 divided by the base-2 logarithm of (p + 1). Chunk A at position 2 gives 1 divided by 1.585, which is 0.631. The best possible ordering puts both relevant chunks at positions 1 and 2, which scores 1 + 0.631 = 1.631. So nDCG at 5 is 0.631 divided by 1.631, which is 0.39.

Question 2 has one relevant chunk, and the retriever returns it at position 1. Recall, reciprocal rank, and nDCG are all 1.0, and precision at 5 is 0.20.

Question 3 has one relevant chunk, and the retriever does not return it in the top 5. Every metric for this question is 0.

Across the three questions, hit rate at 5 is 2 out of 3, or 67 percent. MRR is the average of 0.50, 1.0, and 0, which is 0.50. Mean recall at 5 is (0.50 + 1.0 + 0) divided by 3, which is 0.50.

Which retrieval metric to watch

Recall at k is usually the most important number for RAG, because the model can only use what it receives. A modern model can often ignore two irrelevant chunks, but it cannot invent a fact that was never retrieved. If recall at 5 is low, increasing k to 10 is the first experiment to try, because it is cheap and it shows whether the relevant chunk was close to the cutoff.

Precision at k matters for cost and for answer quality. Every irrelevant chunk adds tokens to the model's input, which increases cost and latency, and it gives the model more chances to quote the wrong passage. MRR and nDCG matter most when the system re-ranks results (sorts the retrieved chunks again with a more accurate model), because they show whether re-ranking moved the useful chunks to the top.

Report retrieval metrics per question type, not only as one average. In the support bot, a mean recall of 0.80 could hide a recall of 0.40 on questions about integrations, which would then explain most of the wrong answers in that area.

Want to build a full evaluation suite for an LLM product? Grokking LLM Evals teaches test sets, graders, judge validation, and eval statistics, with complete eval plans for a RAG bot and a research agent.

Context precision and context recall

The ranking metrics above need labeled chunk IDs. Context precision and context recall are related metrics that a language model computes instead, which makes them useful when chunk labels do not exist or when the chunking changes often. The names come from the RAGAS library, which popularized them, and other tools offer close equivalents.

Context recall checks whether the retrieved context contains everything needed to produce the reference answer. The grader model splits the reference answer into individual statements and checks whether each statement can be found in the retrieved chunks. If the reference answer for Question 1 has four statements and the retrieved chunks support three of them, context recall is 0.75. This metric needs a reference answer but no chunk labels.

Context precision checks whether the relevant chunks were ranked above the irrelevant ones. The grader model decides, for each retrieved chunk, whether it was useful for answering the question. The metric then rewards rankings that put useful chunks first, in a similar way to the ranking metrics above.

These two metrics trade exactness for convenience. They depend on the grader model's judgment, so they vary slightly from run to run and cost money for every evaluation. Teams with stable chunk labels usually rely on recall at k and nDCG for daily work and use the model-graded versions for questions where no labels exist yet.

Step 3: Measure faithfulness

Faithfulness, also called groundedness, measures whether every claim in the answer is supported by the retrieved context. It is the main metric for the generation layer, and it catches the failure that users trust least: an answer that sounds certain but states something the documents never said.

The standard method has two steps. First, a grader model splits the answer into short, independent claims (claim extraction). Second, the grader checks each claim against the retrieved chunks and marks it as supported or not supported. The faithfulness score is the number of supported claims divided by the total number of claims.

Consider an illustrative answer from the support bot: "You can change your card under Billing, then Payment methods. The new card is charged on the next invoice date. Changes take effect within 24 hours. You can also pay by bank transfer on annual plans." The grader extracts four claims and finds the first three in the retrieved chunks. The fourth claim about bank transfer appears nowhere in the context, so faithfulness is 3 out of 4, or 0.75.

Two properties of faithfulness are easy to misread:

• Faithfulness measures support, not truth. If the retrieved help article is out of date, an answer copied from it accurately still scores 1.0. Wrong documents are a knowledge base problem, and faithfulness will never detect them.
• A faithful answer can still be useless. An answer that repeats one supported sentence and ignores the rest of the question scores 1.0. This is why faithfulness is always paired with an end-to-end metric.

Faithfulness also has a practical threshold problem. A score of 0.90 averaged over 200 answers can mean that most answers are perfect and a few contain several unsupported claims. For a support product, the more useful number is often the share of answers with at least one unsupported claim, because one invented policy detail is enough to create a complaint.

Step 4: Measure answer relevance and correctness

The end-to-end layer asks whether the final answer serves the user. Two metrics cover it, and they need different inputs.

Answer relevance checks whether the answer addresses the question that was asked. A grader model compares the question and the answer and scores how directly the answer responds. Some tools compute it by asking a model to generate the likely question from the answer and then measuring how similar that generated question is to the real one. Answer relevance needs no reference answer, so it can run on live traffic.

Answer correctness compares the answer with the reference answer written by a person. The grader checks whether the key facts in the reference appear in the answer and whether the answer contradicts any of them. This is the closest metric to "was the user helped", but it only works on the test set, because live questions have no reference answers.

The unanswerable questions from Step 1 need their own score. For each of them, the correct behavior is to say that the help center does not cover the topic and, where possible, to suggest a human contact. Count the share of unanswerable questions where the bot declined correctly. A bot that answers every question confidently can have high answer relevance and still fail this check badly.

Setting up an LLM judge and checking it against people

Faithfulness, context recall, answer relevance, and answer correctness are all scored by a language model acting as a grader (an LLM judge). A judge is fast and cheap compared with human review, but it is a model, so it makes mistakes. Before trusting its scores, measure its error rate against human labels.

A judge works best with a narrow task and a clear rubric. Ask it one question per call, such as "Is this claim supported by this context? Answer supported or not supported, then give one sentence of reasoning." Avoid asking for a score from 1 to 10, because judges use such scales inconsistently. Binary or three-level labels are easier to check and easier to act on.

To check the judge, take a sample of about 100 to 200 test cases and have a person label them with the same rubric. Then compare the judge's labels with the human labels. Measure two rates separately: how often the judge marks a truly faithful answer as faithful, and how often it marks a truly unfaithful answer as unfaithful. A judge that agrees with people 90 percent of the time overall can still miss half of the unfaithful answers if those answers are rare.

As an illustrative example, suppose people label 160 support bot answers and find 40 with an unsupported claim. The judge flags 30 of those 40, so it catches 75 percent of the real failures. If the goal is to block releases that add unsupported claims, a judge that misses one failure in four needs a better rubric or a stronger model before it can gate releases. Repeat this check whenever you change the judge's model or rubric, because both change its error rates.

Tools that compute these metrics

Several open source libraries implement RAG metrics, so most teams do not write graders themselves. The metric names below come from each project's documentation. Libraries change their APIs between versions, so pin the version you use and read its documentation before upgrading.

• RAGAS offers faithfulness, response relevancy (formerly named answer relevancy), context precision, and context recall, along with other metrics. Its faithfulness metric uses the claim extraction method described above.
• TruLens organizes RAG evaluation around what it calls the RAG triad: context relevance, groundedness, and answer relevance. The three map onto the three layers, and none of them needs a reference answer.
• DeepEval provides contextual precision, contextual recall, and contextual relevancy for retrieval, plus faithfulness and answer relevancy for generation. Each metric takes a passing threshold, which makes it easy to run inside a test framework such as pytest.

All three use LLM judges for most of their metrics, so the judge check from the previous section applies to them as well. None of them replaces the ranking metrics computed from labeled chunks, which remain the cheapest and most stable retrieval signal.

Offline evaluation, online evaluation, and regression gates

Offline evaluation runs the test set against a version of the system before it is released. It is repeatable, so two versions can be compared on exactly the same questions. Online evaluation scores a sample of real traffic after release, using the metrics that need no reference answer, such as faithfulness and answer relevance, together with user signals such as thumbs down, follow-up questions, and escalations to a human.

A regression gate turns the offline evaluation into a release rule. Every change to the chunking, the embedding model, the retriever settings, the instructions, or the language model runs the full test set in continuous integration. The gate blocks the release if a protected metric drops beyond an agreed margin. An illustrative gate for the support bot could be:

• Recall at 5 on the real-question group must not fall by more than 2 points.
• The share of answers with an unsupported claim must stay under 5 percent.
• The correct decline rate on unanswerable questions must stay above 90 percent.

Compare versions on the same questions, case by case, rather than comparing two averages. Only the questions where the versions disagree tell you anything, and reading those questions directly often explains the change faster than any metric. Keep the test set fixed between comparisons, and add new cases from production failures on a regular schedule, so that the set keeps reflecting what users actually ask.

Diagnosing a failure to the right layer

When a test case fails, the layered metrics point to the cause. Work through the questions in order, because each answer rules out a whole class of repairs.

1. Was a relevant chunk in the top k? If not, the problem is retrieval. Look at chunking, the embedding model, hybrid keyword and vector search, the value of k, and re-ranking. Changing the instructions or the model will not help.
2. Is every claim in the answer supported by the retrieved chunks? If not, the problem is generation. Tighten the instructions to answer only from the context, require citations, reduce irrelevant chunks, or try a model that follows the context more closely.
3. Is the supported answer correct? If the answer is faithful but wrong, the source document is wrong or out of date. The repair belongs in the knowledge base, not in the pipeline.
4. Is the answer correct but rated poorly? Check the reference answer and the judge. Sometimes the reference is incomplete, or the judge's rubric penalizes a valid answer.

In the support bot example, a review of 30 failed cases might find 18 retrieval failures, 7 generation failures, and 5 outdated articles. That split, which is illustrative here, tells the team to spend its next week on retrieval, which a single end-to-end accuracy number could never have shown.

RAG evaluation metrics compared

Frequently asked questions

What is RAG evaluation? RAG evaluation measures a retrieval augmented generation system in separate layers: whether retrieval found the right passages, whether the answer stays faithful to those passages, and whether the final answer is relevant and correct. Separating the layers shows which part of the pipeline caused a failure.

What are the most important RAG evaluation metrics? Recall at k for retrieval, faithfulness for generation, and answer correctness or answer relevance for the end result. MRR and nDCG become important once the system re-ranks results.

What is the difference between faithfulness and answer correctness? Faithfulness checks whether the answer is supported by the retrieved context. Answer correctness checks whether the answer matches a reference answer written by a person. An answer copied accurately from an outdated document is faithful but incorrect.

How many test questions does a RAG evaluation need? Many teams start with 100 to 200 questions with labeled chunks and reference answers, then grow the set from production failures. Fewer than about 50 questions gives scores too noisy to compare two versions reliably.

Can I evaluate RAG without labeled data? Partly. Faithfulness, answer relevance, and TruLens-style context relevance need no reference answers, so they can run on any question, including live traffic. Recall at k and answer correctness still need labels, and they are the metrics that catch missing information.

Are synthetic test questions good enough? They are useful for coverage and quick to produce, but they tend to reuse the wording of the source text, which makes retrieval look better than it is. Combine them with real user questions and report the two groups separately.

Should I trust an LLM judge for RAG metrics? Only after checking it against human labels on the same rubric. Measure how many real failures it catches and how many correct answers it wrongly flags, and repeat the check whenever the judge model or rubric changes.

Related reading

• LLM Evaluation: How to Test an LLM App Before It Ships
• System Design for RAG: Vector Databases, Chunking, and Re-ranking
• How Large Language Models Work: Tokens, Embeddings, Attention
• What Is Generative AI? How Models Create Text, Images and Code
• Designing LLM Inference Systems: Batching, Memory, and GPUs

Ready to build evals that catch RAG failures before your users do? Grokking LLM Evals walks through test sets, graders, and judge validation step by step, ending with a complete eval plan for a RAG bot.