LLM Evaluation: How to Test an LLM App Before It Ships

LLM evaluation is the practice of running a fixed set of test cases through an application that uses a language model, grading each output, and making a release decision from the results. Every evaluation (an eval) has four parts: the inputs, the run of your system on those inputs, a grader that marks each output, and a decision rule that turns the grades into ship or do not ship.

An eval differs from a unit test because an LLM feature can produce many correct outputs, and the same input can produce different outputs on different runs. It also differs from a public benchmark, because a benchmark measures a model on general tasks, while an eval measures your product on the failures your users actually see.

This article walks through the order that works in practice. Find the failures first, build a test set from them, and choose a grader for each failure. Then check that the grader itself is correct, and read the results with enough statistics to trust a release decision.

Why do LLM apps need a different kind of testing?

Three properties of language models break ordinary testing habits. The first is that a correct answer has many forms. Two summaries of the same support ticket can both be correct while sharing few words, so an exact string comparison fails good outputs.

The second is nondeterminism. Setting the temperature to 0 makes outputs more consistent, but the major model providers state in their documentation that it does not make them deterministic. The same request can still return a different answer on a later run.

The third is side effects that nobody predicted. A prompt edit that fixes one failure can change behavior on inputs nobody looked at, and a provider can update the model behind a name you did not change. Only a test set that runs on every change can detect these regressions.

This article uses one running example: a job ticket assistant for a home repair app. Customers describe a problem in free text, and the assistant turns the request into a ticket with a trade (plumbing, electrical, appliance), an urgency level (emergency, same day, scheduled), a summary for the technician, and a reply to the customer.

Step 1: Start with error analysis, not metrics

The most common mistake in LLM evaluation is to start from generic metrics such as "helpfulness" or "coherence". These scores rarely match the failures that hurt a specific product. The failure list has to come from reading your own outputs (error analysis).

Error analysis has four steps:

1. Sample real traces. A trace is the full record of one request: the input, any retrieved context, the model output, and tool calls. Read a sample of about 100 traces, chosen across different kinds of input.
2. Write one note per failing trace. Describe the first thing that went wrong, in specific words. "Gas smell booked as scheduled" is a useful note, while "bad output" is not.
3. Group the notes into failure modes. Similar notes become one named mode with a clear pass and fail definition, such as "urgent safety request not marked emergency". Stop reading when new traces add no new modes.
4. Count and rank the modes. Rank them by how often they occur and how much damage each one causes. A rare mode that sends a technician to a gas leak two days late outranks a common formatting slip.

Each failure mode then gets one of three decisions: fix it directly, write an eval for it, or watch it. A typo in the instructions needs a fix, not an eval. A mode that can come back after any prompt or model change needs an eval.

Step 2: Build an eval dataset

A test case contains an input, any context the feature needs, the expected behavior, tags that describe the case, and a note of where it came from. Expected behavior can be a reference answer, or a set of properties that any correct answer must have, such as "urgency is emergency" and "the reply does not promise a time".

Good eval datasets combine three sources:

• Sampled real traffic, stratified by tag so that rare but important inputs are present. Personal data must be removed first.
• Synthetic cases built from dimensions. You choose the dimensions that matter (trade, urgency, message length, language quality), list combinations, and have a model write a natural request for each one. A person reviews every generated case.
• Hard cases on purpose: ambiguous requests, requests with two problems in one message, and inputs that try to inject instructions.

Split the set into a development set, which you look at while changing the prompt, and a held out set, which you run only to confirm a change. If you tune the prompt until the whole test set passes, the score stops predicting production quality. Every failure found in production should become a new test case, which is how the set grows into a regression suite.

Step 3: Choose the cheapest grader that catches each failure

A grader decides whether one output passes. There are three kinds, and the rule is to use the cheapest one that reliably measures the failure mode.

Code checks are fast, cheap, and exact, so they should cover every failure they can. In the ticket example, a code check verifies that the urgency field holds one of the three allowed values, and a reference comparison verifies that a gas smell request is marked emergency.

Word overlap metrics such as BLEU and ROUGE are a poor fit for open text, because two correct answers can share few words. For open text, a key fact checklist often works better: the reference is turned into required facts and forbidden claims, and the grader checks each item.

Step 4: Use LLM as a judge, then check the judge

An LLM judge is a model that grades another model's output against written criteria. It is the right grader for qualities that code cannot check, such as whether a ticket summary invents a detail that the customer never mentioned.

Judges work best under four rules. Each judge checks one failure mode, not general quality. It returns a binary pass or fail rather than a 1 to 5 score, because pass and fail are easier for both people and models to apply consistently. It writes its reasoning before the verdict. Its instructions include examples of passing and failing outputs.

A judge is also a model, so it has biases that must be measured. In the MT-Bench study, a GPT-4 judge comparing two answers gave the same verdict after the answers were swapped only 65 percent of the time (position bias). Judges can also prefer longer answers. Pairwise judges should therefore run both orders and count a disagreement as a tie.

The judge must then be validated against human labels. People label a set of outputs as pass or fail, and you measure two rates on it: how often the judge passes outputs that humans passed (true positive rate). The second is how often it fails outputs that humans failed (true negative rate). Raw agreement alone misleads when most outputs pass.

These two rates also let you correct the judge's pass rate. Suppose a judge reports an 80 percent pass rate, with a true positive rate of 0.92 and a true negative rate of 0.88. The corrected rate is (0.80 + 0.88 - 1) divided by (0.92 + 0.88 - 1), which is 0.68 divided by 0.80, or 85 percent.

Want to build an eval suite like this from start to finish? Grokking LLM Evals teaches error analysis, test sets, graders, judge validation, and eval statistics, with Python code that runs in the browser.

How do you evaluate a RAG system?

A retrieval augmented generation (RAG) system first searches documents for relevant passages, then asks the model to answer from them. It needs two separate measurements, because an answer can fail at either stage.

Retrieval metrics check whether the right passages came back. Recall at k is the share of relevant passages found in the top k results. Precision at k is the share of the top k results that are relevant. These metrics need relevance labels: for each test question, a person marks which passages actually answer it.

Generation metrics check what the model did with the passages. Faithfulness splits the answer into individual claims and checks whether the retrieved context supports each one. Faithfulness measures support, not truth: an answer copied faithfully from an outdated manual still passes. Answer relevance and completeness check whether the answer addresses the question and covers the needed facts, and a separate set of unanswerable questions checks that the system says so instead of guessing.

When a RAG answer is wrong, check retrieval first. If the right passage never reached the model, no prompt change will fix the answer. For the design side of these systems, see System Design for RAG. For the full set of retrieval, faithfulness, and answer quality metrics with a worked example, see How to Evaluate a RAG System: Retrieval Metrics, Faithfulness, Answer Quality.

How do you evaluate an AI agent?

An agent takes several steps, calls tools, and changes things in the outside world, so grading its final message is not enough. Agent evals run in a controlled environment, such as a fake calendar and a test job database, and use three kinds of checks.

• Final state checks grade what the agent left behind: is the booking in the database, at the right time, for the right technician?
• Trajectory checks grade the path: did the agent check availability before booking, did it avoid forbidden actions, and did it stay within the step budget?
• Repeated runs measure reliability, because one successful run proves little.

Repeated runs need two different numbers. Suppose a task succeeds 75 percent of the time on each attempt. The chance that at least one of three attempts succeeds (pass@3) is 98.4 percent. The chance that all three succeed (pass^3) is only 42.2 percent. A product that customers use repeatedly should report the second number. The patterns that make agents easier to evaluate, such as tracing and outcome verification, are covered in Agentic Design Patterns.

How many test cases do you need?

A pass rate from an eval is an estimate, not an exact value, because the test cases are a sample of everything users might send. The width of the uncertainty depends on the number of cases.

With 50 cases, a version that scores 80 percent could plausibly be anywhere from about 69 to 91 percent, so a 3 point improvement means almost nothing. When the observed rate is close to 0 or 100 percent, use the Wilson interval instead of the simple formula above. For 45 passes out of 50, the Wilson 95 percent interval runs from 78.6 to 95.7 percent.

To compare two versions, run both on the same cases and compare them case by case (a paired comparison). Only the cases where the two versions disagree carry information, which is why a paired test needs far fewer cases than comparing two separate scores. For noisy features, run each case several times and average the results, since lowering the temperature does not remove run to run variation.

Before you ship: gates, regressions, and production checks

The final step turns evals into a release process. A release gate runs the suite in continuous integration on every prompt, model, or retrieval change, and blocks the release when a critical failure mode gets worse. Gates should compare intervals or paired results rather than raw thresholds, so that noise does not block a good change.

Two kinds of eval run at this stage. Capability evals measure whether the feature can do something new, and a low score is expected at first. Regression evals protect behavior that already works, and they should pass close to 100 percent of the time.

Offline evals do not replace monitoring. After release, a sample of production traffic is graded by the same judges, user signals such as edits and retries are tracked, and the full suite runs again whenever the provider updates a model. Even a pinned model version can shift slightly through serving changes, so regression evals remain necessary after pinning.

Frequently asked questions

What is LLM evaluation? LLM evaluation is testing an application that uses a language model by running fixed test cases through it, grading each output, and deciding from the results. An eval has four parts: inputs, the system run, a grader, and a decision rule.

What metrics should I use to evaluate an LLM app? Use metrics that come from your own failure modes, found by reading real outputs. Then measure each mode with the cheapest reliable grader: code checks for format and rules, reference comparison for known answers, and an LLM judge for open text. Generic scores such as "helpfulness" rarely predict product quality.

What is LLM as a judge? It is the use of a language model to grade another model's output against written criteria. A good judge checks one failure mode, returns pass or fail, and is validated against human labels before its scores are trusted.

How do you evaluate RAG? Measure retrieval and generation separately. Retrieval metrics such as recall at k check whether the right passages were found. Generation metrics such as faithfulness check whether each claim in the answer is supported by those passages.

How big should an eval dataset be? It depends on the size of the change you need to detect. At an 80 percent pass rate, 100 cases give an interval of about plus or minus 8 points, and 400 cases give about plus or minus 4 points. Comparing two versions on the same cases reduces the number you need.

What is the difference between capability and regression evals? Capability evals measure whether a feature can do something new, so low scores are expected early. Regression evals protect behavior that already works, so they should pass almost every time and block a release when they do not.

Related reading

• Agentic Design Patterns: The Patterns Behind Production AI Agents
• System Design for RAG: Vector Databases, Chunking, and Re-ranking
• Design an AI Agent: The 7-Layer Blueprint
• Designing LLM Inference Systems: Batching, Memory, and GPUs
• The ReAct Pattern: How an Agent Reasons and Acts in a Loop
• How to Evaluate a RAG System: Retrieval Metrics, Faithfulness, Answer Quality
• How Large Language Models Work: Tokens, Embeddings, Attention

Ready to test your LLM features before users find the failures? Grokking LLM Evals walks through every step in this article, ending with complete eval plans for a RAG bot and a research agent.