LLM evaluation is the process of measuring whether a language model, or the application built on top of it, does what your specific use case needs it to do, both before deployment and continuously in production. It answers one question: does this model produce correct, grounded, relevant, safe, and cost-effective output on the inputs your users actually send? It is not a single test you run once. It is a system of metrics, methods, and test sets that operate at different stages, from a unit test on every commit to a quality gate in your CI/CD pipeline.
A working evaluation setup combines three things. First, a clear picture of what to measure, which spans four dimensions: correctness, faithfulness, relevance, and safety, plus the operational constraints of cost and latency. Second, metric categories that score each dimension, from exact match for structured output to an LLM judge for open-ended quality. Third, a way to run those metrics repeatedly, on a versioned test set, and to block regressions before they reach users. This guide walks through all three in order.
What LLM evaluation actually measures
Evaluation is not the same as running benchmarks. Public benchmarks like MMLU, HumanEval, and HellaSwag appear in every model release announcement, yet they predict almost nothing about whether a model will work well in your specific application. A model that ranks second on MMLU is not necessarily better at summarizing enterprise legal documents than a model that ranks sixth. The benchmark measures broad general reasoning, while your task is specific and concrete.
What evaluation measures instead is better captured as three layers:
- Functional quality. Does the model produce correct output for its task? For a customer support bot that means accurate answers. For a code generator that means code that compiles and passes tests. For a summarizer it means summaries that preserve key facts without adding false ones.
- Safety and behavior. Does the model stay inside the boundaries you defined? Does it refuse harmful content, stay on topic, and handle edge cases and adversarial inputs without breaking?
- Production stability. Do functional quality and safety hold over time, across model updates, prompt changes, and shifts in real user input? Most teams discover problems in this layer during an incident rather than before one.
Output quality is not one thing. Teams that track a single quality number miss the specific failures that cause production incidents. Correctness, faithfulness, relevance, and safety are distinct and each needs its own metric or method. Confusing them leads to the wrong fix. A model that produces fluent, plausible text that contradicts its retrieved source documents has a faithfulness problem, not a correctness problem. Correcting the answer against training data does nothing for it.
Two more dimensions matter at the system level even though they are not quality scores. Cost measures the price of each request or task, which depends on token count and chosen model. Latency measures the time from request to response, which depends on model size, provider routing, and network. Neither replaces a quality score, but both constrain which model and setup you can ship. A brilliant model you cannot afford at volume, or one too slow to serve an interactive feature, still fails evaluation.
Evaluation methods: human, model, and heuristic
Three families of methods take the measurements. Each has a different cost, speed, and trust level, and the right system uses all three in layers.
Heuristic and reference-based metrics
Heuristic metrics evaluate output with rules and comparisons you write in code. The most direct is exact match, which checks whether the output equals an expected string. This works for classification, structured extraction, and slot filling, tasks where valid output variation is minimal. Reference-based metrics compare output to a ground truth answer. BLEU and ROUGE measure n-gram overlap between the output and a reference, useful for constrained tasks but blind to meaning. A summarizer that produces a genuinely better summary than the reference gets penalized by ROUGE because the words differ. The metric is doing what it was designed to do, but it is the wrong tool for open-ended generation.
These methods are fast and inexpensive to run, and fully deterministic. They do not capture semantic nuance, tone, or creativity. Think of them as the first filter on every output, catching structural breakage before anything more expensive runs.
Model-based evaluation and LLM-as-a-judge
Model-based evaluation uses another model to score your output against a rubric. This is the LLM-as-a-judge pattern, and it is the dominant method for open-ended tasks where no single right answer exists. You prompt a capable model to rate helpfulness, factuality, or coherence, in absolute scores or in pairwise comparison between two candidate outputs. It scales without human review and can run on thousands of outputs. Read the detailed explainer on what is LLM-as-a-judge to see how judges are prompted, how single-versus-pairwise judging changes reliability, and how to mitigate bias.
LLM judges carry three known biases you must control. Position bias favors whichever answer appears first, one study finding that swapping the order of two identical responses shifted a judge preference by more than ten percent. Verbosity bias scores longer answers higher regardless of content. Self-preference advantages outputs from the same model family as the judge. Randomizing order, capping answer length, and using a judge from a different family than the model you are testing all reduce these effects.
Human evaluation
Human evaluation is the only method that establishes ground truth. Automated metrics and LLM judges measure quality relative to a proxy. Humans determine whether the output is actually good. This is the slowest and most expensive option, so you rarely run it continuously. Its real uses are to label the examples that calibrate your judge, and to spot check held-out cases that confirm the calibration still holds over time.
Do not treat these as competitors. The layered approach used in practice runs heuristic checks first, applies an LLM judge for nuance, and brings humans in for calibration and the highest-risk cases. A production monitoring setup might score ninety percent of traffic with an automated judge and route a small flagged sample to humans for review.
Metric categories
Once you know which dimension matters, pick the metric category that scores it. The full reference lives in the LLM evaluation metrics dictionary, which defines each metric and when to use it. The main categories are:
Correctness metrics judge whether the output is factually accurate relative to a known answer. Use exact match for structured outputs, ROUGE for extractive tasks, or an LLM judge comparing the output to a reference answer. Correctness matters most where wrong answers have real consequences: medical queries, financial data, code generation.
Faithfulness and hallucination metrics check whether the output stays grounded in the provided context. A model can produce fluent, plausible text that contradicts the source documents. Groundedness and contradiction metrics detect output that invents facts or ignores the retrieved material. These are essential for any RAG or summarization system, because hallucination is a distinct failure from being simply wrong.
Relevance metrics assess whether the response addresses the question the user actually asked. Relevance differs from correctness, because you can give a correct answer to the wrong question, and from faithfulness, because a response can be grounded in context yet still tangential. This is the dimension users report when they say the assistant keeps answering questions they never asked.
RAG metrics split the retrieval and generation components so you can localize a failure. Context relevance scores whether the retriever pulled useful information. Answer relevance scores whether the model used that context to answer. Faithfulness and groundedness check that the generation does not contradict the retrieved passages. Some setups also measure context recall, how much of the relevant information in the source corpus the retriever found. About an LLM judge, RAG metrics typically run as judge-based scoring on a few hundred examples drawn from your real retrieval pipeline.
Core metric categories follow your application type. Open-ended assistants lean on relevance, faithfulness, and helpfulness via an LLM judge. Extraction and classification pipelines lean on exact match, F1, and JSON validity run programmatically. Agents add task completion and tool-call accuracy on top of output quality.
Building a test set
Every evaluation method needs data to run against. The dataset is usually the bottleneck, not the method. For offline evaluation and regression gating you need a golden test set, inputs with known-good or human-rated outputs that represent the actual tasks your application handles.
Dataset size follows the size of the quality shift you want to detect. Roughly fifty examples detect large regressions. Two hundred examples give statistical confidence on smaller differences, on the order of a three to five percent quality change. Beyond about five hundred examples you hit diminishing returns unless your application has highly varied subtasks that each need separate coverage. Sizes in the two hundred to five hundred range suit most production systems.
Where the dataset comes from matters more than its size. Sets built only from documentation or synthetic generation are too clean. They miss the edge cases, ambiguous queries, and adversarial inputs that surface once real users interact with the system. The most effective golden sets combine three sources: human-crafted examples covering known edge cases, real production samples with personal data removed, and synthetic expansions for scenarios you rarely see in production.
Methodological direction matters too. Top-down design, picking a metric and building a dataset to measure it, produces high scores against that metric and surprising failures in production. Bottom-up analysis starts with the failures you have already seen and designs the metric to catch them. Evals built bottom-up predict what actually breaks. Start with failures you have already observed, not failures you have imagined.
Treat the test set as critical infrastructure. Version it like code, require review on changes, and treat deletions or edits to ground truth as production risk. Adding easier examples to make scores look better is a real and common failure mode. Hold out a hidden subset that never participates in development so nobody can silently optimize against the labeled set.
Calibrating your judge
If you use an LLM judge, calibration is not optional. Pre-built eval libraries ship generic rubrics designed to work across every task, and they produce confidently wrong scores on specialized applications. The first version of any rubric is wrong in ways you cannot predict until you watch it fail on real examples.
The setup that works starts with one domain expert who owns the product. Have that person grade thirty to fifty examples with binary pass or fail verdicts and a written critique for every failure. Do not use a committee. Different annotators weight dimensions differently, and the resulting labels reflect who graded each example rather than whether the output was actually good. Skip five-point Likert scales too. They produce low-variance data, where scores cluster in a narrow band, and they force the judge to make distinctions it cannot reliably make. Binary pass or fail with a written critique generates better calibration data and forces the judge to commit.
Iterate the judge prompt against those labeled examples, then measure agreement per class rather than overall agreement rate. An overall rate misleads on imbalanced data, because a judge that stamps everything pass achieves high agreement on a set where only a fraction of outputs should fail. Aim for a judge whose scores match the expert verdicts at a strong correlation, commonly around or above a 0.7 threshold. Below that, you are measuring something, but you cannot say it correlates with whether users experience the system as working. Recalibrate periodically as your application and its inputs evolve.
Regression gating in CI/CD as a general pattern
The point of all this measurement is to stop quality regressions before they reach users. In conventional software, a build that fails its tests does not merge. Eval-as-gate applies the same idea to LLM applications: an evaluation run becomes a gate that a change must pass before it ships. This is a general software practice, the same discipline you already apply to unit tests, and no specific product is required to implement it. You can build it with your existing CI/CD runner, a versioned test set, and a scoring harness.
Run evaluation wherever you would run software tests, in layers that match cost and cadence:
- Development and every commit. Run milliseconds-fast heuristic checks. Does the output contain a phone number when it should not? Does the classifier return one of the expected labels? Does the JSON parse? These sanity checks catch obvious breakage before anything reaches a judge, and they keep your slower evals from being buried in trivial failures.
- Pre-release. Before a new model version or a major prompt change, run against an adversarial set covering edge cases and known failure modes such as prompt injection and insecure output handling. The golden set covers failures you have already seen. Adversarial testing finds the ones you have not.
- CI regression on every pull request. Every PR that touches a prompt, a model version, or a retrieval configuration triggers an eval run against the golden set. A PR that regresses quality past the acceptable threshold does not merge. This is the eval-as-gate step, and for most teams it is where the first real pipeline starts.
- Production monitoring. Sample a slice of real traffic, score it with automated evaluators, and watch for drift over time. This is the only layer that catches changes that happen to you rather than changes you make, such as a provider updating model weights without a version change, or new user segments shifting the input distribution.
Design the gate around acceptable fail rates per failure category rather than a single binary threshold. A hard score above which everything ships and below which nothing ships creates false alarms and gets disabled. A more durable design sets tolerances per failure mode. A low hallucination rate on general queries might be tolerated, while safety failures at any rate block the deploy. Different failure modes warrant different tolerances.
Regression tracking completes the picture. Store every eval run with a timestamp so you can answer when a metric started failing. A dashboard showing only the current score cannot tell you anything about causation. With versioned datasets, a stable harness that pins the judge model and rubric, and stored results, you can attribute each regression to the change that caused it.
Keep the harness consistent between runs. Pin the judge model, the rubric, and the sampling and temperature settings. A harness that changes between runs produces scores you cannot compare. Keep level one sanity checks in place, because without them the CI gate carries too much load and runs too slowly to be the fast feedback loop your team needs.
The mistakes that make evaluations useless
Several failure patterns recur across teams that adopt evaluation:
- Optimizing the metric instead of the task. When evaluation scores become a team metric, engineers optimize the eval set rather than improving output quality. A hidden test set that does not participate in development is the standard protection.
- Running evaluations only after something breaks. Retrospective evaluation confirms something went wrong. Running evals on every change tells you what changed and when.
- Trusting judge scores without calibration. A judge that scores everything in a narrow band is averaging, not evaluating. Calibrate it against expert verdicts.
- Using generic judges on specialized tasks. Off-the-shelf rubrics produce confidently wrong scores on your application. Calibration is not optional.
- Conflating model quality with application quality. A better base model is not always a better application. A model upgrade can improve average quality while introducing edge-case behavior that breaks safety filters or changes tone against your requirements. Evaluate the system, not just the model component.
Treat evaluation as critical infrastructure and maintain it like anything else you rely on in production. Version your datasets, prompts, and rubrics. Document the decision criteria. Control for randomness. Make the runs repeatable so a score difference means a real change and not a change in how you measured.
Where to go next
This guide is the hub for the LLM evaluation coverage. Its two closest companions go deeper on the pieces introduced here. The LLM-as-a-judge explainer covers how judges are prompted, how single and pairwise judging differ, and how to handle calibration and bias. The LLM evaluation metrics dictionary catalogs each metric, from exact match and BLEU through faithfulness, groundedness, relevance, and judge-based scoring, with guidance on when each belongs in your pipeline. For RAG-specific metrics, see RAG evaluation. For conversational quality, read our guide to multi-turn LLM evaluation. When human review is part of your pipeline, see human-in-the-loop evaluation. For agent-specific evaluation patterns, read why passing evals isn't enough. To automate quality gates, read our guide to building an LLM eval framework in CI/CD. For comparing prompts and models systematically, see our LLM experimentation guide.
Whatever tooling you use, the shape of a sound evaluation system stays the same. Pick the dimensions that match your failure modes. Choose the metric categories that score them. Build a versioned test set from the failures you have actually seen. Calibrate any judge against expert verdicts. And gate every change on the metric that corresponds to your most expensive failure, so a regression cannot reach users in the first place. That loop, more than any specific framework, is what makes an LLM application safe to trust in production.