LLM Evaluation Metrics Dictionary: Definitions and Use Cases

An LLM evaluation metric scores a model output against a criterion you care about. This dictionary lists the metrics teams actually ship with, grouped by category. Each entry gives a short definition and the situations where that metric earns its place. Use it as a lookup reference, not as a reason to measure everything. The full LLM evaluation guide explains how these metrics fit into a test set and a deployment gate, and the AI glossary defines the underlying terms.

Two principles hold across every entry. First, evaluation measures relative performance across fuzzy quality dimensions, while testing gates deployment with pass or fail checks. Treat a metric score as a guide to iteration, and reserve hard gates for checks with a clear right answer. Second, keep your metric set small. Most sources that build evaluation pipelines in anger land on a handful of metrics, not a wall of them.

Read the dictionary in one pass and you will see the shape of it. Deterministic and lexical metrics compare output against a known reference and measure surface overlap. Semantic metrics compare meaning. Groundedness metrics check an answer against its context. Retrieval metrics judge how well a retriever pulls the right documents. Model-based metrics use a judge, often another LLM, to score qualities no reference can capture. Operational metrics watch cost and latency. Each family answers a different question, and mixing them is how you diagnose a system rather than just label it.

The metrics at a glance

The table below is the quick catalog. Group it by the question you are asking.

Exact match and deterministic metrics

Exact match is the most direct metric. The generated text must equal a reference string, character for character. A QA system that must return a normalized answer, a router that emits one of a fixed set of model names, or an extractor that returns a controlled vocabulary all have a correct answer you can compare directly. When a deterministic answer exists, evaluate directly instead of reaching for a model judge.

Levenshtein distance, also called edit distance, counts the minimum single-character edits that transform one string into another. The Levenshtein similarity ratio turns that distance into a 0 to 1 score. It suits spelling correction, near-duplicate detection, and any task where a character is the unit of correctness.

The broader family is code or functional evaluation. You define deterministic logic and check the output against it: the response is valid JSON, it matches a regex, it stays under a token limit, it contains no PII. These checks return the same result every run, which makes them the right mechanism for hard gates. As LangChain puts it, if you can check a criterion with a schema validator or a regex, a deterministic evaluator is faster, more consistent, and adds no model variability. Recognize that LLM judge calls are variable; a code check is not, so keep the must-pass criteria on the code side.

Feedback scoring shapes

When you do need a score and not just a pass or fail, match the shape to the criterion. Boolean feedback returns true or false and fits binary checks with no middle ground, like PII presence. Categorical feedback selects from predefined categories and captures nuance with less ambiguity than a continuous number. Continuous feedback gives a numerical score along a spectrum and supports relative comparison over time, useful for helpfulness or faithfulness. Boolean and categorical scores have less variance across runs; continuous scores have more granularity but need calibration to read meaningfully.

Classic lexical metrics and their limits

BLEU (Bilingual Evaluation Understudy) began as a machine translation metric. It computes the precision of matching n-grams between the model output and a reference translation, averages over a range of n-gram sizes, and applies a brevity penalty when the output is too short. Microsoft notes that a perfect score requires the candidate to be identical to a reference, which few human translations achieve, so you never need a perfect BLEU. It remains useful for translation and for tasks with a tight reference, but it rewards surface overlap over meaning.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures recall rather than precision. ROUGE-N counts matching n-grams between reference and output, and ROUGE-L uses the longest common subsequence, which captures word order without requiring contiguous match. Because it weights recall, it became the default for summarization, where the question is how much of the reference content the model managed to reproduce.

METEOR goes further than plain n-gram matching. It blends precision and recall into a single score, accounts for synonyms through a lexical database, and applies a penalty for word order differences. For translation and paraphrase tasks where the same idea arrives in different surface forms, METEOR tolerates wording variation far better than BLEU or raw ROUGE.

The limit of all three is that they measure tokens, not meaning. A correct answer phrased with different words can score near zero, and a fluent but wrong answer can score high if it shares vocabulary with the reference. Confident AI calls relying on BLEU and ROUGE for open-ended LLM outputs a common mistake because semantic nuance escapes them. LangChain puts it plainly: these traditional metrics remain useful for specific tasks, but they struggle with the subjective qualities that matter most for agents, like helpfulness, tone, and task completion. Keep lexical metrics for reference-driven, constrained generation; move to semantic or model-based metrics when the output is open-ended.

Semantic similarity metrics

Semantic similarity metrics compare meaning instead of surface words. BERTScore embeds both the reference and the output with a transformer model and measures the similarity of the contextual embeddings, so a correct paraphrase scores well even with no shared vocabulary. MoverScore moves further and uses the Earth Mover's Distance to align the two texts. Cosine similarity between embedding vectors is the underlying operation for several of these.

These metrics are comparatively fast and inexpensive to compute next to an LLM judge. Microsoft warns, though, that studies show they can correlate poorly with human judgment, can lack interpretability, carry inherent bias, and miss subtle language. Use them when a reference exists and you need tolerance of paraphrase without spending on a judge. When the quality you care about is conceptual rather than word-level, they beat BLEU; when you need reasoning about correctness, they do not.

Faithfulness and groundedness

Faithfulness measures whether each claim in the generated answer is supported by the context it was given. Microsoft defines it as factual consistency of the answer against the context: any claim that cannot be deduced from the context is penalized. The computation follows a two-step pattern: extract statements from the answer, then verify each statement against the context, producing a score between 0 and 1 where 1 is fully grounded. LangChain frames low faithfulness as direct evidence of hallucination, the model generating claims the sources do not support.

Confident AI's preferred implementation breaks the output into claims and checks each one against the retrieved nodes, then divides the number of supported claims by the total. The key design choice is to resolve each check to a limited set of answers such as yes, no, or unsure, because a constrained verdict avoids the variance of a free-form score. Faithfulness is the single most important metric for any RAG or grounded generation pipeline. An answer can read well and still fail it, and it is the first metric to move if you start seeing confident errors in production.

Context fields matter

Faithfulness is only meaningful when you pass the retrieval context as an input. If the evaluator has no reference for what the model was allowed to assume, it cannot tell a grounded answer from a confident guess. Every groundedness metric in this dictionary expects the retrieval context alongside the output. Leave it out and the score is meaningless.

Hallucination detection

Hallucination detection is the other side of faithfulness, looking for the fabricated claims themselves rather than scoring groundedness across the whole answer. The standard approach extracts claims from the output and asks whether the context or ground truth supports them, flagging any that do not. Working without a reference is the harder and more valuable version, because in production you rarely hold the perfect answer in your hand.

SelfCheckGPT is one reference-free technique. It samples the model multiple times for the same prompt and checks consistency across the runs. The assumption is that hallucinated outputs are not reproducible, while a model that truly knows a concept produces similar, consistent samples. Confident AI notes this makes detection a sampling-based process that is useful in production yet narrow: it suits hallucination, not summarization quality or coherence. Reference-less detection like this belongs in the online monitoring layer, where you have no ground truth but still need to catch confident errors before they reach users.

Answer relevance

Answer relevance asks whether the response actually addresses the user's question. Microsoft defines it as the degree to which a response directly and appropriately answers a given question, penalizing redundant information and incomplete answers, all without scoring factual correctness. This distinction matters, because an answer can be faithful to its context and still miss what the user asked.

Confident AI computes answer relevancy as the proportion of sentences in the output that are relevant to the input. The fuller version takes the retrieval context into account, because extra context can justify what looks like an off-topic sentence. LangChain makes the same point from the failure direction: a response can be perfectly faithful to the context yet fail relevance by never answering the actual question.

Answer relevance is where generation quality hides in plain sight. Chatbots and question-answering systems drift toward either verbose padding or a literal regurgitation of retrieved text, and this metric is what catches both. Pair it with faithfulness, and you separate the two ways a generator can fail: grounded but unhelpful, versus helpful-sounding but ungrounded.

Context relevance and retrieval quality

Context relevance judges the retriever rather than the generator. Microsoft defines it as how relevant the retrieved contexts are to the question, penalizing redundant information held in the context, and treats it as a measure of retrieval pipeline quality. The ideal context contains only what is necessary to answer.

Two more retrieval metrics sharpen this. Context precision asks whether relevant nodes rank above irrelevant ones in the context, which Confident AI flags as important because the generator weights earlier nodes more heavily. Context recall asks whether the retrieved context holds all the information needed to produce the expected output, catching a retriever that silently drops a necessary document.

Use these when you suspect the retriever and not the generator. The diagnostic frame from LangChain is exact: high context precision with low faithfulness means the retriever finds good documents but the generator hallucinates anyway; high faithfulness with low context recall means the model stays grounded but works with an incomplete picture. Context scores name the failing component before you spend time rewriting prompts that were never the problem.

RAG precision and recall

RAG precision and recall map to the two retrieval concerns above. Precision is the share of retrieved documents that are relevant, so high precision means less noise in the context window the generator must read. Recall is the share of the documents that should have been retrieved, so high recall means nothing important was left out. Because retrieval and generation fail independently, you evaluate them separately and read the combination.

The split is what makes the diagnosis possible. Measure retrieval on its own with context precision and recall, and measure generation on its own with faithfulness and answer relevance. Then combine the two views to locate the fault. If retrieval is strong but generated answers drift from context, the generator is the problem. If retrieval is weak, no prompt rewrite fixes an answer built on missing information. RAG-specific implementations of these metrics exist in open and standards-aligned evaluation libraries precisely because a generic correctness score cannot isolate the retriever.

G-Eval

G-Eval is a framework that uses an LLM to evaluate LLM output, from a paper titled NLG Evaluation using GPT-4 with Better Human Alignment. It builds the evaluation in two stages. First it generates a series of evaluation steps using chain-of-thought style reasoning, driven by a criteria definition you supply. Then it assembles a prompt from those steps plus the text to evaluate, and asks the model to return a score on a scale, typically 1 to 5. An optional step reads the token probabilities and weighs them into a fine-grained score, which the authors introduced to reduce bias in the coarse scale.

G-Eval's value is that it turns any criterion you can describe into a scored metric without a reference. Confident AI recommends it for subjective, task-specific criteria like brand voice or helpfulness, and calls it one of the most common ways to build LLM-as-a-judge metrics. Its cost is reliability: a judge asked for a number is by nature variable, so the score is a signal, not a law. Where the success criteria are clear and decision-based rather than subjective, a deterministic decision tree fits better than a G-Eval rubric.

LLM-as-a-judge scores

LLM-as-a-judge is the broad method where a model scores an output against a rubric prompt. It evaluates criteria that need reasoning and that no reference or deterministic rule can capture, such as helpfulness, professional tone, or domain accuracy. The judge can score a single output, answer reference-free against the text alone, compare an output to ground truth, or run a pairwise comparison between two candidate responses.

The method is powerful and imperfect. LangChain catalogs the known weaknesses: position bias, where the judge favors the response that appears first or last in its context; verbosity bias, where a longer response outscores a concise correct one; and inconsistency, where the same judge and prompt can score the same output differently across runs.

Apply the documented mitigations together. Use boolean or categorical scoring instead of open-ended continuous scores to shrink the surface where bias enters. For pairwise comparisons, run the comparison twice with the positions swapped and aggregate, which cancels position bias. Calibrate the judge against a small set of human-labeled examples and tune the prompt until alignment improves. Run the same evaluation multiple times and average. Above all, treat judge scores as informative signals rather than absolute truth, and validate the judge against human judgment for your own criteria before you trust it at scale. The dedicated explainer, what is LLM-as-a-judge, covers single versus pairwise judging and the bias calibration in more depth.

Cost and latency guardrails

Cost and latency are the operational metrics that keep evaluation and inference sustainable. Latency measures the time from request to response, and it matters most on interactive surfaces where users wait on a clock. Cost per request or per token measures the spend of a call, and it compounds when an agent loops or when a judge calls a model for every evaluation run.

These metrics act as guardrails rather than quality scores. Two concrete ones belong in nearly every setup. First, token limits: a code evaluator that fails any output over a length cap is a deterministic guard that also bounds cost, because you pay for the tokens you generate. Second, evaluation spend itself: every LLM judge call costs tokens, so sampling a subset of traffic instead of scoring every response, and running less expensive checks before the expensive judges, keeps the evaluation layer within budget.

Set thresholds the way you set any metric: define a ceiling, watch how the system behaves against it, and turn the worst violations into failing gates. The principle that holds for quality metrics also holds here, a metric guides, a test gates. A latency budget that the system routinely misses is a signal to route differently; a spend ceiling that triggers on a sample is a signal to trim evaluation volume.

How to choose your metrics

Choose a small set and let the architecture decide it. Confident AI offers a practical rule: keep the pipeline to roughly five metrics, split as a couple of task-specific ones and a couple of generic ones tied to how the system is built. For a RAG assistant, that means faithfulness and answer relevance for the generator, context precision and recall for the retriever, plus one custom metric for something like tone or helpfulness. Every extra metric adds noise and maintenance, so the default should be fewer.

Match the shape to the question. Reference-free, groundedness, and model-based metrics fit online and production monitoring where no ground truth exists. Reference-based metrics need curated datasets and fit offline regression tests. And remember the split between evaluation and testing: subjective measures guide iteration, while deterministic checks gate deployment. A faithfulness score of 0.7 can be excellent for one use case and inadequate for another, so do not bolt it onto a hard deployment gate.

This dictionary gives you the catalog. To see how the pieces assemble into a test set and a pre-deployment gate, work through the LLM evaluation guide. It walks the metric categories, the human versus model versus heuristic tradeoff, and the regression gate as a concept, all in vendor-neutral terms.