RAG Evaluation: How to Measure Retrieval Quality

RAG pipelines fail in two places: retrieval brings back the wrong context, or the model answers without grounding itself in what was retrieved. RAG evaluation measures both - so you know whether your retrieval actually works, not just whether the model talks fluently.

Two failure modes

Classic RAG debugging falls into either:

• Bad retrieval - the relevant documents never made it into the context.
• Bad generation - the context had what it needed, but the model hallucinated beyond it.

Good RAG evaluation targets each with its own metric.

The metrics that matter

• Context relevance - are the retrieved documents relevant to the question? This isolates retrieval quality.
• Faithfulness / groundedness - is every claim in the response supported by the provided context? This catches hallucination.
• Citation accuracy - do citations actually point to sources that support the claim?

Building a RAG evaluator

Most of these are reference-based LLM evals: they compare the response against the retrieved context using an AI judge with clear criteria and evaluation steps.

# A groundedness evaluator's judge criteria (simplified)

"Is every claim in the response supported by the

retrieved context? A faithful response only states

facts verifiable from the provided documents."



# Steps

# 1. Identify every factual claim

# 2. Find supporting evidence in the context

# 3. Mark SUPPORTED / UNSUPPORTED / HALLUCINATED

# 4. Score by the proportion of supported claims

From scores to action

RAG evaluation is only useful when it drives changes. Track scores over time and trace regressions back to the deployment:

1. Run evaluation on your eval dataset before every prompt release.
2. Monitor groundedness on live traffic with online evals.
3. When faithfulness drops, check the retrieval step - chunking, embedding, and reranking matter as much as the prompt.

If you can't measure retrieval quality, you can't improve it - RAG evaluation makes the invisible visible.

Try RAG evaluation in Infere

Infere ships groundedness, context relevance, faithfulness, and citation-accuracy evaluators out of the box, plus an AI observability platform to tie scores to every production request. Build an eval dataset, run it, and gate your RAG deployments on the results.

RAG pipelines add up in tokens - our true cost of agentic coding guide breaks down how retrieval context and agentic loops drive spend. For the broader evaluation taxonomy, see our LLM evaluation guide.