RAG Evaluation Metrics Cheat Sheet
RAG metrics broadly fall into four layers:
Can the Retriever find it?
↓
Is the Context complete and clean?
↓
Does the Generator answer correctly and stay grounded in evidence?
↓
Is the whole system safe, fast, and economical?
Traditional retrieval metrics are usually calculated directly from manually labeled relevant documents and do not require an LLM Judge. NVIDIA’s current Retriever evaluation includes Precision, Recall, nDCG, MAP, Reciprocal Rank, and Success@k. Ragas, AWS, and Databricks add metrics for complete RAG systems, including Context, Groundedness, Correctness, and Citation.
@kmeans that only the firstkretrieval results are checked. For example,Recall@10looks only at the first 10 results.
1. Retrieval and Ranking Metrics
| Metric | Why it matters | Rough calculation |
|---|---|---|
| Hit@k / Success@k | Checks whether at least one correct piece of evidence appears in the first k results; useful for single-evidence questions | Record 1 when Top-k contains at least one relevant result and 0 otherwise, then average across all questions |
| Recall@k | Checks whether required evidence was missed; it is one of the most important baseline metrics for a RAG Retriever | Number of relevant pieces of evidence found in Top-k ÷ total number of relevant pieces that should be found |
| Precision@k | Checks how many retrieved results are genuinely useful and reflects the amount of noise | Number of relevant results in Top-k ÷ k |
| MRR | Checks how early the first correct result appears; useful when finding one authoritative answer is enough | Take the reciprocal of the rank of the first relevant result; rank 2 is 1/2, then average across all questions |
| MAP@k | Considers both whether multiple relevant results are found and whether they are ranked highly | Compute Precision whenever a relevant result appears, then average those Precision values across all questions |
| nDCG@k | Distinguishes core evidence, supporting evidence, and partially relevant evidence rather than treating all relevance as binary; especially useful for evaluating a Reranker | Label results with different relevance grades; more important results ranked higher contribute more, then compare with the ideal ranking |
| All-Evidence@k | Checks whether the complete evidence chain required by a multi-hop question was found | Record 1 when every required evidence group enters Top-k; missing any group gives 0 |
| Document Recall@k | Checks whether the correct documents were found and helps locate whether a problem is at the document or Chunk level | Number of correct documents hit in Top-k ÷ number of correct documents that should be hit |
| Evidence/Chunk Recall@k | Checks whether the passages inside the correct documents that actually support the answer were found | Golden Evidence Chunks or evidence ranges hit in Top-k ÷ all required evidence |
Quick Intuition
Hit@k:
Was at least one result found?
Recall@k:
How much of the evidence that should be found was found?
Precision@k:
How much of what was retrieved is genuinely useful?
nDCG@k:
Was the most important evidence ranked first?
All-Evidence@k:
Was the complete evidence chain found?
All-Evidence@k is a useful custom engineering metric, but unlike Recall, nDCG, and MAP, it does not have one universally standardized name. It is best designed around labeled “evidence groups,” allowing multiple equivalent pieces of evidence to satisfy the same requirement.
2. Context Quality Metrics
Traditional Retriever metrics generally depend on manually labeled relevant documents. Context metrics more often use a reference answer or an LLM Judge to determine whether the Context finally passed to the generation model is clean, complete, and sufficient.
| Metric | Why it matters | Rough calculation |
|---|---|---|
| Context Precision | Checks whether the final Context contains too much noise and whether relevant Chunks are ranked first | Judge each Chunk’s usefulness; the more relevant Chunks there are and the earlier they appear, the higher the score |
| Context Recall | Checks whether all facts required by the reference answer can be found in the Context | Split the reference answer into factual claims and calculate how many are supported by the retrieved Context |
| Context Coverage | Similar to Context Recall, but emphasizes whether the Context covers the information required by the reference answer | Information from the reference answer covered by Context ÷ all information required by the reference answer |
| Context Relevance | Checks whether retrieved content is genuinely related to the user’s question | Usually ask an LLM Judge to score the relevance of each Chunk to the question, then average the scores |
| Context Sufficiency | Checks whether the current Context is already sufficient to produce a complete answer | Use expected_facts or a reference answer to judge whether the Context supports a complete response, usually with binary or graded scoring |
| Context Noise Ratio | Checks how many tokens are occupied by irrelevant content, which affects quality, latency, and cost | Number of irrelevant Context tokens ÷ total Context tokens |
| Duplicate Context Rate | Checks whether the same information is repeatedly passed to the model, avoiding duplicate bias and wasted tokens | Number of duplicate or highly similar Chunks ÷ total number of final Context Chunks |
Context Recall vs. Recall@k
Recall@k:
Calculated from manually labeled relevant documents or evidence IDs.
Context Recall:
Based on facts in the reference answer, judging whether those facts can be supported by the Context.
3. Answer Generation Quality Metrics
| Metric | Why it matters | Rough calculation |
|---|---|---|
| Answer Correctness | Checks whether the final answer is correct in its facts and conclusions; it is the most direct outcome metric | Compare the answer with a reference answer or expected_facts and calculate the proportion of correct facts, or have an LLM Judge score it against a Rubric |
| Faithfulness / Groundedness | Checks whether the answer stays faithful to the retrieved evidence and helps detect hallucinations | Split the answer into factual claims and calculate how many are supported by the Context |
| Completeness | Prevents an answer from being correct but covering only part of the question | Required facts in the reference answer that are covered by the actual answer |
| Answer Relevance / Response Relevancy | Checks whether the answer genuinely responds to the user’s question instead of being correct but off-topic | Judge the relevance of the answer to the user’s intent and penalize irrelevant or repetitive content |
| Helpfulness | Checks whether the answer can genuinely help the user complete a task | Usually scored by a human or LLM Judge based on usefulness, clarity, and actionability |
| Logical Coherence | Checks for contradictions, logical jumps, or inconsistencies inside the answer | Inspect the answer against a Rubric for complete and consistent reasoning |
| Exact Match | Suitable for fixed-form answers such as model names, dates, numbers, and enum values | Record 1 when the normalized system answer exactly matches the reference answer and 0 otherwise |
| Fact-level Accuracy | More suitable for enterprise RAG than whole-paragraph text similarity | Split the answer into multiple facts and calculate correct facts ÷ total evaluated facts |
Correctness and Faithfulness Are Different
Correctness:
Is the answer correct in reality or according to the reference answer?
Faithfulness:
Can the answer be supported by the retrieved Context?
For example:
The answer is correct, but the Context does not support it.
→ The model may have guessed correctly from its own memory.
The answer is faithful to the Context, but the answer is wrong.
→ The knowledge-base document may be outdated or incorrect.
4. Citation Quality Metrics
| Metric | Why it matters | Rough calculation |
|---|---|---|
| Citation Precision | Prevents citations from existing but failing to support the corresponding claim | Number of citations that correctly support the corresponding claim ÷ total citations |
| Citation Coverage / Citation Recall | Prevents a system from giving a few correct citations while leaving most facts uncited | Factual claims with valid citations ÷ all claims that require citations |
| Citation Validity | Checks whether citations genuinely exist and whether links and page numbers are valid | Number of valid citations ÷ total citations |
| Citation Entailment | Checks whether the cited source actually entails the conclusion in the answer | Classify each claim-citation pair as supported, unsupported, or contradicted |
For example:
The answer contains 10 facts.
Only 2 are cited, and both citations are correct.
Citation Precision = high
Citation Coverage = low
5. Refusal and Answerability Metrics
It is not enough to evaluate whether an answer is good; the system must also be evaluated on whether it knows when to refuse.
| Metric | Why it matters | Rough calculation |
|---|---|---|
| Refusal Accuracy | Checks whether the system refuses in the right situations | Correct answers or correct refusals ÷ all cases |
| Unsupported Answer Rate | Measures how often the system answers anyway when the documents provide no evidence | Cases where the system answers despite an unanswerable question ÷ total unanswerable questions |
| Over-refusal Rate | Measures how often the system refuses even though sufficient evidence exists | Cases incorrectly refused among answerable questions ÷ total answerable questions |
| Clarification Accuracy | Checks whether the system correctly asks for clarification when a question is ambiguous | Cases that should trigger clarification and actually do so ÷ all cases that should trigger clarification |
The ideal state is:
Evidence exists → Answer correctly
No evidence → Refuse
Ambiguity exists → Clarify first
No permission → Deny access
6. Security, Authorization, and Production Metrics
These are not necessarily standardized academic metrics, but they should usually be hard Release Gates for enterprise deployment.
| Metric | Why it matters | Rough calculation |
|---|---|---|
| Unauthorized Retrieval Rate | Checks whether the Retriever returned Chunks the user is not authorized to access | Unauthorized results ÷ total retrieved results |
| Cross-tenant Leakage Rate | Prevents tenant A from receiving tenant B’s data | Tests with cross-tenant exposure ÷ total authorization tests |
| Document Injection Success Rate | Checks whether malicious document instructions can control the model | Cases successfully manipulated by document injection ÷ total injection tests |
| p50 / p95 / p99 Latency | The average can hide a small number of very slow requests; production usually focuses more on p95 | Sort request latencies; p95 means 95% of requests do not exceed that latency |
| Cost per Query | Prevents a small quality improvement from causing a large increase in token and model costs | Total Retriever, Reranker, Embedding, and LLM cost ÷ request count |
| Token per Query | Checks whether Context is too long or contains duplication and waste | Average or percentile of input and output tokens used per request |
| Error Rate | Monitors timeouts, model errors, and parsing failures | Failed requests ÷ total requests |
Security metrics should normally require:
Unauthorized Retrieval Rate = 0
Cross-tenant Leakage Rate = 0
Permission leakage cannot be offset by improving other quality metrics.
7. The Recommended Minimal Metric Set
When first building a RAG system, there is no need to implement every metric at once. Start with this set:
| Evaluation layer | Priority metric |
|---|---|
| Did retrieval miss anything? | Recall@k |
| Is ranking reasonable? | nDCG@k |
| Was the complete evidence found? | All-Evidence@k |
| Is the Context clean? | Context Precision |
| Is the Context sufficient? | Context Recall or Context Sufficiency |
| Is the answer correct? | Answer Correctness |
| Is it grounded in evidence? | Faithfulness / Groundedness |
| Was information omitted? | Completeness |
| Are citations reliable? | Citation Precision + Citation Coverage |
| Does the system make things up without evidence? | Unsupported Answer Rate |
| Does it refuse too often? | Over-refusal Rate |
| Production performance | p95 Latency + Cost per Query |
| Authorization safety | Unauthorized Retrieval Rate |
This combination can be remembered as:
Recall:
Did we find everything?
nDCG:
Is the ranking good?
All-Evidence:
Is the evidence chain complete?
Context Precision:
How much noise is there?
Correctness:
Is the answer correct?
Faithfulness:
Is the answer based on evidence?
Completeness:
Did the answer omit anything?
Citation:
Are citations correct and complete?
Refusal:
Will the system make things up without evidence?
Latency / Cost / ACL:
Can the system safely enter production?
References
- NVIDIA NeMo Evaluator Retriever Metrics https://docs.nvidia.com/nemo/microservices/latest/evaluator/metrics/retriever.html
- Ragas Context Precision https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/
- AWS Bedrock Knowledge Base Evaluation Metrics https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-eval-llm-results.html
- Microsoft RAG Evaluation Guidance https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase