Technical GUIDE
Retrieval Metrics: Recall@K, MRR and nDCG
Recall@K measures how many known relevant items appear in the top K results, MRR emphasizes the rank of the first relevant item, and nDCG evaluates ranking quality using graded relevance with position discounts.
On this page3 min read
Overview
Each metric reflects different judgments and use cases, and none directly measures whether a generated answer is correct.
Deep Dive
Retrieval evaluation starts with queries, a document collection, and relevance judgments. A metric summarizes how a ranked result list compares with those judgments. Recall@K asks what fraction of relevant items appear within the first K results. It is useful when finding all relevant evidence matters, but it ignores the ordering among those retrieved items.
Mean reciprocal rank (MRR) scores the rank of the first relevant result: a relevant item at rank one contributes 1, at rank two contributes one-half, and so on; query scores are averaged. This fits tasks where users mainly need one good result near the top. It does not reward additional relevant results after the first.
Normalized discounted cumulative gain (nDCG) uses relevance grades and discounts items lower in the ranking. It can distinguish highly relevant from partially relevant documents and rewards placing stronger matches earlier. The exact gain and discount conventions should be documented when comparing implementations. These formulas summarize judged rankings and do not measure answer generation or user satisfaction.
These measures depend on a test collection and relevance labels. Incomplete judgments can make a relevant but unjudged result look wrong, and query selection affects conclusions. For retrieval-augmented generation, report retrieval metrics separately from answer-level grounding, correctness, and usefulness. Choose metrics that match the product goal, inspect examples, and evaluate on representative queries rather than selecting whichever score looks largest.
Strategic Impact
Cost and budget
Architecture decisions drive performance and operating cost for years.
Clearer decisions
Technical education helps teams choose the right stack, not just the newest one.
Quality control
Better engineering choices reduce reliability incidents in production.
The Future of Retrieval Metrics: Recall@K, MRR and nDCG
Retrieval systems will increasingly combine lexical, vector, hybrid, and reranking stages, making stage-specific evaluation important. Better relevance judgments and task-aligned graded labels can make nDCG more useful, while user studies can test whether rank gains improve outcomes. RAG evaluation will continue to pair retrieval metrics with answer-level checks. Future benchmarks should publish query sets, judgments, metric settings, and limitations to support meaningful comparisons. Teams should revisit judgments as collections and user needs change over time for all relevant users consistently.
Real-World Implementation
A legal search team uses Recall@20 when it needs to surface most known relevant passages for review.
A help center tracks MRR when users need one useful answer near the top.
A recommender uses graded relevance labels and nDCG to favor stronger matches earlier.
A RAG evaluation reports retrieval recall separately from whether the generated answer is supported.
Risks & Guardrails
Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.
Implementation Roadmap
Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Retrieval Metrics: Recall@K, MRR and nDCG quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Retrieval Metrics: Recall@K, MRR and nDCG?
Recall@K measures how many known relevant items appear in the top K results, MRR emphasizes the rank of the first relevant item, and nDCG evaluates ranking quality using graded relevance with position discounts. Each metric reflects different judgments and use cases, and none directly measures whether a generated answer is correct.
What does Recall@K measure?
Recall at a cutoff measures relevant-item coverage in the top K.
What does Recall@K fail to distinguish by itself?
Recall@K measures coverage at the cutoff, not ordering within those results.
Why can incomplete relevance judgments distort retrieval metrics?
Metrics depend on the judged collection and can penalize unjudged positives.
Does a high retrieval Recall@K prove that a RAG answer is correct?
Retrieval and answer generation are distinct evaluation stages.
Keep learning
Related guides
More guides picked for this topic