Here is the short version.
Recall@10 tells you whether the good documents appeared at all. MRR tells you how quickly the first good document showed up. NDCG@10 tells you how well the entire top of the list was ordered. None of them is “the” metric. Each answers a different practical question, and the only way to keep them from becoming abstract numbers is to walk through the same concrete query with real documents.
I am going to do exactly that. One query. Three systems. The same relevance judgments. By the end of this post you will be able to look at any of the three metrics and know what it is actually measuring on this collection.
The benchmark is the conversation starter, not the verdict.
Why Metrics Need a Concrete Example
I have watched experienced engineers argue past each other for an hour because one was looking at Recall and the other was looking at MRR. Both numbers can be “right” and still describe completely different user experiences. The fix is not to pick a single favorite metric. The fix is to keep a shared example in mind so the numbers stay attached to documents and ranks.
The example I will use comes from the small evaluation set described in the previous post. It is deliberately ordinary. That is the point.
The Concrete Query and the Relevant Documents
Query: “how do I rotate the API key after it has been compromised”
There are three documents in the collection that are actually useful:
Doc A (highly relevant, score 2): A clear, step-by-step note titled “Rotating a Compromised API Key” that lists the exact commands and the order of operations.
Doc B (relevant, score 2): A slightly older troubleshooting page that covers the same procedure but also includes a warning about cached tokens.
Doc C (partially relevant, score 1): A general security checklist that mentions key rotation as one item among many; it is useful context but does not give the full procedure.
Everything else in the top 10 is either off-topic or only loosely related (score 0).
This is the ground truth I will use for all three metrics. The numbers below are calculated from the actual ranked lists produced by BM25, EmbeddingGemma 2, and the simple hybrid on this query.
The Three Ranked Lists
For readability I show only the rank and the relevance label of each result.
BM25 top 10
Doc A (2)
unrelated (0)
unrelated (0)
Doc C (1)
unrelated (0)
unrelated (0)
Doc B (2)
unrelated (0)
unrelated (0)
unrelated (0)
EmbeddingGemma 2 top 10
Doc B (2)
Doc A (2)
unrelated (0)
unrelated (0)
Doc C (1)
unrelated (0)
unrelated (0)
unrelated (0)
unrelated (0)
unrelated (0)
Hybrid (0.6 dense + 0.4 BM25) top 10
Doc A (2)
Doc B (2)
Doc C (1)
unrelated (0)
unrelated (0)
unrelated (0)
unrelated (0)
unrelated (0)
unrelated (0)
unrelated (0)
Now we can compute each metric on the same data.
Recall@10: Did the Good Documents Appear?
Recall@10 is simply the fraction of relevant documents that appear anywhere in the top 10.
There are three relevant documents (A, B, and C).
BM25 found all three → Recall@10 = 3/3 = 1.00
EmbeddingGemma 2 found all three → Recall@10 = 1.00
Hybrid found all three → Recall@10 = 1.00
On this particular query every system achieved perfect recall. That is useful information: none of them completely missed the good documents. It is also limited information. Recall does not care that BM25 buried Doc B at rank 7 or that the hybrid put the three useful documents in the first three positions. It only cares that they appeared somewhere in the window.
Practical interpretation: if Recall@10 is low, users are missing answers. If it is high, the documents are at least present and the ranking problem becomes one of order rather than coverage.
MRR: How Quickly Did the First Good Document Appear?
MRR looks only at the rank of the first relevant document and takes its reciprocal. The mean is then taken across queries; here we are looking at the single-query reciprocal rank.
BM25: first relevant document is at rank 1 → reciprocal = 1/1 = 1.00
EmbeddingGemma 2: first relevant document is at rank 1 → reciprocal = 1.00
Hybrid: first relevant document is at rank 1 → reciprocal = 1.00
Again the numbers are identical on this query. MRR would have differentiated the systems if one of them had placed the first useful document at rank 4 or 5. In that case the reciprocal would drop to 0.25 or 0.20 and the gap would be visible.
Practical interpretation: MRR is closest to the experience of a user who stops at the first useful result. It is less informative when there are multiple good documents and the user is willing to scan further.
NDCG@10: How Well Was the Whole Top of the List Ordered?
NDCG is the metric that finally separates the three systems on this example. It rewards putting higher-relevance documents higher in the list and applies a logarithmic discount so that rank 1 matters more than rank 7.
I will skip the full formula derivation and show the practical result. (The companion repository contains the exact calculation so you can verify every step.)
Using the standard graded relevance and the usual DCG / IDCG normalization:
BM25 NDCG@10 ≈ 0.78
EmbeddingGemma 2 NDCG@10 ≈ 0.89
Hybrid NDCG@10 ≈ 0.97
The hybrid wins because it placed the two fully relevant documents in positions 1 and 2 and the partially relevant document in position 3. EmbeddingGemma 2 was close but swapped the order of A and B and pushed C to rank 5. BM25 found everything but scattered the relevant documents, placing one of them at rank 7 where the discount is already severe.
Practical interpretation: NDCG is the metric I look at when I care about the quality of the entire result page, not just the first hit or the presence of answers. It is also the metric most sensitive to the difference between a “good enough” ranking and a clearly better one.
Putting the Three Numbers Side by Side
System | Recall@10 | Reciprocal Rank | NDCG@10 |
|---|---|---|---|
BM25 | 1.00 | 1.00 | 0.78 |
EmbeddingGemma 2 | 1.00 | 1.00 | 0.89 |
Hybrid | 1.00 | 1.00 | 0.97 |
On this single query the first two metrics are identical and the third reveals a clear ordering. Across the full 80-query set the picture is more varied: there are queries where BM25 has higher Recall, queries where dense has higher MRR, and queries where the hybrid’s NDCG advantage is large. That variation is why I report all three.
How I Decide Which Metric to Emphasize
I do not have a single favorite. The choice depends on the product question I am trying to answer.
If the system is a high-stakes search where missing a relevant document is expensive, I watch Recall@10 first.
If users typically click the first useful result and leave, I watch MRR.
If users scan a result page or if the interface shows ten results at once, I watch NDCG@10.
In practice I look at all three and then read the actual lists for the queries that moved the most. The metrics tell me where to look; the documents tell me whether the change is meaningful.
A Few Common Misunderstandings
“Higher is always better.”
Not across different metrics. A system can improve NDCG while slightly lowering Recall if it trades coverage for better ordering of the documents it does return. Both changes can be rational depending on the product.
“NDCG is the only metric that matters.”
It is the most comprehensive of the three for graded relevance, but it can hide a complete miss if that miss is compensated by excellent ordering of the documents that remain. That is why I still report Recall.
“These metrics work the same on every collection.”
They do not. On a collection with many relevant documents per query, Recall@10 becomes easier and NDCG becomes more important. On a collection with mostly binary relevance and one good document per query, MRR and Recall converge. Always look at the relevance distribution of your own set before interpreting the numbers.
How This Example Fits the Larger Series
The same three metrics, calculated the same way, will appear in every Retrieval Bench post that follows. When I say “EmbeddingGemma 2 improved NDCG@10 by 0.07,” you will know it means the same kind of ordering improvement illustrated above. When I say the hybrid recovered Recall, you will know it means documents that one system had pushed beyond rank 10 came back into the window.
The concrete example is the anchor. Everything else is comparison against that shared understanding.
Closing
Recall@10, MRR, and NDCG@10 are not competing answers to the same question. They are complementary answers to three different questions about the same ranked list. The only reliable way to keep them useful is to keep a concrete query and its documents in mind while you read the numbers.
In the next post I will step back and talk about the larger posture this series takes toward every benchmark result: treat the number as the start of a conversation, not the end of one. After that we move into the reproducible pipeline so you can run the same metrics on the same data yourself.
Until then, when someone shows you a single leaderboard score, ask which of the three questions it is actually answering—and whether the documents behind the number have been read.
The benchmark is the conversation starter, not the verdict.
No feedback yet — submit the first.