Here is the short version.
A single number on a retrieval benchmark is useful the moment it makes you ask better questions. It becomes dangerous the moment you treat it as a final ranking of models, teams, or approaches. This post is the operating principle behind every experiment that will appear on Semantic Workshop. The title is also the signature phrase I intend to keep repeating until it becomes automatic.
I am writing this after three concrete posts: the origin of the site, the first head-to-head comparison of EmbeddingGemma 2 against BM25, and a walk-through of how the small evaluation set and its metrics actually work. Those posts produced real numbers. This one is about what those numbers are allowed to mean.
Why the Phrase Exists
I have sat in too many meetings where a 0.04 gain in NDCG was treated as decisive evidence that one system should replace another. I have also watched people dismiss a solid experimental result because it did not match a public leaderboard run on a completely different distribution. Both mistakes share the same root: treating the benchmark as a verdict instead of a conversation starter.
A conversation starter does three useful things:
It surfaces a difference worth investigating.
It forces you to name the conditions under which the difference appeared.
It invites the next, more careful question.
A verdict does the opposite. It ends the discussion. It encourages people to stop reading the actual retrieved documents, stop checking hardware cost, and stop asking whether the test collection still looks like the data they will serve in production.
I prefer the first pattern. The rest of this post explains how I try to enforce it.
What a Benchmark Result Is Allowed to Claim
When I report that the hybrid system reached 0.71 NDCG@10 on the 80-query set, the claim is narrowly scoped:
On this collection
With these relevance judgments
Using this exact embedding model, this BM25 configuration, and this fusion weight
On this hardware and software stack
Under the evaluation script published in the repository
That is already a lot of information. It is also deliberately limited. The result does not claim that the hybrid will win on legal documents, on a different language, on a ten-times-larger corpus, or after the vector index has been approximated. It does not claim that every team should adopt the same weights. It simply says: under the conditions I actually tested, this is what happened.
If a later experiment on a different collection reverses the ranking, that is not a contradiction. It is the next sentence in the conversation.
The Questions a Good Benchmark Result Should Trigger
After any number appears, I try to ask the following in order:
What exactly moved?
Was it Recall, MRR, or NDCG? Did the gain come from a few queries or from a broad improvement?Which documents changed rank?
I always read the lists for the queries that moved the most. Metrics can improve while the new top results become more plausible-looking failures.What was the cost of the gain?
Extra memory, higher latency after the first request, longer indexing time, more complex deployment. A quality improvement that doubles the hardware requirement is a different decision from one that fits in the same envelope.Under what conditions would I expect the result to change?
Different document length distribution, more near-duplicates, queries that rely on rare proper nouns, a shift from exact-match heavy traffic to paraphrase heavy traffic.What is the smallest follow-up experiment that would increase my confidence?
Sometimes that follow-up is simply “run the same test on a second small collection.” Sometimes it is “add a second judge and measure agreement.” Sometimes it is “measure the same ranking quality behind an approximate index.”
If a published result does not help me answer at least the first three questions, I treat it as incomplete.
How This Principle Shapes the Rest of the Series
Every future Retrieval Bench post will try to obey the same discipline.
Numbers will appear with the exact test conditions attached.
I will show at least one concrete query and its ranked lists when the metrics move.
Limitations will be listed in plain language rather than buried in a final paragraph.
When an experiment is inconclusive or negative, it will still be published if it teaches something about the failure mode.
The same posture applies outside pure ranking quality. When I later measure embedding throughput on a Mac Mini versus a low-power NUC, the numbers will be conversation starters about hardware choice, not verdicts that one machine is universally better. When I measure the monthly electricity cost of a small private retrieval service, the figure will be a starting point for budgeting discussions, not a claim that every team will see the same cost.
A Personal Note on Why This Matters
I left a large company environment in part because I was tired of seeing carefully measured offline gains ignored, and tired of seeing unmeasured claims accepted because they arrived with a confident slide deck. Working independently has not removed the pressure to sound certain. It has simply made the cost of being wrong more personal: wasted evenings, confusing results for anyone who follows the work, and a slow erosion of trust in my own notes.
Keeping the benchmark in its proper place is one of the ways I protect that trust. The moment I start writing as if a single experiment has settled the question, the work becomes less useful to me and less honest to anyone reading it.
I still like strong results. I still celebrate when the hybrid recovers a set of queries that pure dense retrieval missed. I just refuse to let the celebration become a closing argument.
What I Ask of Readers
If you use any of the numbers published on this site, I ask for the same discipline I am trying to practice:
Cite the conditions, not only the headline metric.
Read the limitations section before generalizing.
Treat a contradictory result on your own data as information, not as a reason to discard the earlier experiment.
Prefer a small, well-understood test you can re-run over a large opaque leaderboard you cannot inspect.
If you find a case where one of my reported results does not hold under clearly stated conditions, I want to know. That is the conversation continuing, not the verdict being overturned.
Closing the Opening Arc
The first five posts on Semantic Workshop were designed as a single arc:
Why the site exists
The first concrete comparison
How the evaluation set was built
How the three core metrics actually work on one example
The principle that governs every number that follows
From here the work becomes more applied. The next posts move into reproducible pipelines, Dockerized workflows, hybrid retrievers that can be started with a single command, and the small-system measurements that determine whether any of this is practical on the machines people actually own.
The numbers will keep appearing. The principle stays the same.
A benchmark result is the beginning of a discussion about what was tested, what was held constant, what was left out, and what should be tried next. It is never the final word.
The benchmark is the conversation starter, not the verdict.
No feedback yet — submit the first.