Here is the short version.
You do not need a massive academic benchmark to learn whether your retrieval system is useful. You need a small, honest collection of real documents, a short list of real queries, and relevance judgments you are willing to defend. The set I describe below contains 1,200 documents and 80 queries. It is deliberately imperfect. It is also the most useful evaluation resource I currently own, because every number I report from it is tied to documents and questions I actually understand.
This post walks through exactly how I built that set, why I made each design choice, and where the limits sit. The goal is not to give you a universal test collection. The goal is to show a repeatable process that stays small enough to finish in a weekend and honest enough that the metrics still mean something.
The benchmark is the conversation starter, not the verdict.
Why Most Off-the-Shelf Benchmarks Quietly Mislead Small Teams
I spent years evaluating ranking systems at scale. The standard academic collections are excellent for comparing methods under controlled conditions. They are less excellent when your actual problem is a private archive of support notes, policy snippets, and half-written internal docs that never looked like Wikipedia.
Three common mismatches keep showing up:
Cleanliness. Academic documents are usually well-formed and non-overlapping. Real collections contain near-duplicates, conflicting versions, and documents that share vocabulary but answer different questions.
Query style. Benchmark queries are often written by annotators who already know the collection. Real users type incomplete, ambiguous, or slightly wrong questions.
Scale illusion. A system that looks strong on 100,000 clean documents can still fail on 1,200 messy ones if the failure modes are different.
I do not discard the big benchmarks. I simply refuse to treat them as the only source of truth for the systems I actually ship. A small evaluation set built from the same distribution as the target data is often more diagnostic for the first several months of a project.
Design Goals for the Small Set
Before writing a single query I wrote down four constraints:
The entire collection must fit comfortably in memory on a 16 GB machine so that every experiment stays fast.
Building and judging the set must be finishable in two focused weekends.
The documents must contain the kinds of noise I see in real private collections (near-duplicates, version drift, mixed formality).
Every relevance judgment must be made by a person who has read the documents, not by an automatic heuristic.
Those constraints produced a target of roughly 1,000–1,500 documents and 60–100 queries. I landed on 1,200 documents and 80 queries. The numbers are not magic; they are simply large enough to surface patterns and small enough that I can still read the top-10 lists for every query without losing a week.
Step 1: Collecting the Documents
I started with material I already had permission to use and that resembled the support and internal-documentation domain I care about most:
Anonymized product FAQ entries
Troubleshooting notes written by engineers
Short policy and process snippets
A handful of deliberately created near-duplicates and lightly conflicting versions
I removed any personally identifying information and any content that could not be shared. The final collection is a mix of short answers (a few sentences) and medium-length notes (a few hundred words). Average document length is roughly 180 words—long enough to contain real semantic content, short enough that embedding and indexing stay cheap.
I did not try to make the collection perfectly balanced or perfectly clean. I left in a few near-duplicates on purpose. I left in a few documents that use overlapping vocabulary for different procedures. Those imperfections are the point. If a ranking method collapses under mild duplication or vocabulary overlap, I want to know it early.
Step 2: Writing the Queries
Queries were written in three deliberate styles:
Exact or near-exact lexical matches – questions that use the same key terms as the best document.
Paraphrases – the same intent expressed with different words.
Mildly underspecified or ambiguous – questions that a real user might type when they only half-remember the problem.
I wrote the first 40 queries myself while looking at the collection. I then asked two colleagues who had never seen the documents to write another 40 from a short description of the domain. Their queries were noticeably more varied and occasionally more realistic than mine. I kept most of them.
The final 80 queries are all natural-language questions a person might actually type. None of them are keyword salads. None of them are artificially long. That choice keeps the evaluation closer to the interaction pattern I care about: a busy person typing a short question into a search box.
Step 3: Making Relevance Judgments
This is the step most people skip or automate too early. I did not.
For every query I retrieved the top 20 results from both BM25 and EmbeddingGemma 2, merged the lists, and judged every unique document that appeared. Judgments used a simple three-point scale:
2 = directly answers the question
1 = partially useful or related but incomplete
0 = not useful
I judged in batches of ten queries, with at least a day between batches, to reduce fatigue and consistency drift. Borderline cases were marked and revisited at the end. The entire judging process took roughly twelve focused hours spread across several evenings.
I am the primary judge. That is both a strength and a limitation. The strength is that every label is grounded in a careful reading of the documents. The limitation is that another careful reader might disagree on the borderline cases. I accept that trade-off for a set this size. When the collection grows or the stakes rise, I will bring in a second judge and measure agreement. For now, transparency about who judged is more important than pretending the labels are objective truth.
Step 4: Checking the Set for Obvious Pathologies
Before locking the set I ran a few sanity checks:
Query length distribution (most queries are 6–14 words)
Document length distribution
Number of relevant documents per query (median is 2, a few queries have 4–5, a few have only 1)
Overlap between the BM25 and dense top-10 lists (they agree on some queries and diverge sharply on others—exactly the behavior I want to study)
I also forced myself to read the top-10 lists for the ten queries where the two systems disagreed most. That manual inspection caught two labeling errors and one near-duplicate cluster that was distorting results. Both were fixed before the set was frozen.
What the Finished Set Looks Like
1,200 documents
80 queries
Relevance judgments for every document that appeared in the top 20 of either system
A simple JSON format that any evaluation script can read
A short README that states the collection is small, domain-specific, and not a universal benchmark
The companion repository will contain the full set so anyone following this series can reproduce the exact numbers I report. If you prefer to build your own, the process above is the entire recipe.
How I Use the Set (and How I Do Not)
I use it to answer questions like:
Does EmbeddingGemma 2 surface the right documents more often than BM25 on this distribution?
Does a simple hybrid recover the cases each system misses?
When I change quantization or approximate nearest-neighbor settings, how much ranking quality moves?
I do not use it to claim that one model is universally better than another. I do not treat a 0.05 gain in NDCG@10 as a product decision by itself. I always read the actual retrieved documents for the queries that moved the most. The metrics tell me where to look; the documents tell me whether the change is real.
Limitations You Should Keep in Mind
The set is small. Patterns that appear here may weaken or reverse on larger or differently distributed collections.
The domain is technical English support and internal documentation. Behavior on legal text, medical notes, or other languages is outside the scope.
Relevance is judged by one primary reader. Inter-annotator agreement has not yet been measured.
The documents were selected and lightly curated by me. A completely organic crawl of a real company wiki would contain additional messiness I have not captured.
These limits are not flaws to be hidden. They are the reason the set is useful. A small, well-understood evaluation resource teaches different lessons from a large, opaque one. Both have value. I simply refuse to pretend the small one is the large one.
Practical Advice If You Build Your Own
Start with documents you already understand. Judging is much faster when you recognize the content.
Write or collect queries in more than one style. Pure keyword queries and pure paraphrase queries stress different parts of the system.
Judge more results than you plan to evaluate. Looking at the top 20 instead of the top 10 catches ranking problems that pure Recall@10 will miss.
Budget real time for reading the lists. The highest-leverage hour I spent on this set was the hour I spent reading disagreements between BM25 and the dense model.
Publish the set if you can. Even a small public collection with clear documentation is more valuable than another private spreadsheet.
Closing
A retrieval evaluation set does not need to be universal to be useful. It needs to be honest about its size, its domain, and its limitations. The 1,200-document collection described here is the foundation for the comparisons that follow on this site. Every Recall@10, MRR, and NDCG number I publish from it can be traced back to documents and judgments I can explain.
If you build a similar set for your own data, you will learn more about your ranking problems in one weekend than you will learn from another leaderboard screenshot. That is the entire point.
Next we will put the set to work in a full comparison of metrics explained through one concrete experiment, and then move into the reproducible pipeline so anyone can run the same tests.
Until then, keep the collection small, the judgments careful, and the claims proportional to the evidence.
The benchmark is the conversation starter, not the verdict.
No feedback yet — submit the first.