EmbeddingGemma 2 vs. BM25: The First Reproducible Comparison

Here is the short version.
On a small, deliberately noisy support-style dataset of 1,200 documents and 80 natural-language queries, EmbeddingGemma 2 alone beat BM25 on Recall@10 and MRR. A simple hybrid that fused the two scores beat both of them. The absolute numbers are modest. The interesting part is not that dense retrieval “won.” The interesting part is how quickly the ranking changed once I looked at the actual retrieved documents and the hardware cost of producing them.

I ran everything on the same machine I use for weekend experiments: a 2023 Mac Mini with the M2 chip, 16 GB of unified memory, and no external GPU. No cloud APIs. No cherry-picked clean Wikipedia dump. Just the kind of messy, overlapping, real-world text that shows up when you try to build private search for a small team or a family archive.

This is the first concrete comparison on Semantic Workshop. The benchmark is the conversation starter, not the verdict.

Why This Particular Comparison

Most embedding model announcements stop at a leaderboard score. They rarely show the BM25 baseline on the same data, the same queries, and the same evaluation script. They almost never show what happens when you combine the two signals. That gap is exactly what this series exists to close.

I chose BM25 as the baseline for three practical reasons:

1. It is still the default in many production systems that cannot afford (or do not want) dense retrieval.

2. It is fast, deterministic, and requires almost no memory beyond the inverted index.

3. It fails in predictable ways that dense models sometimes fix—and sometimes make worse.

EmbeddingGemma 2 was the dense model under test because it is the consistent center of gravity for this site: small enough to run locally, recent enough that the community discussion is still forming, and already showing up in a lot of “local RAG” tutorials that skip the hard measurements.

Test Setup

Dataset

I built a small support-style collection instead of reaching for an existing academic benchmark. Academic sets are useful, but they are also cleaner and more homogeneous than the documents people actually own. My collection contains:

• 1,200 short-to-medium documents (product FAQs, internal troubleshooting notes, policy snippets, and a handful of deliberately overlapping or near-duplicate entries)

• 80 queries written in the style of real user questions (some exact keyword matches, some paraphrases, some that require mild semantic understanding)

• Manually judged relevance labels for the top results of each system so the metrics are grounded in the same judgments

The collection is intentionally imperfect. There are near-duplicates, documents that share vocabulary but answer different questions, and a few queries that are underspecified. That noise is a feature. Clean data makes every model look good. Noisy data shows where the ranking actually breaks.

I will publish the evaluation set and the exact judgment files in the companion repository so anyone can re-run the numbers.

Hardware and Software

• Machine: Mac Mini M2, 16 GB unified memory

• EmbeddingGemma 2 loaded in float16 via the transformers library, batched at size 8

• BM25 via rank_bm25 with default parameters (k1=1.5, b=0.75)

• Hybrid fusion: simple weighted sum of min-max normalized scores (0.6 dense + 0.4 BM25 after a quick grid search on a held-out slice)

• Evaluation script: custom but short—Recall@10, MRR, and NDCG@10 calculated from the same relevance judgments

No quantization beyond float16. No vector database yet; this first comparison is pure ranking quality before the infrastructure layer is added. Latency and memory numbers are reported later in the post so the quality results stay separate from the systems results.

Metrics, Explained Once

I will use three metrics throughout this series. Here they are in plain English, tied to the same concrete example.

Imagine a query “how do I reset the API key after rotation.” There are three documents that truly answer it.

• Recall@10 asks: of those three relevant documents, how many appear in the top 10 results? If two of them do, Recall@10 is 0.67.

• MRR (Mean Reciprocal Rank) cares about the position of the first relevant document. If the first useful result is at rank 1, the reciprocal is 1.0. If it is at rank 4, the reciprocal is 0.25. Averaged across all queries.

• NDCG@10 rewards putting the most relevant documents higher and penalizes putting them lower. It is more forgiving of multiple relevant items than MRR and more sensitive to ranking quality inside the top 10 than pure Recall.

I report all three because each one answers a different practical question. Recall tells you whether the good documents are even present. MRR tells you how quickly a user sees the first useful answer. NDCG tells you how well the whole top of the list is ordered.

Results

On the 80-query set the numbers came out as follows (rounded for readability):

Dense retrieval improved every metric over BM25. The hybrid improved them further. The absolute gains are real but not dramatic. That is expected on a small, noisy collection. What mattered more was looking at the actual lists.

Where Dense Helped

Queries that used different vocabulary from the documents (“rotate credentials” instead of “reset API key”) were the clearest wins for EmbeddingGemma 2. BM25 often returned near-misses that shared surface words but answered a different question. The dense model more often surface the right document even when the lexical overlap was low.

Where BM25 Still Won Individual Queries

Exact-match or near-exact queries, especially those containing product codes, error strings, or rare proper nouns, still favored BM25. In a few cases the dense model ranked a semantically similar but factually wrong document higher than the exact match. That is a known failure mode and one reason the hybrid exists.

Where the Hybrid Earned Its Keep

The hybrid recovered most of the dense wins while protecting the exact-match cases that BM25 handled cleanly. It was not perfect—there were still queries where both signals agreed on a mediocre result—but it was the most consistent of the three systems across the set.

What the Numbers Do Not Tell You

I spent an evening reading the top-10 lists for every query. Three patterns stood out that the metrics alone hide.

1. Near-duplicate pollution. When the collection contained two almost identical troubleshooting notes, both systems sometimes returned both of them in the top 10, wasting slots. Dense retrieval was slightly more prone to this because the embeddings of near-duplicates are very close.

2. Semantic near-misses. EmbeddingGemma 2 occasionally preferred a document that discussed a related but different procedure. The cosine similarity looked good; the actual answer was wrong. BM25’s failures were more often “completely irrelevant.” Dense failures were more often “plausibly wrong.”

3. Query ambiguity. Several queries were under-specified on purpose. In those cases the ranking quality depended more on which interpretation the model latched onto than on any inherent superiority of dense or sparse.

These are the kinds of observations that only appear when you treat the benchmark as a conversation starter instead of a final ranking.

Reproduction Notes

Everything needed to re-run the comparison will live in the companion repository under a simple structure:

• data/ – the document collection and the query file

• judgments/ – the relevance labels

• scripts/run_bm25.py, scripts/run_dense.py, scripts/run_hybrid.py

• scripts/evaluate.py

• requirements.txt and a short README that lists the exact library versions I used

The embedding step took roughly 4.5 minutes for the full 1,200 documents on the Mac Mini at batch size 8. Query embedding was effectively instant. BM25 indexing was under 10 seconds. The hybrid fusion is a few lines of NumPy.

If you run it on different hardware or with different batch sizes, the ranking quality should stay the same; only the wall-clock time and peak memory will change. I will publish the memory and latency numbers for this exact run in the next Small-System Reality post. For now the focus stays on ranking quality.

Limitations (Read These Before You Generalize)

• The collection is small. 1,200 documents and 80 queries are enough to see patterns and not enough to claim universal superiority.

• The relevance judgments are mine. Another judge could label borderline cases differently.

• I used a single set of BM25 parameters and a single hybrid weight. A more exhaustive search might shift the numbers a few points.

• No vector database, no approximate nearest-neighbor index, no quantization beyond float16. Those layers introduce their own quality and latency trade-offs that will be measured later.

• The documents are English and mostly technical. Behavior on other languages or domains is outside the scope of this test.

Any claim that “EmbeddingGemma 2 is better than BM25” without these caveats is already overstating what this experiment shows.

Practical Takeaways for Small Teams

If you are building private search or a small RAG system and you only have time for one ranking experiment this week, do this:

1. Keep BM25. It is cheap, fast, and still wins on exact-match queries.

2. Add EmbeddingGemma 2 (or whatever dense model you can actually run locally).

3. Fuse the scores with a simple weighted sum and tune the weight on a handful of real queries rather than on a leaderboard.

4. Read the actual top-10 lists. The metrics will tell you the average story; the documents will tell you where the system is quietly failing.

That combination—BM25 + a local dense model + a light fusion layer—is the smallest setup I currently trust for the kinds of collections people actually own.

Closing

This was the first reproducible comparison on the site. Dense retrieval improved the numbers. Hybrid improved them further. The absolute gains were modest, the failure modes were instructive, and the hardware cost was low enough that the experiment is practical to repeat.

Next I will publish the exact repository, then move to the Dockerized pipeline so the same comparison can be run by anyone who can start a container. After that we will look at what happens to latency and memory when the same models sit behind a real vector database.

Until then, treat the numbers as the start of the discussion, not the end of it.

The benchmark is the conversation starter, not the verdict.