Build a Hybrid Retriever with BM25 and Dense Embeddings

Here is the short version.
Pure dense retrieval with EmbeddingGemma 2 already beats BM25 on the small evaluation set I use. A simple hybrid that fuses the two signals beats both of them. The implementation does not require a research library or a complicated training loop. It requires the BM25 index you already know how to build, the dense vectors you already know how to store in Qdrant, and a few dozen lines of score normalization and weighted addition.

This post takes the Dockerized EmbeddingGemma 2 + Qdrant stack from the previous article and adds a BM25 component plus a hybrid ranker. At the end you will have a single search endpoint that returns results from BM25 alone, dense alone, or the fused hybrid, all from the same containers.

I ran the exact code below on the same Mac Mini (16 GB) and Ubuntu laptop used for the earlier posts. The quality numbers match the first Retrieval Bench comparison; the focus here is on making the hybrid reproducible and easy to run.

Why Hybrid Still Matters

Dense models are strong on paraphrase and semantic matches. BM25 remains stronger on exact tokens, product codes, error strings, and rare proper nouns. Real collections contain both kinds of queries. A hybrid that keeps the strengths of each side is still the smallest reliable improvement I know for small-team retrieval systems.

The fusion method I use is deliberately boring: min-max normalize the two score lists, then compute a weighted sum. No learning-to-rank, no query-dependent weights, no extra model. The boring version is easy to debug, easy to explain, and already good enough to move the metrics on the evaluation set I care about.

Updated Layout

text

hybrid-pipeline/
├── docker-compose.yml
├── embedder/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py
│   └── entrypoint.sh
├── documents/
├── qdrant_storage/
├── bm25_index/          # persisted BM25 artifacts
└── README.md

Only two new pieces appear: a small on-disk BM25 index and the hybrid logic inside the same FastAPI service.

Dependencies

Add two packages to embedder/requirements.txt:

text

rank-bm25>=0.2.2
nltk>=3.8.1

Everything else stays the same as the previous Docker post. The BM25 index is built with pure Python and stored as a couple of pickle files; it does not need its own container.

Building the BM25 Index

I keep the BM25 construction in the same service so there is only one place to trigger indexing. On startup or via an explicit /index call the service:

1. Reads every .txt file in /app/documents

2. Tokenizes with a simple lowercase + NLTK word tokenizer

3. Builds a BM25Okapi index

4. Saves the index and the document ID list to /app/bm25_index

Python

from rank_bm25 import BM25Okapi
import nltk
import pickle
from pathlib import Path

nltk.download("punkt", quiet=True)

def build_bm25(docs_dir: Path, index_dir: Path):
    texts, ids = [], []
    for path in sorted(docs_dir.glob("*.txt")):
        text = path.read_text(encoding="utf-8").strip()
        if text:
            texts.append(text)
            ids.append(path.name)
    tokenized = [nltk.word_tokenize(t.lower()) for t in texts]
    bm25 = BM25Okapi(tokenized)
    index_dir.mkdir(exist_ok=True)
    with open(index_dir / "bm25.pkl", "wb") as f:
        pickle.dump(bm25, f)
    with open(index_dir / "ids.pkl", "wb") as f:
        pickle.dump(ids, f)
    return len(ids)

The same /index endpoint that upserts vectors into Qdrant also calls this function. One curl request therefore refreshes both the dense and the sparse sides.

Hybrid Fusion Logic

The core fusion function is short:

Python

import numpy as np

def min_max_normalize(scores):
    scores = np.asarray(scores, dtype=float)
    if scores.max() == scores.min():
        return np.zeros_like(scores)
    return (scores - scores.min()) / (scores.max() - scores.min())

def hybrid_search(query: str, k: int = 10, alpha: float = 0.6):
    # dense
    dense_vector = model.encode([query], normalize_embeddings=True)[0].tolist()
    dense_hits = client.search(
        collection_name=COLLECTION,
        query_vector=dense_vector,
        limit=k * 2,          # over-retrieve a little
    )
    dense_scores = {h.payload["filename"]: h.score for h in dense_hits}

    # bm25
    tokens = nltk.word_tokenize(query.lower())
    bm25_scores_raw = bm25.get_scores(tokens)
    bm25_scores = {ids[i]: float(bm25_scores_raw[i]) for i in range(len(ids))}

    # union of candidates
    candidates = set(dense_scores) | set(bm25_scores)
    d_scores = np.array([dense_scores.get(c, 0.0) for c in candidates])
    b_scores = np.array([bm25_scores.get(c, 0.0) for c in candidates])

    d_norm = min_max_normalize(d_scores)
    b_norm = min_max_normalize(b_scores)
    fused = alpha * d_norm + (1 - alpha) * b_norm

    ranked = sorted(zip(candidates, fused), key=lambda x: x[1], reverse=True)[:k]
    return [{"filename": doc, "score": float(score)} for doc, score in ranked]

alpha = 0.6 is the value that performed best on my 80-query set. It is not sacred. The endpoint accepts an optional alpha parameter so you can experiment without rebuilding anything.

Updated API Endpoints

The service now exposes three search modes from the same process:

• POST /search/dense – pure EmbeddingGemma 2

• POST /search/bm25 – pure BM25

• POST /search/hybrid – fused result

All three accept the same JSON body:

JSON

{"text": "how do I rotate an API key", "limit": 5, "alpha": 0.6}

The /index endpoint remains the single place that rebuilds both the Qdrant collection and the BM25 index.

Running the Hybrid Stack

Bash

docker compose up --build
curl -X POST http://localhost:8000/index
curl -X POST http://localhost:8000/search/hybrid \
  -H "Content-Type: application/json" \
  -d '{"text": "how do I rotate an API key", "limit": 5}'

You can also hit the pure dense and pure BM25 endpoints with the same payload (the alpha field is simply ignored). This makes side-by-side comparison trivial.

Quality Check Against the Evaluation Set

I re-ran the same 80-query evaluation used in the earlier Retrieval Bench post. The numbers were consistent with the original comparison:

The hybrid recovered most of the exact-match cases that pure dense missed and most of the semantic cases that pure BM25 missed. The absolute gains remain modest because the collection is small and noisy; the important part is that the same improvement is now available from a single reproducible service.

Latency and Resource Notes

On the Mac Mini (CPU inside Docker):

• BM25 query: < 5 ms

• Dense query (model already warm): 40–80 ms

• Hybrid (both + fusion): 50–90 ms

The extra cost of the hybrid is dominated by the dense encode, not by the fusion arithmetic. Peak memory stayed essentially the same as the pure dense service; the BM25 index for 1,200 short documents is only a few megabytes.

Failure Modes and Practical Fixes

BM25 returns empty results for some queries
Usually caused by tokenization mismatch. Keep the same tokenizer for indexing and querying. I standardized on nltk.word_tokenize and lower-casing.

Hybrid scores all collapse to zero
This happens when one of the systems returns identical scores for every candidate (rare but possible on tiny collections). The min-max function guards against division by zero; you will simply fall back to the other signal.

Index drift
If you add or remove documents and forget to call /index, the two sides fall out of sync. I keep a simple document-count check at startup and log a warning when the BM25 and Qdrant counts differ.

Alpha sensitivity
On my set the sweet spot sat between 0.55 and 0.70. Outside that range the hybrid gradually collapses toward one of the pure systems. Sweeping alpha on a small held-out slice is cheap and worth doing once.

What This Implementation Still Leaves Out

• Learned fusion weights or query-dependent routing

• Score normalization schemes more sophisticated than min-max (e.g., RRF)

• Incremental BM25 updates without a full rebuild

• A proper configuration file instead of environment variables and hard-coded alpha

All of those are reasonable next steps. None of them is required to have a working, reproducible hybrid that already improves the metrics I care about.

Practical Habits

1. Always expose the pure BM25 and pure dense endpoints alongside the hybrid. When the fused result looks wrong, the separate lists tell you which side caused the problem.

2. Persist both indexes. Rebuilding BM25 is fast; rebuilding dense embeddings is not.

3. Record the exact alpha value used for any published number.

4. Keep the fusion function pure and side-effect free so it can be unit-tested with synthetic scores.

Closing

You now have a Dockerized hybrid retriever that combines EmbeddingGemma 2 and BM25 with a transparent weighted fusion. The same docker compose up that previously gave you dense search now gives you three ranking modes and a measurable quality improvement on the evaluation set used throughout this series.

The next post steps back from features and looks at repository structure: the five files every reproducible retrieval project should include so that six months later you (or anyone else) can still understand, re-run, and modify the stack without archaeology.

Until then, run the hybrid against your own documents, compare the three endpoints on a handful of real queries, and keep the fusion simple enough that you can explain every score.

The benchmark is the conversation starter, not the verdict.