The Five Files Every Reproducible Retrieval Repository Should Include

Here is the short version.
After building the local pipeline, the Dockerized Qdrant service, the hybrid ranker, and the command-line evaluation tool, one pattern became obvious: the repositories that stay usable six months later all contain the same five files in roughly the same places. Everything else can vary. These five cannot. They are the minimum set that lets another person (or future you) understand, run, and modify the stack without reverse-engineering a notebook or guessing at hidden state.

This post names the five files, explains why each one earns its place, and shows the exact content I now require before I consider a retrieval experiment “done.”

The Five Files

1. README.md

2. requirements.txt (or equivalent lock file)

3. run_eval.py (or the single entry-point command)

4. docker-compose.yml

5. configs/default.yaml (or a clearly named configuration file)

That is the entire mandatory list. You can add more—tests, notebooks, data cards, CI workflows—but if any of these five is missing or incomplete, the repository is not yet reproducible by the standard used on this site.

1. README.md — The Map

A README is not marketing copy. It is the shortest path from zero to a verified result.

Mine always contain, in this order:

• One-paragraph purpose statement

• Exact prerequisites (OS, RAM, disk, Python version)

• The single command that builds and starts the stack

• The single command that runs the evaluation and prints the metrics

• A short “What you should see” section with example output

• A limitations section that states what the repository deliberately does not do

• A pointer to the configuration file and the evaluation data

I keep the README under 200 lines. If it grows longer, material moves into linked documents. The test of a good README is simple: a competent engineer who has never seen the project should be able to produce the published metrics in under thirty minutes.

2. requirements.txt — The Exact Environment

Floating dependencies are how “it worked last month” becomes “it does not work today.”

I pin both the top-level packages and, when practical, the transitive ones that have broken me before:

text

torch==2.2.2
transformers==4.40.2
sentence-transformers==3.0.1
qdrant-client==1.10.1
rank-bm25==0.2.2
nltk==3.8.1
fastapi==0.110.3
uvicorn==0.27.1
numpy==1.26.4
pyyaml==6.0.1
tqdm==4.66.2

When I use Docker, the same pins appear in the image. When I run locally, I install from the identical file. I also record the Python version (3.11.x) in the README. Environment drift is the most common reason a supposedly reproducible retrieval experiment stops reproducing.

3. run_eval.py — The Single Source of Truth for Numbers

Every published metric on this site must be producible by one command. That command lives in run_eval.py (or a similarly obvious name).

The script:

• Accepts paths to documents, queries, and judgments

• Accepts model name, alpha, device, and output path as arguments

• Writes a self-describing JSON result that includes the metrics, the configuration, and a timestamp

• Prints a one-line summary so you can see the numbers without opening the file

No hidden notebooks, no manual cell execution, no reliance on global state left over from a previous run. If the number cannot be regenerated by this script, it does not belong in a post.

4. docker-compose.yml — The Runtime Contract

Local virtual environments are convenient until someone else tries to use them. Docker Compose turns the runtime into a declarative artifact.

The file I require does three things:

• Defines the Qdrant service with a persistent volume

• Defines the embedder / hybrid service that talks to it

• Mounts the documents and configuration so that changing data does not require rebuilding the image

A new contributor should be able to run docker compose up --build and then exercise the same HTTP endpoints or the same run_eval.py (via docker compose exec) that I used. If the compose file is missing or only works on my exact machine, the repository is not finished.

5. configs/default.yaml — The Explicit Choices

Magic numbers scattered through code are the enemy of reproducibility. Every important choice—model identifier, fusion weight, batch size, collection name, score normalization method—lives in a configuration file.

Example:

YAML

model:
  name: google/embeddinggemma-2
  device: cpu
  batch_size: 8

hybrid:
  alpha: 0.6
  over_retrieve: 2

qdrant:
  host: qdrant
  port: 6333
  collection: documents

paths:
  documents: /app/documents
  bm25_index: /app/bm25_index

The evaluation script and the FastAPI service both read this file. Changing alpha or switching models becomes a one-line edit plus a re-run, not a hunt through source. When I publish a result, I also publish (or at least record) the exact configuration that produced it.

Why Only Five

I have seen repositories with dozens of carefully written files that still cannot be run by a second person. The failure is almost never lack of code. It is lack of a clear entry point, an ambiguous environment, or undocumented configuration. The five files above directly attack those failure modes and nothing else.

Everything beyond the five is optional until the five are solid:

• Notebooks are welcome for exploration; they are not the delivery mechanism.

• Test suites are valuable; they are secondary to the ability to regenerate the published metrics.

• Fancy packaging (Poetry, uv, etc.) can come later; a pinned requirements.txt is enough to start.

• CI is excellent once the local and Docker paths already work.

A Minimal Repository Skeleton

text

retrieval-experiment/
├── README.md
├── requirements.txt
├── docker-compose.yml
├── configs/
│   └── default.yaml
├── scripts/
│   └── run_eval.py
├── embedder/               # FastAPI + hybrid logic
├── documents/              # or a download script
├── data/                   # queries + judgments
└── results/                # gitignored, but the schema is documented

That layout has survived every experiment in this series. When I open it months later, I know exactly where to look.

Practical Checks Before Declaring “Done”

I ask four questions:

1. Can a stranger produce the published metrics with only the README and the five files?

2. Does docker compose up followed by the evaluation command succeed on a clean machine?

3. Is every number in the result JSON traceable to an explicit configuration value?

4. If I change alpha or the model name in configs/default.yaml, does the next run reflect that change without code edits?

If any answer is no, the repository is not yet reproducible.

Closing

Reproducibility is not a feature you add at the end. It is the discipline of putting the same five files in the same places and refusing to publish numbers that cannot be regenerated from them.

Take any of the pipelines built in the previous posts and give it these five files today. Once they exist and the evaluation command works from a clean checkout, the experiment has become a durable artifact rather than a temporary notebook state.

With this post the Reproducible Builds arc is complete. The next articles move into Small-System Reality: what EmbeddingGemma 2 actually costs in memory, latency, and electricity on the machines individual developers and small teams already own.

Until then, make the five files non-negotiable. Everything else in the repository can evolve; these five keep the work honest.

The benchmark is the conversation starter, not the verdict.