Run Your First EmbeddingGemma 2 Search Pipeline Locally

Here is the short version.
You can have a working EmbeddingGemma 2 semantic search pipeline on a laptop or Mac Mini in under an hour if you keep the scope small and the dependencies honest. This post gives you the exact steps I use, the files that matter, the commands that actually run, and the failure modes I hit so you do not have to rediscover them.

No vector database yet. No Docker. No cloud API keys. Just Python, a modest local model, a folder of documents, and enough code to embed, index in memory, and query. Once this baseline works, everything else in the Reproducible Builds series builds on top of it.

I ran every command below on a 2023 Mac Mini with 16 GB of unified memory. The same steps work on a recent Linux laptop with similar RAM. If your machine is tighter on memory, the notes at the end show the adjustments that kept it alive.

What You Will Have at the End

• A local folder of plain-text documents

• EmbeddingGemma 2 loaded once and reused

• An in-memory index of document embeddings

• A simple search function that returns the top-k most similar documents for any natural-language query

• Timing and memory observations so you know what “local” actually costs on ordinary hardware

This is deliberately the smallest useful pipeline. It is not production-ready. It is the foundation that later posts will turn into a Dockerized service, a hybrid retriever, and a maintainable repository.

Prerequisites

• Python 3.10 or 3.11

• Roughly 8–10 GB of free RAM for comfortable operation (6 GB is possible with smaller batches)

• Ability to install packages with pip

• A terminal and a text editor

I strongly recommend a fresh virtual environment. The model and its dependencies are not exotic, but they are large enough that mixing them with an existing data-science environment often creates version conflicts that waste more time than they save.

Bash

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install --upgrade pip

The Five Files That Matter

I keep the first pipeline to five files so the mental model stays clear:

text

first-pipeline/
├── documents/          # your .txt files
├── embed_and_index.py  # loads the model, embeds, saves the index
├── search.py           # loads the index and runs queries
├── requirements.txt
└── README.md

That is the entire surface area for this post. Later posts will add a vector database, a hybrid ranker, and proper packaging. For today these five pieces are enough.

Step 1: Prepare a Small Document Collection

Create a folder called documents and drop in a few dozen plain-text files. They can be FAQs, meeting notes, personal memos, or any text you are allowed to use. For a first run I recommend 50–200 documents of a few hundred words each. Larger collections work, but the first successful end-to-end pass is more valuable than scale.

If you want a ready-made starter set, the companion repository for this series includes a small anonymized support-style collection. You can also just write ten short text files by hand. The pipeline does not care.

Step 2: Pin the Dependencies

Create requirements.txt with versions that are known to work together on the hardware I tested:

text

torch>=2.1.0
transformers>=4.40.0
sentence-transformers>=3.0.0
numpy>=1.24.0
tqdm>=4.66.0

Install them:

Bash

pip install -r requirements.txt

On Apple Silicon the PyTorch wheel that supports MPS will be pulled automatically. On Linux with a recent CUDA setup you can add the appropriate extra index if you want GPU acceleration; for this first pipeline the CPU or MPS path is sufficient and simpler.

Step 3: Embed and Build the In-Memory Index

The script embed_and_index.py does three things: loads EmbeddingGemma 2 once, walks the documents folder, embeds every file, and writes two artifacts—an embeddings matrix and a simple metadata list—so the search script never has to reload the model.

Here is the core of the script I actually use (simplified for clarity, full version in the repository):

Python

from pathlib import Path
from sentence_transformers import SentenceTransformer
import numpy as np
from tqdm import tqdm
import json

MODEL_NAME = "google/embeddinggemma-2"   # or the exact Hub identifier you prefer
DOCS_DIR = Path("documents")
INDEX_DIR = Path("index")
INDEX_DIR.mkdir(exist_ok=True)

model = SentenceTransformer(MODEL_NAME, device="mps")  # "cpu" or "cuda" as needed

texts = []
ids = []
for path in tqdm(sorted(DOCS_DIR.glob("*.txt"))):
    text = path.read_text(encoding="utf-8").strip()
    if text:
        texts.append(text)
        ids.append(path.name)

embeddings = model.encode(
    texts,
    batch_size=8,
    show_progress_bar=True,
    convert_to_numpy=True,
    normalize_embeddings=True,
)

np.save(INDEX_DIR / "embeddings.npy", embeddings)
with open(INDEX_DIR / "metadata.json", "w") as f:
    json.dump({"ids": ids, "model": MODEL_NAME}, f, indent=2)

print(f"Indexed {len(ids)} documents. Shape: {embeddings.shape}")

Run it:

Bash

python embed_and_index.py

On the Mac Mini this step took roughly 90 seconds for 120 short documents at batch size 8. Peak memory sat around 7.5 GB. If you see pressure, drop the batch size to 4 or 2. The quality of the embeddings does not change; only the wall-clock time and peak RAM do.

Step 4: Search

The search script loads the saved matrix and metadata, embeds the query with the same model, and returns the top-k documents by cosine similarity (which is just a dot product once the vectors are normalized).

Python

from pathlib import Path
from sentence_transformers import SentenceTransformer
import numpy as np
import json

INDEX_DIR = Path("index")
model = SentenceTransformer("google/embeddinggemma-2", device="mps")

embeddings = np.load(INDEX_DIR / "embeddings.npy")
with open(INDEX_DIR / "metadata.json") as f:
    meta = json.load(f)
ids = meta["ids"]

def search(query: str, k: int = 5):
    q = model.encode([query], normalize_embeddings=True)
    scores = (embeddings @ q.T).squeeze()
    top = np.argsort(scores)[::-1][:k]
    return [(ids[i], float(scores[i])) for i in top]

if __name__ == "__main__":
    import sys
    query = " ".join(sys.argv[1:]) or "how do I rotate an API key"
    for doc_id, score in search(query):
        print(f"{score:.3f}  {doc_id}")

Run it:

Bash

python search.py how do I rotate an API key

You should see a short ranked list of document filenames with similarity scores. That is the entire pipeline working end to end.

What I Observed on Real Hardware

• Model load time (first run): 12–18 seconds

• Subsequent query embedding: 30–80 ms

• Full scan of 120 normalized embeddings: well under 1 ms

• Peak RSS while the model stayed resident: approximately 6.8–7.5 GB

These numbers are not impressive by datacenter standards. They are the numbers that matter for a personal or small-team tool that must stay on a machine you already own. Cold start is dominated by model loading; once the model is warm, interactive search feels instantaneous for collections of this size.

Common Failure Modes and How I Fixed Them

Out-of-memory during embedding
Reduce batch size. On 16 GB machines I rarely need to go below 4. On 8 GB machines batch size 2 is the safe default.

Model download hangs or fails
The first run pulls several gigabytes from the Hugging Face Hub. If your network is unstable, set HF_HUB_ENABLE_HF_TRANSFER=1 or pre-download the model with the huggingface-cli.

MPS backend errors on older macOS
Fall back to device="cpu". It is slower but reliable. The ranking quality is identical.

Empty or near-empty documents
The script above skips zero-length files. If your collection contains many boilerplate headers, strip them before embedding or the nearest neighbors become less useful.

Scores all look the same
Confirm that normalize_embeddings=True is set on both the document and query encode calls. Without normalization the dot product is not cosine similarity and the ranking can look strangely flat.

What This Pipeline Deliberately Leaves Out

• Persistent vector database (Qdrant, Chroma, etc.)

• Hybrid fusion with BM25

• Approximate nearest-neighbor indexing

• Authentication, API layer, or multi-user access

• Automatic re-indexing when documents change

Each of those belongs in a later post. Adding them too early makes the first successful run harder and obscures which part of the stack is responsible for any problem you hit.

Practical Advice for Keeping the Baseline Healthy

1. Always embed documents and queries with the identical model and the identical normalization setting.

2. Save both the embedding matrix and the exact model identifier. Future you will thank present you.

3. Keep a tiny held-out set of queries whose expected results you know by heart. Run them after every change.

4. Record wall-clock time and peak memory for the embedding pass. Those two numbers tell you whether the pipeline is still practical on the hardware you care about.

Closing

You now have a minimal, fully local EmbeddingGemma 2 search pipeline. It is small enough to understand, fast enough to iterate on, and honest enough about its hardware cost that you can decide whether the next layer of complexity is worth adding.

The next post in this series takes the same pipeline and puts it inside a Docker container so the environment itself becomes reproducible. After that we add BM25 and a simple hybrid ranker, still keeping every step executable on the same class of machine.

Until then, get the five-file version working on your own documents. Once you can ask a question and see ranked filenames appear, the rest of the stack becomes much easier to reason about.

The benchmark is the conversation starter, not the verdict.