Why I Do Not Start with the Most Expensive GPU

Here is the short version.
The fastest way to slow down a local retrieval project is to begin with the most expensive GPU you can justify. The card arrives, the drivers take an evening, the first embedding run looks impressive, and then the real work—data cleaning, evaluation design, hybrid fusion, failure analysis—still sits untouched. I have made this mistake. The measurements in the previous posts exist in part because I eventually forced myself to start on the hardware I already owned and only add acceleration when a specific, measured bottleneck demanded it.

This post explains the practical reasons I now treat a high-end GPU as a late-stage optimization rather than a starting assumption for EmbeddingGemma 2 and similar small-to-medium embedding workloads.

The Seductive Story

The story goes like this: embedding is matrix multiplication, matrix multiplication loves GPUs, therefore the biggest GPU will produce the best research and the fastest iteration. The story is not false. It is incomplete. It omits the parts of the work that dominate calendar time for most personal and small-team retrieval systems:

• Collecting and cleaning documents

• Writing realistic queries

• Making relevance judgments

• Debugging why a high cosine score still retrieves the wrong document

• Deciding how to fuse BM25 with dense scores

• Measuring whether the system still works after a week of real use

None of those tasks runs faster on an RTX 4090 than on a Mac Mini. Some of them become slower because the expensive machine encourages premature optimization and larger experiment matrices that postpone the harder questions.

What the Measurements Actually Showed

On the 1,200-document collection used throughout this series:

• Mac Mini M2 (16 GB): 4 min 40 s to embed

• Intel NUC 11 (CPU only): 11 min 20 s

• Used RTX 3060: 58 s

The GPU wins by a clear margin. It does not win by enough to change the overall shape of the project. Indexing is an occasional cost. The daily costs are thinking, judging, reading result lists, and adjusting the system. Those costs are measured in hours and days, not in the fifty minutes the NUC needs to finish a full re-embed.

For collections of a few thousand documents, the absolute time difference between a mid-range CPU and a used GPU is smaller than the time most people spend choosing the GPU, installing drivers, and troubleshooting CUDA versions.

The Hidden Costs of Starting Big

Opportunity cost of attention
Every hour spent researching which card to buy, waiting for delivery, and fighting driver issues is an hour not spent on the evaluation set or the hybrid fusion logic. Early in a project those hours are more valuable when applied to the retrieval problem itself.

Premature scaling
A large GPU makes it easy to test batch sizes, quantization schemes, and multi-process configurations that do not yet matter. You end up with beautiful throughput curves and still no confident answer to whether the ranked lists are actually useful.

Environment fragility
CUDA, driver, and framework version interactions remain a leading source of “it worked yesterday” failures. CPU and Apple Silicon paths are not immune to problems, but they have fewer moving parts for this class of model.

Financial and energy overhead
A high-end card draws more power, produces more heat and noise, and represents a larger sunk cost if the project later moves to a different model size or a different deployment target. The monthly electricity numbers in the previous post already show the difference; the difference grows with newer, hotter cards.

False confidence
Fast hardware can mask inefficient code and bloated pipelines. When an embedding pass takes ten minutes on a NUC, you notice unnecessary data loading and redundant conversions. When it takes twenty seconds on a 4090, those problems stay invisible longer.

The Decision Rule I Use Now

I start every new retrieval experiment on the slowest machine that can still run the model in float16 without swapping. For EmbeddingGemma 2 that is currently a 16 GB Mac Mini or a 32 GB NUC. I move to a GPU only when one of the following becomes true:

1. Re-indexing is required so often that the CPU time is a genuine daily friction.

2. The collection size has grown large enough that the CPU path exceeds an hour and the delay blocks other work.

3. I need to test GPU-specific features (certain quantization kernels, multi-GPU strategies, etc.) that simply do not exist on the CPU path.

4. The service must support a concurrent load that saturates the CPU while still meeting latency targets.

Until one of those conditions appears, the expensive card stays powered off.

What Starting Small Forces You to Do Well

Working on modest hardware creates useful pressure:

• You keep batch sizes honest and measure memory.

• You notice when a dependency quietly loads three extra models.

• You design the evaluation set early because you cannot hide behind “I’ll run the big benchmark later.”

• You treat cold-start and warm latency as first-class concerns instead of afterthoughts.

• You build the hybrid retriever and the reproducible pipeline while the embedding step is still slow enough to feel the cost of every change.

In other words, constrained hardware makes the engineering problems visible sooner. That is a feature.

When a Strong GPU Is the Right First Purchase

There are clear cases:

• You already know the collection will be tens or hundreds of thousands of documents and re-indexing will be frequent.

• The primary workload is bulk embedding rather than interactive retrieval.

• You are developing kernels, quantization methods, or training loops rather than application-level retrieval systems.

• Multiple people will share the machine and the aggregate throughput requirement is high from day one.

If your situation matches one of those, buy the card. If your situation is “I want private search over my own documents and notes,” the card is optional and usually premature.

A Practical Sequence I Recommend

1. Get the model running on the machine you already own.

2. Build the smallest evaluation set that reflects your real data.

3. Measure ranking quality and the obvious failure modes.

4. Add BM25 and hybrid fusion.

5. Put the stack in Docker so the environment is repeatable.

6. Only then, if indexing time is still a bottleneck, move the embedding step to a GPU.

Each step produces information that improves the decision about whether the GPU is necessary. Starting with the GPU inverts the order and postpones the information.

Closing

The most expensive GPU is rarely the highest-leverage place to begin a local retrieval project built around EmbeddingGemma 2. The leverage is in the evaluation set, the hybrid logic, the failure analysis, and the reproducible pipeline. Modest hardware is sufficient to do all of that work, and the constraints it imposes are often beneficial.

Buy the fast card when a measured bottleneck says you need it. Until that moment, the slower machine will teach you more and cost you less.

With this post the Small-System Reality arc is complete. The remaining articles move into Search in the Wild: concrete applications that keep private data private, stay maintainable on ordinary hardware, and solve problems people actually have.

Until then, start with what you already own. The numbers you collect there will tell you whether anything more is required.

The benchmark is the conversation starter, not the verdict.