Memory, Batch Size, and Latency: The Tradeoff Nobody Shows in the Demo

Here is the short version.
When you embed documents with EmbeddingGemma 2, three variables move together: peak memory, batch size, and wall-clock time. Increase the batch size and throughput usually rises—until you run out of memory and the process slows down, swaps, or crashes. Decrease the batch size and memory drops, but the embedding pass takes longer. Latency for a single query is almost unaffected by the batch size you chose at indexing time; it is dominated by model load and the cost of encoding one short string. The demos rarely show the full curve. This post does.

I measured the relationship on the same three platforms used earlier (Mac Mini M2 16 GB, Intel NUC 11 32 GB, and a desktop with an RTX 3060 12 GB). All numbers come from the identical 1,200-document collection and the same model revision. The only intentional changes were batch size and the resulting memory pressure.

Why the Tradeoff Matters

Most model cards and quick-start notebooks pick a convenient batch size (often 16 or 32) and report a single throughput number. That number is accurate for the machine they used and the memory headroom they had. It is less accurate for a 16 GB laptop that is also running a browser, a vector database, and a local editor.

If you treat the demo batch size as a constant, you eventually hit one of two failure modes:

1. Out-of-memory errors or heavy swapping that makes the “fast” configuration slower than a smaller batch.

2. Unnecessarily long indexing runs because you chose a conservative batch size on a machine that could have handled more.

The useful skill is knowing where the knee of the curve sits on the hardware you actually own.

Measurement Protocol

• Model: EmbeddingGemma 2, float16, sentence-transformers

• Document set: 1,200 files, average ~180 words

• Batch sizes tested: 1, 2, 4, 8, 16, 32 (where memory allowed)

• Metrics: wall-clock time for the full pass, peak resident set size (RSS), and, on the GPU machine, peak GPU memory

• Single-query latency measured separately after the model was warm, using batch size 1 for the query itself

Each batch size was run three times; the median is reported. I cleared caches between runs and kept background activity as low as practical.

Results on the Mac Mini M2 (16 GB)

Batch size 8 was the practical maximum. At 16 the process began swapping heavily and the wall-clock time became worse than batch size 8. Peak memory at batch size 8 left enough headroom for the operating system and a light desktop session. Query latency after the model was warm stayed in the 45–75 ms range regardless of the batch size used during indexing.

Results on the Intel NUC 11 (32 GB, CPU only)

With 32 GB available, larger batches continued to help, but the returns diminished sharply after 8. Batch size 16 shaved another 90 seconds off the run; batch size 32 saved only an additional 30 seconds while consuming noticeably more memory. For this collection the sweet spot was 8–16. Query latency remained 90–140 ms across all configurations.

Results on the RTX 3060 12 GB

The GPU absorbed larger batches easily. Throughput improved until batch size 16 and then flattened. Peak GPU memory at batch size 32 stayed well under the 12 GB limit, leaving room for a concurrent vector database or a second small model. Warm query latency was 20–35 ms across the board.

What Stays Constant and What Moves

Single-query latency is almost independent of indexing batch size.
Once the model is resident, encoding one short query costs the same regardless of whether you indexed with batch size 2 or 16. The big latency differences appear only between cold and warm states, or between CPU and accelerator paths.

Peak memory scales sub-linearly with batch size, then suddenly becomes linear (or worse) when the allocator or the framework starts holding onto intermediate buffers.
On the Mac Mini the jump from 8 to 16 was catastrophic. On the 32 GB NUC and the 3060 the growth stayed manageable.

Throughput gains diminish.
Doubling the batch size never doubled the speed. The largest relative gains occurred at the low end (1 → 2 → 4). Beyond the hardware’s comfortable point you pay memory for smaller and smaller time savings.

Practical Decision Rules I Use

1. Start at batch size 8. It is the highest value that has been safe on every 16 GB class machine I have tested with this model.

2. Watch peak memory, not just whether the process finishes. If RSS approaches 80–85 % of physical RAM, drop the batch size even if the run succeeds. Swapping destroys interactive latency for everything else on the machine.

3. On GPU machines, push until GPU memory sits around 60–70 % of capacity. Beyond that you gain little and lose the ability to run other work concurrently.

4. Measure the actual knee on your hardware. The tables above are starting points, not universal constants. A different model revision, longer documents, or additional framework overhead can shift the curve.

5. Keep indexing batch size and query batch size as separate parameters. They solve different problems and should not be coupled.

Failure Modes Worth Avoiding

Silent swapping
The process does not crash; it just becomes five times slower and the rest of the system feels laggy. Monitor RSS or use a tool that surfaces pressure.

Assuming the demo batch size is safe
Many quick-start notebooks use batch size 32 because they were written on 24 GB or 40 GB cards. On a 16 GB laptop that choice is often the difference between a four-minute run and a twenty-minute run that thrashes.

Chasing peak throughput for interactive services
If you re-index only occasionally, a slightly longer indexing pass is usually preferable to a configuration that leaves no memory headroom for the query service and the rest of the machine.

What I Still Have Not Quantified

• The effect of int8 / int4 quantization on the memory–batch-size curve

• Multi-process or multi-GPU embedding

• How approximate indexing (HNSW, IVF, etc.) changes the latency picture once the vectors exist

• Power draw at each batch size

Those measurements will appear in later posts. The relationships above are the baseline I trust for float16 EmbeddingGemma 2 on ordinary hardware.

Closing

Memory, batch size, and latency form a tight tradeoff that demos rarely surface. Larger batches improve throughput until memory pressure reverses the gain. Query latency is largely independent of the indexing batch size you chose. The only reliable way to find the right operating point is to measure peak memory and wall-clock time on the machine you will actually use.

Run the same sweep on your own hardware. Record the highest batch size that leaves comfortable headroom, and treat that as your default. Everything else in the retrieval stack becomes easier once the embedding step is both fast enough and safe enough to leave running without watching it.

The next post turns these hardware observations into dollars and cents: what a small private retrieval service actually costs to keep online for a month. After that we will look at why starting with the most expensive GPU is rarely the right engineering decision for this class of workload.

Until then, measure the curve. The demo numbers are the conversation starter; your own memory and latency figures are the part that matters.

The benchmark is the conversation starter, not the verdict.