EmbeddingGemma 2 on Cloud Run: a 1 vCPU int8 service beats an L4 GPU until batches get big

A dev.to article walks through deploying Google's EmbeddingGemma 2 as a text-embedding service on Cloud Run, measuring the smallest instance shape that can serve the model and pricing the same workload on CPU and on an NVIDIA L4 GPU. The verdict is blunt: a basic 1 vCPU instance is the cheapest option for most of what an embedding service does, and the GPU is worth its cost only on large batch jobs.

EmbeddingGemma 2 is compact. With the vision and audio encoders removed and the linear layers stored in int8, the text model weighs 203.4 MiB. The piece is a follow-up to an earlier dev.to write-up that priced three deployment shapes per million documents, and it narrows in on two questions: how little memory the CPU service needs, and what an L4 actually bills for a short burst of work, idle time included.

The minimum CPU spec is 1 vCPU and 2 GiB

Measurements put the floor at 1 vCPU and 2 GiB running int8 weights. Loading the model plus PyTorch consumes about 1.35 GiB before a single document is embedded, so a 1 GiB instance is ruled out at every batch size. Peak memory then depends on document length: a batch of 32 documents of roughly 135 tokens peaks at 1,871 MiB, which fits in 2 GiB, while the same batch of 512-token documents (the model's limit) peaks at 3,551 MiB. A 2 GiB service therefore needs batches of eight or fewer for long documents, or 4 GiB to keep a batch of 32.

On that minimum shape, queries return in a median 133 ms, a million documents cost USD 27.67 to embed, and an idle service bills nothing, since Cloud Run scales it to zero.

An int4 repack is smaller but slower on CPU

The article also builds an int4 repack that stores every linear layer in 4-bit groups, shrinking the file to 146.1 MiB. Retrieval quality barely moves: SciFact nDCG@10 comes out at 0.834 versus 0.836 for int8 and 0.854 for the float32 original, so both repacks keep roughly 98 percent of the original's score.

The repack backfires on CPU, though. Because the server unpacks the int4 weights to float32 once at load time, queries take 1.52x as long (a 201 ms median), a million documents cost USD 38.09, and the service needs 3 GiB rather than 2. The smaller file buys a smaller container image and nothing else, so the int8 repack remains the recommended minimum spec.

When the L4 GPU wins

The GPU's catch is billing granularity. According to the dev.to measurements, each isolated burst was billed about eight minutes of L4 time, USD 0.14, even when the work itself took seconds. That idle charge sets the crossover: the L4 becomes cheaper than the CPU service once a single burst passes roughly 5,300 documents. At volume, a million documents cost USD 1.83 on the L4 against USD 27.67 on CPU.

Keeping an L4 permanently warm costs USD 764 a month, money the CPU service would spend embedding 27.6 million documents.

Throughput, as seen by a client with network and JSON parsing included, peaked at 171.8 documents per second, achieved at 64 documents per request with two concurrent requests. The server's internal timer reported 328.3 documents per second at a batch of 32, falling to 192.5 at a batch of 256, so smaller batches suit this GPU. Peak GPU memory reserved by PyTorch reached 14,956 MiB out of the L4's 22,563 MiB under four concurrent requests of 256 documents, while the model itself holds 516 MiB.

The practical recipe the article lands on: serve queries and small indexing batches from the 1 vCPU int8 service, and for a job of tens of thousands of documents, deploy the L4, run the job, and let it scale back to zero.

Why it matters

Embedding services tend to be bursty: steady low-volume query traffic punctuated by occasional re-indexing runs. This benchmark puts hard numbers on the split many teams would otherwise guess at, showing exactly where a scale-to-zero CPU instance stops being the cheap option and where a rented-by-the-burst GPU takes over.

The memory figures are directly reusable too. Knowing that model-plus-runtime overhead is 1.35 GiB before any work, and that long documents multiply peak memory per batch, turns instance sizing into arithmetic rather than trial and error. And the int4 result is a useful negative finding: smaller weights do not automatically mean cheaper CPU inference when the runtime dequantizes to float32 anyway.