Baseten maps the efficient frontier of LLM inference: tradeoffs versus frontier-pushing wins

An economics lens applied to model serving

Baseten has published an engineering guide, picked up on Hacker News's front page, that borrows the "efficient frontier" from economics to explain how LLM inference should be tuned. In portfolio theory the term describes the set of investments delivering the best return at each level of risk; in AI, Baseten uses it for the balance between what a model costs and what it can do. A model earns the "frontier" label when it offers more intelligence per dollar or per parameter than anything cheaper beneath it.

The article's central claim is that inference engineers hold two fundamentally different kinds of levers, and confusing them leads to poor deployment decisions.

Levers that trade one thing for another

The first category moves a deployment along the frontier rather than improving it. Baseten highlights three:

Batch sizing sets how many requests run concurrently. Because token-level continuous batching removes the wait for a batch to form, the configured size mostly determines per-user latency versus total tokens produced per GPU. Small batches feel fast but waste capacity and keep cost per token high; large batches drive cost per token down while individual requests slow.

Parallelism strategy decides how a model spanning hundreds of billions of parameters is split across GPUs. Tensor parallelism carries expensive all-to-all communication, but over high-bandwidth NVLink it is fast enough to be the go-to for latency-sensitive serving. Expert parallelism can help both metrics, with narrow configurations favoring latency and configurations spanning a full rack of GPUs favoring throughput. Attention data parallelism replicates attention layers for parallel computation, buying system throughput at the price of per-request speed.

Quantization lowers the precision of weights, activations or KV cache values. Baseten notes it actually pushes the serving frontier outward, but it opens a second frontier between efficiency and model quality, one that behaves erratically, with large serving gains available at little or no quality cost, particularly through microscaling number formats such as MXFP4 and NVFP4.

Levers that push the frontier outward

The second category creates genuine, allocatable efficiency, and Baseten argues these gains compound: doubling performance through better hardware while doubling it again through better software yields roughly four times the overall serving capability, which can then be spent on latency, throughput or both.

Kernel and runtime optimization, meaning faster low-level functions like matrix multiplications and a cheaper end-to-end forward pass, reduces the resources needed to generate each token, and those savings stack up through the serving stack.

Speculative decoding, which drafts likely tokens and then verifies them against the model, used to be a pure latency-versus-throughput tradeoff, feasible only at small batch sizes when acceptance rates were low. According to Baseten, modern methods such as EAGLE-3, DSpark and DFlash still contend with the main decode loop for resources, but on predictable output like code generation they now win on both axes, skipping forward passes for accepted tokens.

Prefill/decode disaggregation assigns the two inference phases to dedicated workers, letting each be optimized for its own characteristics and letting the ratio between prefill and decode workers track real traffic, sequence lengths and cache hit rates. In practice, Baseten says, it mostly raises throughput while keeping latency flat or slightly better.

The frontier is jagged

The most practical warning in the piece is that the frontier is not a smooth curve. Small configuration changes can produce large, counterintuitive jumps, and Baseten says the only reliable way to locate those cutoff points is empirical sweeping. Hitting a production target is usually less about discovering a novel technique than selecting the right configuration for the nature of your traffic.

The article frames its examples around serving an agentic-coding model with KV cache reuse enabled and KV-aware routing, and points readers to the author's freely available book, Inference Engineering, for a deeper treatment of each technique.

Why it matters

For teams choosing between models and deployment configurations, the distinction between moving along the frontier and pushing it out is the difference between a real efficiency gain and a hidden sacrifice. A latency win achieved by shrinking batches raises your cost per token; a win from better kernels lowers it. Knowing which benchmark numbers reflect trades and which reflect genuine progress is what makes cost-versus-capability decisions rational rather than guesswork, and it explains why the same model can be served at wildly different price points depending on the engineering behind it.