deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Why AI gateway benchmarks disagree: they measure different latency intervals

A dev.to analysis argues published AI gateway benchmarks disagree by orders of magnitude because they measure different latency intervals, and offers a reproducible k6 method for teams comparing gateways.

Why AI gateway benchmarks disagree: they measure different latency intervals

What the analysis argues

Published benchmarks of AI gateways routinely crown different winners, and headline figures claim leads as large as a factor of forty. According to a dev.to analysis published in November 2026, the disagreement has a structural cause: the runs are not measuring the same interval. Some figures count only time inside the gateway process, leaving out JSON marshalling and the HTTP call. Others report latency added at the client with logging and spend tracking switched off. Others still enable production features, aim at a slower mock provider, or rest on a single pass of a few thousand requests. The author, who checked claims against project repositories and documentation as of November 2026, observes that each vendor's published run tends to put that vendor first — and that this needs no dishonesty, only an undeclared choice about what to measure.

The published figures, with their context

The article gathers the public numbers it could find and labels who produced each:

  • Maxim, Bifrost's maker, reports roughly 11 microseconds of added time per request at 5,000 RPS, measured with internal timing that excludes JSON marshalling and the upstream call; the same run recorded peak memory of about 3.3 GB.
  • LiteLLM's July 2026 run found p99 added latency near 0.7 ms for its Rust gateway, against 2.3 ms for Portkey, 4.5 ms for Bifrost and 257.7 ms for LiteLLM's own Python gateway — single runs, a local mock upstream, logging and spend tracking disabled, and an author-recommended order-of-magnitude reading.
  • A November 2026 LiteLLM run with production features enabled reported median added latency at 50 RPS of 0.9 ms (Rust), 1.2 ms (Bifrost), 2.1 ms (Portkey) and 79.3 ms (Python); Portkey ran without the listed features because it lacks them.
  • Maxim also reported a throughput tie between Bifrost and LiteLLM's Rust beta at about 2,720 RPS at concurrency 64, with Python near 180 RPS — though Bifrost ran with SQLite logging on while LiteLLM ran with logging off, and a second run reversed the p99 ordering.
  • Kong claimed in July 2025 to be more than 200% faster than Portkey and more than 800% faster than LiteLLM, in a proxy-only comparison with no plugins against older versions of both.
  • An engineer at Ferro Labs, which builds a competing Go gateway, measured Kong at about 15,900 req/s at 1,000 virtual users, Portkey plateauing near 850 req/s, LiteLLM capped around 175 req/s and Bifrost failing above 300 VUs — one 8-vCPU VM, default configurations, and a competing stake the author disclosed.

The analysis adds that Kaeso has no published benchmarks, and that an open benchmark effort called Gatebench is in progress.

Where the time actually goes

Whatever the product name, the per-request work is similar: verify the caller's key, decode the request, resolve model aliases to a concrete provider, apply budgets and rate limits, pick and decrypt the upstream credential, encode and send, relay the response including token streams, and log tokens, cost and metadata. Language choice surfaces mainly in the CPU-bound steps — parsing, translating and relaying streams. Deployments that feel slow are usually slowed elsewhere: auth, limits and logging tend to touch a database or Redis, and a gateway writing synchronously to Postgres on every request can trail a gateway in a slower language that records logs asynchronously.

The article also argues for proportion: a hosted chat completion commonly takes hundreds of milliseconds or several seconds, so the gap between 1 ms and 5 ms of overhead is imperceptible to a user. Overhead matters in three places — agent loops that cross the gateway thirty times with each crossing on the critical path, p99 tails at peak concurrency where CPU-bound runtimes queue, and replica counts, where needing ten replicas instead of one is a compute, memory and on-call cost regardless of perceived latency.

A method teams can rerun

The proposed recipe needs no lab: a mock provider with a configurable delay (starting at 300 ms) emitting SSE chunks from a separate machine; a k6 ramping scenario moving from 50 to 300 to 1,000 virtual users; a baseline against the mock, then each gateway in front, reporting added latency — gateway minus baseline — at p50, p95 and p99 for every concurrency stage; a second pass with production features on, including per-key auth, budgets, rate limits, logging and a fallback route. It also calls for recording CPU cores, peak memory per replica and every network hop before the provider, dividing throughput by total cores for a fairer efficiency figure, and running everything three times with the configuration published alongside, since figures that arrive without a configuration deserve skepticism.

Why it matters

Teams choosing gateway infrastructure are comparing numbers that measure different things, produced largely by parties with an interest in the outcome. The useful reframe, per the dev.to analysis, is to ask how much CPU and memory a gateway needs to keep p99 stable at your expected peak concurrency once the features you plan to run are enabled — and to answer that with a measurement simple enough to repeat and publish.

  • #ai
  • #benchmarks
  • #api-gateways
  • #latency
  • #infrastructure
  • #k6

Related posts