NVIDIA MPS on EC2 cut ASR inference cost 75% without changing the model

A walkthrough published on dev.to, written as a collaboration between AWS, NVIDIA and Heidi with input from Jerron Chua of the Generative AI Innovation Center at AWS, describes cutting automatic speech recognition (ASR) inference costs by 75% on Amazon EC2. The saving came not from a new model or a redesigned pipeline, but from running several inference clients concurrently on a single GPU through NVIDIA's Multi-Process Service, or MPS.

Three ways to share a GPU

Running one model instance per GPU is often wasteful at scale, the post argues, and it lays out three sharing options. Time-slicing interleaves work from multiple processes on one device. MIG partitions supported hardware into isolated slices. MPS takes a third route: multiple CUDA clients execute at the same time through a single GPU context.

The detail that makes MPS practical is that it is an alternative implementation of the CUDA API that remains binary-compatible, so existing CUDA-based serving stacks gain concurrent execution without being rewritten around a different programming model. The open question for inference workloads is whether that concurrency lifts utilization before latency degrades past what the service can tolerate.

The setup on EC2

The pipeline runs on EC2 g6e.4xlarge and g7e.4xlarge instances, both equipped with NVIDIA L40S GPUs carrying 48 GB of memory. Three containerized components make up the deployment, which keeps responsibilities separated and makes benchmarking repeatable when different sharing strategies are compared.

Serving is anchored on the NVIDIA Triton Inference Server container nvcr.io/nvidia/tritonserver:26.03-py3. From that base, the team builds a combined image for the model under test, a Parakeet NeMo package, using a Dockerfile.single build file: the model artifact is passed in as a LOCAL_NEMO_FILENAME build argument, and the result is tagged parakeet-mps:latest. Baking the model into the image build keeps the runtime environment reproducible across experiments.

Choosing a concurrency level that holds up

For the Triton-plus-MPS configuration evaluated on the g6e.4xlarge instance, the optimum was defined by a latency envelope rather than peak throughput: the highest concurrency at which mean latency stayed under 650 ms and the 99th percentile under 1,000 ms.

The reasoning is that throughput alone can mislead. A system will happily accept more in-flight requests as concurrency rises, but if tail latency climbs too far the configuration stops being useful for production ASR. The tuning recipe that falls out of the benchmark is simple: raise concurrency until utilization improves, watch both mean and p99 latency, and stop at the last level that still satisfies the service objective.

The caveats

MPS is not presented as a universal answer. The same latency ceilings that make a configuration viable also cap how far concurrency can be pushed; crowd the GPU too much and latency moves outside the acceptable range, erasing the benefit.

The post also flags an operational detail that is easy to overlook once benchmarking wraps up: attached Amazon EBS volumes should be deleted. They can hold model checkpoints and a TensorRT cache, keep billing after the experiment ends, and leave artifacts behind that muddy clean reproduction of later runs.

Why it matters

The 75% cost reduction is interesting less as a number and more as evidence of where the leverage was. The model stayed the same and the pipeline stayed the same; the gains came entirely from how the GPU was shared. That makes the approach attractive whenever a deployed model already behaves correctly and the bottleneck is hardware utilization rather than model quality.

For production AI teams, the broader lesson is to treat GPU sharing as a tunable cost lever, judged against explicit latency targets: pick the highest concurrency level that still behaves like a dependable production service, not the highest request count at any price. That discipline is what turns a benchmark win into a cost saving that survives contact with real traffic.