Open-source gpuwaste tool shows default Kubernetes settings hide wasted GPU spend

A developer has released gpuwaste, an open-source tool that measures how many GPU-hours a Kubernetes cluster pays for without actually using. According to a post on dev.to, running the tool against real production clusters produced a consistent result: not one of them could answer the question out of the box, and every failure traced back to configuration defaults that nobody had changed.

The standard utilization metric measures busy, not useful

Nearly every GPU dashboard is built on DCGM_FI_DEV_GPU_UTIL. As the author explains, this counter reports the fraction of time at least one kernel was resident on the device — it says nothing about whether that kernel did meaningful work. A process that keeps the GPU occupied with a trivial loop still registers as fully utilized while computing nothing.

The metrics that reflect real computation, per the post, are DCGM_FI_PROF_SM_ACTIVE, which tracks streaming multiprocessor occupancy, and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, which tracks tensor core activity. A wide gap between them — heavy engine activity alongside near-zero tensor activity — suggests the GPU is busy with something other than machine learning. The author calls this pattern "ghost work" and reports that it was invisible on every dashboard examined, because those dashboards were plotting the wrong number.

GPU metrics get attributed to the wrong pod

A subtler problem affects attribution. dcgm-exporter only attaches workload pod labels when Kubernetes pod mapping is enabled via DCGM_EXPORTER_KUBERNETES=true. Without it, Prometheus tags GPU series with the scrape target's own identity, so usage lands on a name like nvidia-dcgm-exporter-xxxxx in the monitoring namespace.

The data therefore looks complete: every series carries a namespace and a pod, and grouping by pod yields a tidy chart. But as the post puts it, every GPU-hour ends up attributed to the exporter that measured it rather than the workload that consumed it. The cheap tell is comparing the pod names in GPU metrics against those in kube_pod_container_resource_requests — if the two sets don't intersect, the attribution is fiction.

The most valuable counters are off by default

The DCGM_FI_PROF_* metrics that separate real work from ghost work require profiling to be enabled in the exporter's counter configuration. On the clusters examined, it wasn't. The post emphasizes that this failure is silent: the metric simply returns nothing, with no error, so anything built on top quietly reports less than the truth.

A confident wrong number nearly shipped

The most instructive failure came from the tool's own fallback path. When profiling counters are missing, utilization can be coarsely inferred from power draw — an idle A100 sits around 55W against a 400W TDP, while a busy one nears the ceiling. The author built this fallback with a generic 50–350W envelope for unknown GPU models, then ran it against a card whose actual range was 15–130W.

The tool reported a GPU at 5% utilization with 88% of its framebuffer resident — a model loaded and serving nothing — and attached a confident $3,327 per month of waste. The author initially believed it and presented it as the first real evidence the approach worked. The card was healthy; the generic envelope had compressed its entire operating range into what looked like idle. Switching the fallback to GR_ENGINE_ACTIVE, a direct measurement rather than an inference from watts, made the finding disappear.

The stated lesson is that a cost tool producing confident numbers from degraded inputs is worse than no tool at all — the output was headed for a budget meeting as a screenshot. Later versions of gpuwaste assume incomplete data and say so: they refuse to project monthly cost from observation windows shorter than a day, name every metric they could not find, and label utilization as estimated when it is.

Checking your own cluster

The post offers a five-minute, read-only check against Prometheus:

count({name=~"DCGM_FI_PROF_.*"}) count by (pod) (DCGM_FI_DEV_FB_USED) count by (pod) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"})

An empty result on the first query means profiling counters are disabled. If the pod names in the second and third queries don't overlap, GPU metrics aren't attributed to the workloads consuming them. Both fixes are one-line configuration changes, and neither is on by default.

The tool itself

gpuwaste is deliberately unglamorous: it ingests exported metrics, joins utilization against allocation, and reports paid-but-unused GPU-hours with a dollar figure. It works strictly offline from a CSV export, because production metric endpoints usually sit behind network policies and mTLS that block direct access. It also ships a synthetic data generator so the output can be evaluated in thirty seconds without touching a live cluster.

Why it matters

GPU capacity is among the most expensive line items in modern infrastructure, and decisions about it are increasingly made from dashboard telemetry. If the default pipeline overstates utilization, attributes spend to monitoring pods instead of workloads, and silently drops the only counters that distinguish real training from ghost work, then FinOps conversations are being run on fiction — in both directions.

The post's practical value is a two-line audit that any cluster operator can run today, plus a design principle that generalizes beyond GPUs: tooling that discloses its own blind spots is more defensible than tooling that hides them behind confident numbers.