Audit of 110 open-source AI usage metering tools finds 45+ billing bugs

AI products are shifting from flat subscriptions to usage-based and outcome-based pricing: per-token gateway billing, premium request quotas, support agents charged per resolved ticket. Every figure on those invoices comes from a meter, and according to a field report published on dev.to, the meters themselves are frequently wrong.

The report describes a month-long audit of 110 open-source tools that count tokens, track costs or enforce budgets. The auditors verified more than 45 bugs across five recurring families, and 23 fixes have since merged upstream into projects including langfuse (about 34,000 GitHub stars) and codeburn (about 11,000). An independent auditor reproduced the full 236-check conformance suite and confirmed a cache-accounting path that under-reports usage by close to 99 percent.

Five recurring failure patterns

The most common defect was also the least dramatic: stale pricing tables. In one batch of seven tools, five shipped rate tables that were outdated or missing current-generation models, so every downstream total inherited numbers that had stopped being true months earlier.

A second family concerns cache multipliers. Providers bill cache reads and writes at different ratios, and a common shortcut hardcodes one provider's rates for everyone. One tool applied Anthropic's cache-read discount to OpenAI models and underestimated cache reads by a factor of five — an error that looks plausible on each individual line while the totals are wrong.

Third, retry double-counting in stream aggregation. When a streaming request is retried byte-for-byte, some aggregators count both attempts. In one public corpus of 604 re-emitted events, 46 percent were identical at the byte level, and naive aggregation counted them twice. Tools that deduplicate too aggressively make the opposite mistake and miss genuinely retried work; both directions cost someone money.

Fourth, missing usage fields treated as zero. An absent value silently becomes free — code that reads as defensive but behaves like an unauthorised discount. The auditors argue the fix is semantic rather than syntactic: missing data must stay missing, and rollups should be able to say a figure cannot be proven rather than returning zero.

Fifth, window boundary errors. Quota windows anchored to wall-clock time, combined with test fixtures pinned to absolute dates, can hold up in tests for weeks and then break everywhere at the same moment — a failure mode the audit team says it hit in its own CI.

Independent checks and merged fixes

An external auditor working separately reproduced all 236 conformance checks, then contributed a quantified audit of a commercial provider's cache accounting: four failing code paths, with usage under-reported by 98.9 and 95.1 percent on the affected paths, tied to a specific commit. That report now sits in the audit's evidence tree with named credit. The 23 merged upstream fixes span pricing table updates, per-provider cache multipliers, retry-collapse guards with regression fixtures, and explicit absent-versus-zero semantics. According to the post, none were adversarial discoveries; each began in a thread where the maintainer was already discussing the problem.

No vendor publishes a correction process

The audit also checked 20 commercial vendors for a single thing: a published process for what happens when the meter is wrong — a dispute path, a correction policy, anything at all. None had one. When a vendor operates the meter, assesses its own accuracy and publishes no correction procedure, customers are left relying on the vendor's word alone. The post proposes two minimum standards: a named dispute path and machine-checkable billing disclosures. The auditors' conformance pack, built around a settlement specification they call AMS-1, is open source under the MIT license; it runs locally on exported usage data, separating logical operations from physical attempts, cache reads from cache writes, and absent values from zero, with every verdict traceable to a named rule.

Why it matters

As AI pricing moves to per-token, per-request and per-outcome models, the meter effectively becomes the invoice. This audit shows the failures are mundane engineering problems — stale reference data, copy-pasted constants, ambiguous handling of missing values — recurring across dozens of independent tools. When the vendor also runs the meter and publishes no correction process, customers have no way to verify what they are paying for. A local conformance checker, plus the proposed standards for dispute paths and machine-checkable billing disclosures, points to a simple test: a number on an invoice should survive an independent recount, and if it does not, the customer should be able to prove it.