Postmortem: A Working Budget Alert Failed to Stop a $1,900 LLM Bill

The incident

A postmortem published on dev.to describes a failure that should feel familiar to anyone running LLM-backed workloads. An email reporting that 84% of the monthly budget had been used arrived on a Tuesday. The author read it, flagged it for later attention, and left a batch job running overnight anyway. By Thursday morning, the account had burned through $1,900 against a $600 cap.

The uncomfortable detail, according to the post, is that nothing malfunctioned. The alert arrived on time, reached the right person, and was read and understood. The failure was structural: the alerting pipeline could only observe spending, never interrupt it. Much of the bill came from a retry loop with no ceiling that kept re-sending a context window that grew on every pass.

Why the warning changed nothing

The author argues the problem is partly behavioral, citing a 1975 study by Worchel, Lee, and Adewole published in the Journal of Personality and Social Psychology. In that experiment, with 200 participants, identical cookies taken from a nearly empty jar were rated as more valuable and attractive than the same cookies from a full jar — and explicitly telling participants the supply had dropped because others wanted the cookies made the effect stronger, not weaker.

A dashboard banner reading "16% of budget remaining" is, in the author's framing, the same kind of depletion cue with an explanation attached. Knowing the cap is an arbitrary number someone configured changes how you describe the banner. It does not change what you do next, which is usually to run the job anyway.

How soft limit checks fail in code

The typical implementation reads month-to-date spend, logs a warning above 90% of budget, then makes the API call and records usage afterward. The post identifies three specific failure modes in that pattern:

• The ledger lags behind the calls that matter. Usage is recorded only after a response returns, so during a burst the spend you most need to see has not landed yet.
• Concurrency reads stale totals. Twelve workers querying spend within the same second can each see 92%, each pass the check, and each fire — jointly blowing past a limit that none of them individually exceeded.
• Agent loops outrun aggregation. A tool-calling agent hitting a malformed schema can make forty calls in ninety seconds; if usage rollups run on a one-minute cron, the loop finishes before the monitoring graph moves.

The common thread, the author writes, is that the check observes and then steps aside. It amounts to a very expensive log line.

Reserve before you spend

The proposed fix borrows from databases: hold funds against a balance instead of merely checking it. Before the LLM call, open a transaction, select the budget row with FOR UPDATE so concurrent workers serialize rather than reading the same stale total, and reserve an estimate computed as an upper bound from input tokens plus max_tokens. If committed spend plus reservations plus the estimate exceeds the cap, raise a typed exception — a 402 — before any network request is made. On success, commit the actual cost and refund the difference; on failure, release the hold.

Slightly over-reserving is a rounding error, the post argues. Under-reserving is how a $600 cap turns into a $1,900 Thursday.

The author also points to existing tooling: baar-core, an open-source Python library that wraps the reservation pattern as a decorator around an LLM client and raises a 402 before the provider is contacted, and noburn.dev, built on top of it for teams, adding per-user caps and a ledger view.

Three rules from the incident

• Enforce at one boundary. A single wrapper around the client, rather than checks scattered across call sites where the newest one, added in a hurry, has none.
• Reserve, don't observe. If a limit reads a number that another process is responsible for updating, it is a lagging indicator, not a limit.
• Fail typed and loud. A machine-readable 402 lets retry middleware and agent loops distinguish "out of budget" from a transient error, so they stop instead of backing off and spending again.

Why it matters

LLM spend differs from classic cloud costs in one dangerous way: an automated agent can generate it far faster than a human can respond to an email about it. As teams wire LLM calls into background jobs and tool-calling loops, an alert that assumes someone will intervene within minutes is not a control at all. The deeper lesson generalizes beyond LLMs — any budget enforced by after-the-fact observation is advisory, and its characteristic failure, concurrent requests jointly exceeding a cap that none of them individually breached, stays invisible until the invoice arrives. Enforcement has to sit on the request path, before tokens are billed, or it is not enforcement.