Replay bug in AWS Lambda durable functions lets agents execute unapproved actions

A developer writing on dev.to has documented a failure mode in AWS Lambda's durable functions that lets a human-in-the-loop agent carry out an action its approver never sanctioned: a human reviewed a "refund" proposal and clicked approve, and the workflow went on to perform an "escalate" instead. According to the post, nothing was tampered with and the business logic was correct — the culprit was the replay mechanism that makes durable execution work in the first place.

How durable functions resume

Per the write-up, Lambda durable functions — introduced in late 2025 and now available in several regions — let a single ordinary handler express a long-running, multi-step workflow that can pause for as much as a year while waiting on a human or an external event, without paying for idle compute. Two primitives do the work: context.step wraps a unit of work and checkpoints its result, and context.wait_for_callback suspends the entire execution until something outside calls back with an answer.

The catch is how resumption happens. Lambda does not continue the paused Python mid-function; it re-runs the handler from the top, and for every step-wrapped operation it injects the stored checkpoint instead of executing that code again. The author measured this directly: across a single suspend-and-resume cycle, a counter placed outside any step incremented twice, while a counter inside a step incremented once. The contract is therefore literal — code inside a step runs once and is remembered, and code outside a step runs again on every replay.

Where the approval gate breaks

The demo workflow follows the now-common agent pattern: the agent proposes an action, a human approves or rejects it, and the system executes. The failure appears when the agent's proposal is generated outside a step. Agents are non-deterministic — asking the same question twice can yield different answers, which is inherent to what an agent is.

On the first pass the agent proposes a refund, the human sees that proposal and approves it, and the function suspends. When the approval arrives, replay starts the handler over from the top; the un-checkpointed agent call runs a second time and returns a different action, and that second value is what flows into the execute step. The author reproduced this with a mock agent that returns a different action each time it genuinely runs, building an unsafe and a safe variant side by side. In the unsafe variant the human approved "refund" and the system executed "escalate"; in the safe variant the two matched. The post notes that the divergence is asserted by a passing test in the accompanying repository, and that the setup runs locally without an AWS account as well as on real Lambda.

The fix is structural

The remedy is to move the non-deterministic call inside a step — replacing a bare agent_proposes(amount) with context.step(agent_proposes(amount), name="agent"). Once the proposal is checkpointed, replay injects the stored value rather than re-consulting the agent, so the value the human reviewed and the value the executor acts on are guaranteed to be identical. The rule the author draws from the episode: anything non-deterministic or side-effecting — agent calls, API requests, timestamps, random values — belongs inside a step; code outside a step is only safe for pure, deterministic glue.

Other pitfalls flagged

The post also lists smaller surprises that cost the author real time. The handler's input changes type between passes: it arrived as a JSON string on the first run and as an already-decoded dict on replay, so naive access raises a TypeError only after resume, which the author describes as a miserable thing to debug. The suggested guard is to decode only when the input is a string. Callback results, meanwhile, come back as raw bytes.

Why it matters

Human approval gates are the default safety pattern for agentic systems handling refunds, moderation calls and infrastructure changes. This bug is unusually nasty because it defeats that gate silently: the code reads correctly top to bottom, the human does everything right, and the system still acts on a decision nobody made. Durable functions genuinely collapse the Step Functions machinery — separate state machines, task tokens, JSONPath plumbing — into readable straight-line code, but they quietly shift the correctness burden onto the developer to internalise the replay contract. Any team wiring agent output into an approval flow on this model needs to checkpoint the proposal itself, and ideally test for approval-to-execution divergence the way the original author ended up doing.