Two weeks of real books: field report logs 17 silent failures in an autonomous AI agent

A field report on dev.to walks through two weeks of running an autonomous AI agent unsupervised on a small construction company's books, with access to real files and a real bank account. According to the author, every one of the seventeen failures catalogued in the post shares the same shape: the agent reported success, and the success was not there. None of them were crashes. Each failure was silent enough to pass as normal operation.

Files that corrupt without an error

The first group of failures involves data that arrives damaged while every check stays green. Having the model reproduce a binary file as a base64 string proved reliable only below roughly 6,000 characters; the report records corrupted output at around 11,564, 22,108 and 37,000 characters. A corrupted base64 string still decodes, so the result is a valid-looking file with wrong bytes inside — in one case a spreadsheet that opened to an empty grid, in another an invoice image whose content below the header was unreadable. The author's rule is blunt: never have the model emit long base64 at all; decode from a saved file programmatically.

A second failure: the agent's "verified" meant only that the upload call succeeded and the file size matched. Same-size, wrong-content is described as the most common corruption mode, which makes file size close to worthless as a signal. The fix was to define verification per file type — diff text against the source, compare spreadsheet totals by exporting to CSV, render images and check that specific fields are legible — plus one instruction that changed the behavior: never write "verified" without stating what was compared.

Batch generation added a third: two payment vouchers built from the same template and uploaded one after the other swapped contents three times in a row, apparently because they shared a long identical byte prefix. Making similar documents differ early, then verifying each by a string unique to it, solved the problem. Even plain text was not exempt — a Hebrew document gained a word absent from the source, and another word came out with mixed Hebrew and Latin characters, both without any error. The suggested defence is a byte-for-byte comparison or a hash after upload.

Confident reasoning on unread data

The report calls this next one the most dangerous failure in the list. An email listed three amounts: 216, 864 and 2,610. The agent summed them, reported 3,690 outstanding and recommended approving payment. The actual figure was 64,118.41, sitting in a PDF attachment that the agent's own log said it had not read, one line earlier. The output was fluent, confident and off by roughly a factor of seventeen.

The recommended hard rule: never derive a number from a source the agent did not read. When part of the information sits in an unread attachment, the agent should report what is known, what is not known, and why the missing part could not be read — never a number.

Several related traps follow. The Gmail tools available return attachment metadata only, so an unaware agent summarizes emails as though it had read the attachments; the fix is to state the limitation explicitly and ask the user to move files to cloud storage. A conclusion that a request had gone unanswered for six days missed a reply that arrived about an hour after it was sent, from a QuickBooks notification address in a separate thread; the fix is three searches — company name, keyword, date range — before declaring silence, and otherwise phrasing it as "no reply found in the thread itself." Searching Gmail with a label ID instead of the label's display name returns an empty list every time, so the agent's tracking mechanism never worked while reporting "nothing new" on every run. And the agent carried "tomorrow" across a day boundary, describing a meeting as tomorrow when it was ninety minutes away; the fix is a real clock check at the start of every response and scheduled run, plus one enforced business timezone.

Structure that breaks itself

The final group is architectural. The agent's instructions referenced configuration files by ID, but in this stack updating a file produces a new ID — so a rule requiring the agent to update its own file map guaranteed that every pointer to it would break, and all three scheduled tasks failed within hours. Referencing files by stable name, and halting when a name matches more than one file, is the proposed fix. Separately, opening a raw CSV through the Google Sheets interface does not edit it; it silently creates a second file with the same name and different contents.

Why it matters

Most agent writing is a clean demo. This is a failure log from a deployment with real money attached, and its value is the pattern it names: agents fail fluently. Silent corruption, plausible numbers from unread sources, empty results that read as answers and self-breaking configuration are all cheap to guard against once they are named — explicit per-file-type verification, unique content markers in batches, mandatory clock checks, name-based references and hard rules against inferring from unread material. Anyone pointing an agent at files, email or payments should expect to hit several of these.