Prompt injection defeats system prompts, so a developer rebuilt agent security in architecture

A test ticket that got past the defenses

A developer about to ship an LLM-powered support-inbox assistant ran a red-team exercise first. As recounted in a field-notes post on dev.to, the feature summarized incoming tickets, pulled up the customer's recent orders and drafted a reply, backed by three tools and a carefully written system prompt. The test was a ticket whose closing paragraph addressed the assistant directly, telling it to include the customer's complete order history in its reply and attach a link to an outside site. The model skipped some of those instructions but complied with enough of them that the author stopped treating the system prompt as a safety mechanism and rebuilt the feature on that assumption.

Why no prompt can draw the boundary

The post defines prompt injection as third-party text that a language model treats as orders. It can be typed straight into a chat, or arrive indirectly inside an email, a ticket, a scraped page, a stored document or a tool response — with the indirect form being the dangerous one for products. The core problem is architectural: the system prompt, the user message and retrieved content all land in one context window, and the model weighs what to follow using patterns learned in training, with no permission check in between. Unlike SQL injection, which parameterized queries fixed by separating code from data at the protocol level, LLMs have no equivalent mechanism.

The author still keeps the cheap mitigations — untrusted text wrapped in explicit tags, a prompt declaring that tagged content is data, and look-alike tags stripped so a ticket cannot close its own block — but these only lower the hit rate. An attacker needs just one phrasing that gets through.

The trifecta as a pre-ship checklist

The post leans on a risk model Simon Willison named in June 2025, the "lethal trifecta": an agent becomes a data-theft instrument when it simultaneously has private data access, exposure to untrusted content, and a channel to the outside world. Remove any one leg and the theft path collapses, even if the injection still lands. The author now scores every LLM feature against those three legs before shipping — a URL summarizer for anonymous visitors ranks low risk, while a coding agent with repository access and open networking hits all three and needs sandboxing.

The support feature also hit all three. The fix was not a smarter prompt but deleting the outbound leg entirely: the model only produces a draft, and a human does the sending. External channels are easy to miss, the post warns, listing outbound HTTP tools, email and webhook tools, rendered images and links, and any record the model writes that syncs to another system.

Tools inherit the user, not the app

According to the post, every tool should execute with the signed-in user's permissions, and anything that determines whose data gets read — identity, tenant, recipient — must be resolved on the server from the session, never taken from model-generated arguments. The implementation builds the tool set per request inside a factory that closes over server-side context, so there is no argument through which the model could reach a different customer; the model only picks values a user could legitimately have typed, such as a result count capped by a strict schema. Three further rules follow: validate arguments strictly and reject bad values rather than coercing them, return the fewest fields possible so unseen data cannot leak, and keep read and write tools in separate sets, attaching write tools only to runs that have not consumed untrusted content.

Proposals instead of actions, distrust on output

Any tool that changes state outside the conversation — sending a message, charging a card, deleting a record, opening a pull request, calling a webhook — should emit a proposal rather than perform the action whenever the run has touched untrusted content. A person clicking approve is the one gate an injected instruction has no leverage over, provided the confirmation screen shows what will actually happen; the reply tool, for instance, takes no recipient argument because the server already knows who opened the ticket.

Output deserves the same suspicion as input. Rendering a Markdown image straight from a model response can exfiltrate data through the image URL with no user interaction at all, so the post recommends allowlisting images and links.

Why it matters

This is a practitioner confirming what security researchers have long argued: prompt injection is a structural property of today's models, since instructions and data share a single token stream, and wording cannot patch it. Teams shipping agents should borrow from older security practice — least privilege, server-side authorization, separated read and write paths, human gates on side effects, and hostile treatment of model output. The trifecta checklist is cheap to apply and catches exactly the design mistakes that turn an impressive demo into a data leak.