Five GitLab Duo agents in a real pipeline: what they fixed and where they stopped

A developer has published a hands-on account of wiring five GitLab Duo agents — plan, review, fix, secure and release — into the CI/CD pipeline of a real Python service, and the results give a grounded picture of what post-code AI automation can and cannot do. According to the write-up on dev.to, the agents drafted a feature plan in roughly two minutes, pushed one merge request through three rounds of review, diagnosed a deployment race condition nobody had planted, and helped cut container scan findings from 166 to 1. They never merged anything. Every merge was a human decision, and the human made one mistake doing it.

The setup

The test project, called ro-cited-answers, is an MIT-licensed demo that answers questions in Romanian about public procedures and refuses to respond unless a source document supports the answer. The author built the service first, then added a GitLab CI/CD pipeline covering the stages the GitLab Transcend hackathon, themed Life After Code, treats as the work that follows the code: verify, package, secure, release, configure, monitor and govern.

The pipeline runs offline tests, pushes a container image to the GitLab and Google Artifact registries, performs SAST, secret detection and container scanning, and stages releases on Cloud Run as a zero-traffic candidate that is checked before promotion, with automatic rollback if production checks fail. A drift job compares live settings against a config file, and a scheduled job re-checks production every six hours. There are no service account keys anywhere: GitLab issues per-job tokens that Google Cloud trusts through Workload Identity Federation.

What each agent did

Each agent is a custom Duo flow with its own service account, defined in the repository, and all of them are bound by an AGENTS.md rules file: they may open merge requests and write notes, but they may not merge or deploy.

The plan agent reacts to newly created issues and produced a full implementation plan for the first one in about two minutes, including a decision to keep trace attributes on an allowlist that excludes question text. That plan became the feature that eventually shipped. The review agent, which asks one question — whether a change could make the system answer something it should refuse — took one merge request through three rounds, flagging two high issues, then three medium ones, before approving, and later caught a genuine bug involving an incorrect assumption about which revision was serving traffic.

The fix agent triggers on pipeline failures. It repaired a deliberately planted NameError with a one-character fix, ran the offline test suites itself, and got the pipeline green again. More strikingly, it later diagnosed a failure the author had not planted: two merges landing eight seconds apart shared a single Cloud Run tag, so one pipeline validated the other's revision and failed six of seven checks. The agent opened an issue describing the race, offered three remedies, wrote that it "needs a human decision", and changed no code.

The secure agent was handed the scanner output and produced a table of 173 findings — 166 container, 7 SAST, zero secrets. Rather than dismissing any of them, it opened a merge request switching the base image to Alpine with a non-root user; after the merge, container findings fell to a single Medium-severity issue in zlib. The release agent writes release summaries as issues. On its first run it was blocked from checking production by Duo's sandbox proxy and reported "not verified" rather than guessing — which the author considers the project's most satisfying result, given that the whole service is built on refusing to claim what it cannot back.

Where it broke

The failures are as instructive as the wins. The race condition was the author's own design error, caught by the pipeline's version check rather than prevented by any agent. A command that read the review verdict and merged in a single step let a NOT READY merge request through; the high finding behind that verdict turned out to be correct, and the agent's suggested remedy failed the author's own test, so a replacement approach was merged only after a READY verdict. The rule the author kept: read the verdict in one step, merge in a separate one.

There was duplicate work, too — the fix agent opened two merge requests for one YAML failure, and the plan agent double-ran on an issue assigned to another agent. Duo's redaction of strings shaped like Romanian personal ID numbers left the reviewer seeing placeholder text where a test used a sample value, and a Duo context exclusion initially blocked the reviewer from reading its own configuration directory. One remedy also remains unproven in production: it depends on the sandbox's git accepting a lease flag on push, something tested only locally.

Why it matters

Most coverage of AI agents stops at code generation. This dev.to account tests the unglamorous half — planning, review, triage, release — on a real pipeline with public numbers, and finds agents genuinely useful at diagnosis, triage and first drafts while drawing a hard line at merges and deployments. The design lesson is the split itself: agents propose, inspect and summarise; humans decide. Tellingly, the one bad merge in the whole experiment happened exactly where that line got blurred.