Scaling AI agents to a million is a distributed systems problem, benchmarks suggest

An essay on the InstaCloud blog, widely discussed after reaching Hacker News's front page, argues that the hard part of running a million AI agents is not intelligence at all. It is the problem distributed systems engineers have worked on for decades: scheduling finite resources, surviving failures, and coordinating workers that cannot all see the full picture.

The author's framing is organizational. One agent is a worker; a thousand agents is an organization; an organization made of machines is a distributed system. And like human workers, agents are finite — they draw on limited tokens, context, compute, memory, tools and budget, and something always runs out. The essay's central design recommendation follows from that: make the state durable rather than the agent. Agents should be treated as schedulable, interruptible, restartable units that write down what matters and let themselves or another agent pick up from there. The author notes that their own coding agents at InsForge are routinely killed, restarted and run out of context, and that the failures that genuinely hurt are the ones where the plan lived only inside a context window.

The research already treats agents like processes

The essay leans on an emerging research line. AIOS, an LLM agent operating system from Rutgers, decomposes every agent request into system calls — a model invocation, a memory read, a storage write, a tool use — and routes them through a scheduler using policies such as first-in-first-out and round robin. A context manager snapshots agents mid-task so they can be paused and resumed, and the paper reports up to 2.1x faster execution when serving agents built on existing frameworks.

One level up, an ACL 2026 paper called LLM-as-Scheduler starts from the observation that most queries do not merit a heavy multi-agent workflow, and lets a scheduler pick the workflow per query. According to figures cited in the essay, that yielded 43% fewer tokens and more than 36% lower end-to-end latency, at a cost of at most 1.4 percentage points of accuracy.

More agents can make results worse

Coordination, not scheduling, is where scaling bites. A controlled study from Google Research and MIT — "Towards a science of scaling agent systems," dated January 2026 — covered 180 agent configurations across five architectures and three model families, and concluded that simply adding agents often hits a ceiling and can degrade performance when it does not match the task's properties.

The numbers split sharply by task shape. On tasks demanding sequential reasoning, every multi-agent variant tested performed 39% to 70% worse than the alternatives, which the authors attribute to communication overhead fragmenting the reasoning and starving the available "cognitive budget." On parallelizable financial reasoning, centralized coordination improved performance by 80.9%. A second finding matters just as much for platform designers: agents working in parallel without an orchestrator amplified errors by up to 17.2x, while a centralized orchestrator contained the effect at 4.4x. Coordination is real work, and someone — or something — has to do it.

Communication is not coordination

A benchmark called Silo-Bench, accepted at ACL 2026, tested teams of two to 100 agents on thirty distributed-algorithm tasks where the data was sharded so no single agent could see everything, across 54 configurations and 1,620 experiments. The agents could message peers, broadcast and share files, and they self-organized into coordination topologies. It barely helped. The authors report that agents failed to synthesize distributed state into correct answers, and they name the effect the Communication-Reasoning Gap: capable communicators, weak distributed reasoners.

The gap widens with scale. On the hardest, globally shuffled tasks, success fell from 36.2% with two agents to zero at 50 and 100 agents; even the easiest aggregate tasks dropped from 85% with two agents to roughly 40% at a hundred. Those figures are for DeepSeek-V3.1, averaged across three communication protocols.

Why it matters

The argument lands squarely on platform design. If agents behave like processes, agent platforms need the machinery of operating systems and cluster schedulers — context snapshots, access control, interruption and recovery — rather than an assumption that each agent lives forever. If coordination has a measurable cost, orchestrators stop being optional scaffolding and become the component that keeps error amplification bounded. And if the durable thing is state, then checkpointing and handoff become first-class primitives instead of afterthoughts.

The benchmark evidence adds a sobering constraint for anyone planning to scale agent fleets: more agents can actively make a task worse, and models currently reason most poorly exactly where distributed systems need them most — combining partial views into a correct whole. The essay's implicit bet is that the fastest route to reliable million-agent systems runs through decades of accumulated distributed systems practice, applied to workers that happen to think.