One agent, three frameworks: proxy logs quantify the Strands vs LangGraph vs CrewAI trade-off

Same task, three frameworks, every call logged

A developer writing as sunnydachs has published a data-backed comparison on dev.to: the same agent implemented in Strands, LangGraph and CrewAI, executed 27 times, with every LLM call routed through a local recording proxy so the traces are directly comparable. The headline result is that LangGraph's explicit verify/revise loop cut output variance by 77 percent, at roughly 2.5 times the token consumption and latency.

The experiment

The task, identical in all three builds, was a tech-news digest agent: collect five headlines with a fetch_headlines tool, write a roughly 100-word digest, then check the length with a word_count tool and revise if it lands outside the target band. The tools are local and deterministic — no network access, no model calls inside them — so the measurements isolate framework behavior rather than tool performance. All runs used the same model behind the proxy.

The implementations expose each framework's philosophy. In Strands (78 lines), the developer hands the model two tools and a system prompt; the model decides which tool to call, in what order, and when it is done. LangGraph (115 lines) is graph-driven, with typed state, explicit nodes and edges, and a developer-defined loop condition, while model decisions stay confined to individual nodes. CrewAI (110 lines) is role-based: agents carry a role, goal and backstory, and a Crew object manages the handoffs.

Install weight also diverged. Per the post, Strands pulled in 262MB across 81 packages, LangGraph 71MB across 45, and CrewAI 699MB across 142 — the heaviest footprint being the flip side of its batteries-included ergonomics.

How the calls were captured

What makes the comparison credible is the observability layer. A single-file HTTP proxy forwards requests to any OpenAI-compatible endpoint and writes every request/response pair as JSONL: full messages including system prompts, tool schemas and conversation history, plus raw responses, token usage, latency and status, with API keys stripped before writing. An X-Run-Label header separates traces per run, and a small reassembler normalizes SSE-streamed responses into the same shape as non-streamed ones. Uniform trace shape is what makes exact diffs of messages, tokens and latency possible across frameworks.

What the numbers show

Three scenarios were tested: base (an 80–120 word band), tight (95–105 words, forcing the revision loop to fire) and drift (the word_count tool's argument renamed from text to content).

According to the author, LangGraph in the base scenario made a single LLM call and drafted in one shot, with a word-count spread of 13 words across runs. Under the tight scenario the explicit loop fired: spread dropped to 3 words — the 77 percent improvement — while tokens climbed from 2,007 to 5,262 and latency from 4.7 to 11.8 seconds.

CrewAI produced byte-identical output across all three base runs and all drift runs, at 96 words each time, which the author attributes to temperature 0 combined with the role prompt. The trade-off is rigidity: exactly four LLM calls per run, every run.

Strands behaved adaptively. In one tight run the model wrote a 77-word draft, invoked the word-count check, reasoned that it was short, expanded to 103 words and verified again — five LLM calls. Depth varied run to run (5, 3 and 4 calls), a direct consequence of letting the model decide when to stop.

Base-scenario totals came in at 2,370 tokens and 4.2 seconds for Strands, 2,007 tokens and 4.7 seconds for LangGraph, and 2,532 tokens and 3.8 seconds for CrewAI.

Schema drift was absorbed

Every framework's model handled the renamed tool argument correctly — zero wrong-argument calls and no error recovery triggered. The author is careful about scope, though: a one-argument rename is the gentlest possible schema change. Type changes, removed arguments or altered return shapes would likely break the model-driven side, which relies on the prompt rather than code, while LangGraph would stay unaffected because its tool calls live in code.

Stated limitations

The post flags its own caveats: three runs per cell is a trend check, not a statistical claim, with roughly 30 runs treated as the floor for median estimation; one task and one tool is exactly where model-driven frameworks shine, and LangGraph's graph would pay off in branching, approval or parallel workflows that went untested; a single model was used, so reasoning volume and variance would shift with a different one; and dependency footprints move quickly. The full setup — three virtual environments, the proxy and the run commands — is available in a public GitHub repository, with the 27-run matrix completing in about 270 seconds.

Why it matters

Framework debates are usually settled by anecdote and preference. This post offers a reusable method — one recording proxy in front of everything — that turns the debate into diffable data, and it quantifies a trade-off teams actually face: explicit control flow buys determinism at a measurable token and latency cost, model-driven loops adapt but vary run to run, and role-based abstraction delivers stability at the price of a fixed call structure and a heavy install. For anyone building agent evaluation pipelines, the uniform-trace technique is cheap and framework-agnostic.