Meta's 816B open-weight Muse Spark Open trails hosted models in coding agent test

The result

An independent benchmark published on dev.to by mihai_leanzero has given Meta's new open-weight model, Muse Spark Open, its first real-world test as a coding agent, and the outcome is rough. According to the post, the model — released on 15 October with 816 billion parameters, 42 billion of them active — scored 0.091 and 0.0877 out of 1 on the Forge 2.0 agentic benchmark (a Jira Forge app upgrade task), ranking 53rd and 54th on a 64-entry board whose best build scores 0.8523. Its run on a second benchmark, Gauntlet 7.2, never earned a verdict: the provider connection timed out after 13 calls, at a cost of 19 cents.

The runs went through OpenRouter, with all traffic served from a single host, Modal. Muse Spark Open is distinct from the API-only Muse Spark 1.3, and the comparison with Meta's hosted siblings is unflattering. Muse Wonder scored 0.4203 and 0.2366 on Forge 2.0 and 0.7638 on the Gauntlet, good for 9th of 44, while Muse Spark 1.3 posted 0.2968 and 0.2202 on Forge and 0.5826 on the Gauntlet.

Where the run broke

The Gauntlet task asks for a payments console within 150 calls — two services that sync with a vendor and take its webhooks, plus a 3D view — with most of the work in one Python file. The author reports a fast start: the model wrote the transport layer and the notifier service inside its first minute. Then it turned to the large file, beginning it four times, and each attempt went silent for roughly two minutes before failing with an upstream idle timeout. The agent, a fork of the open-source goose, resent the request three times to the same silence, and the target file stayed the 23-line stub every model starts with.

The author's hypothesis, explicitly flagged as unproven, is that the serving route sends nothing while the model composes a long tool call, so a large write looks like an idle connection and trips a timeout near the two-minute mark. The same route's longest completed Forge call took 80 seconds, and a file the size of what the hosted models produced would need closer to four minutes. Whether OpenRouter or Modal raised the timeout is unclear from the logs.

The hosted siblings point the same way. Muse Wonder wrote an 845-line file in a single 45,841-character call, and Muse Spark 1.3 wrote 1,173 lines in one 54,910-character call after a 168-second request. Both ran on Meta's own endpoint, which tolerated long calls that the open-weight route did not.

Early stopping and the local field

Muse Spark Open also quit early on Forge 2.0. The prompt granted 300 calls and described time as virtual, yet the model used only 58 and 44 across its two runs, with its reasoning citing limited calls and limited time.

Other open models fared no better. Gemma 4 in several sizes, Granite 4.2 30B, LFM2.5-24B-A2B, Xing 4.0 and Nemotron 3.5 Bolt ran locally on the team's Macs, along with their own Qwen3.8-27B fine-tunes; only the LFM2.5 run made the board, at 0.0187. On Forge 2.0 the unchanged starter app scores 0.0617, so a model at that number wrote nothing — exactly where Xing 4.0 and Granite 4.2 landed. Nemotron 3.5 Bolt, the one local model to build a full app, never checked its totals: lint passed while the widget displayed a raw fractional figure. The author is candid that some failures were the team's own — a memory guard cut runs short, several runs were cancelled, one fine-tune failed on a missing flag — and recurring modes included files that never arrived, endless reading with no writing, and tool calls returned as prose.

One awkward detail stands out: the plain Qwen3.8-27B base, an 8-bit conversion on a 64 GB MacBook, scored 0.2576 on the same Forge build — nearly triple the 816B model's score through OpenRouter. The author cautions that it ran on a newer test build with thinking enabled, so it is not a clean comparison.

Card numbers versus practice

On paper, open weights had a strong autumn. Meta's card puts Muse Spark Open at 76.4 on Terminal-Bench 2.1, a figure the author traces to Artificial Analysis running it at xhigh effort in the Terminus 2 harness. Artificial Analysis also gives it 39 on its Intelligence Index against a median of 19 for comparable models, and Vals AI lists 43.33% accuracy, with 6.57% on the harder Terminal-Bench 4.0. The dev.to team ran the model at medium effort in their own harness; their board numbers were read on 20 October.

Why it matters

The gap between a strong model card and a working coding agent is the story. A model can post competitive benchmark scores and still fail to deliver a single large file in a live tool-calling environment, with the fault sitting in the serving path rather than the weights. For teams weighing self-hosted open models against paid hosted agents on cost or privacy grounds, these results suggest open weights are not yet a drop-in substitute. And a small Qwen base out-scoring an 816B model is a reminder that agentic benchmarks measure the whole stack — model, harness and infrastructure — not raw capability alone.