Mistral Small returned empty replies on 19% of tool-calling benchmark tasks

What the benchmark measured

A 300-task function-calling evaluation, published on dev.to by Grain64 through its Clearbar tool and run via OpenRouter, tested nine models against public tool-calling prompts. Mistral Small (build 2603) came back with a completely empty response on 58 of the 300 tasks — 19.3% of the set — even though each task was allowed up to three identical attempts at temperature 0. Priced at $0.15 per million input tokens, the model still ended up at $0.000098 per successful task once failed attempts were billed, versus $0.000051 for GPT-6 Luna, which the same run shows passing 93.0% of tasks with zero retries.

How the harness decides what fails

The write-up draws a single line between two kinds of failure. A reply that cannot be parsed at all is retried; a reply that parses but is wrong is final. The reasoning, according to the author, is that production code can detect unreadable output but can never know an answer is wrong, so the harness does not pretend to either. With the retry cap set to 2 and temperature pinned at 0, a task means at most three identical requests, each one billed and logged. Cost per successful task divides total spend by the tasks that passed, not by the attempts.

Where the retries went

Mistral Small needed 434 attempts to cover the 300 tasks, which means 134 retries. Of those, 116 went to the 58 tasks that never produced a readable reply, and only 18 went to tasks that recovered. By the stored counts, at least three in four tasks that needed a retry stayed unreadable (76.3%). Retries accounted for 23.9% of the model's total spend; the author calculates that stripping them out leaves roughly $0.0000742 per successful task, still above Luna's figure, so retries widened the cost gap without creating it.

The model's remaining 30 misses were ordinary: 14 replies carrying the right value phrased differently, 10 with a different value, 5 with a number written in another form and 1 with a missing argument. Counting the empties, Mistral Small passed 212 of 300 tasks, about 70.7%.

One model's behaviour, not a class effect

The other eight models tell a different story. Six logged no retries at all, DeepSeek V4.1 Flash logged two and Qwen 3.8 Flash logged five, and none of them finished a task on an unreadable reply. That pattern, the author argues, points to how this particular model behaves on this prompt rather than to cheap models struggling with tool calling in general.

What the empties look like

Several failing rows fall into the easiest category in the dataset: short conversations where exactly one tool obviously matches. In row tc-2, the available tools are generate_random_number(min, max) and calculate_loan_payment(loan_amount, interest_rate, loan_term); the user asks for a random number between 1 and 100, and the expected output is a generate_random_number call with min 1 and max 100. Mistral Small returned nothing, three attempts in a row. Rows tc-10, tc-19, tc-21 and tc-37 failed the same way.

What the data cannot explain

The stored results include the reply text but not the finish reason or completion-token counts, so the write-up cannot say why the replies come back empty — whether it stems from content filtering, a stop-token quirk or something on the API side is unknown. The author is asking anyone who reruns a failing row to report what finish_reason and usage.completion_tokens an empty reply shows on OpenRouter. Everything needed to reproduce the run — the dataset, the prompt, the script and the prices — sits on the public verdict page, and the full nine-model pass cost $0.503184, about $0.0017 per row. One caveat: temperature-0 results are not perfectly deterministic, so a second run counts as its own data point.

Why it matters

Anyone wiring an LLM into tools has to treat an empty, unusable response as a first-class failure mode. It is the one error production code can detect and retry automatically, and this benchmark shows the retry may not rescue you, since more than three quarters of retried tasks here never came back readable. It also shows why headline token prices mislead: Mistral Small is cheap per token, but billed failed attempts pushed its cost per useful answer to nearly double a rival that costs less in practice. And because only one model in nine displayed the behaviour, a small eval on your own tasks — a single failing row costs well under a cent to rerun — is the kind of check that catches this before it reaches production.