LLM judges flip RAG eval verdicts at temperature 0, upending a top-k tuning result

The setup

Scores from RAG evaluation pipelines drift between runs even when nothing in the system changes. According to a dev.to post by hualiphd, prompted by a reader's question about an earlier top-k sweep, that drift has two distinct causes: the LLM judge scoring the same answers differently, and the generator writing different answers. They call for opposite fixes, so the author measured each in isolation.

The system under test was deliberately small: 15 research papers as the corpus, 18 questions, gpt-oss-20b on Groq as the generator, and Gemini 3.5 Flash Lite as the judge, wired together with RAGAS 0.2.15 to compute faithfulness, answer relevancy, context precision and context recall. The judge ran at temperature 0, the usual setting for reproducible grading; the generator ran at its provider default.

Two experiments separated the effects. In the first, one set of answers was generated, saved with its retrieved chunks, and graded three times — the prompts had identical hashes in all passes, so any difference was the judge's. In the second, three fresh answer sets were generated under identical settings and each judged once, capturing judge and generator noise together.

The judge does not repeat itself

Grading the same fixed answers three times moved the 18-question means considerably: context precision ranged from 0.601 to 0.686, faithfulness from 0.611 to 0.653, context recall from 0.648 to 0.694. Only answer relevancy held steady, varying by 0.003.

The post traces each wobble to how RAGAS builds its metrics. Faithfulness asks the judge to split an answer into claims and check each against retrieved text; the judge segmented identical answers into different claim counts — one answer came out as 5, 10 and 11 claims across the three passes — and faithfulness changed on exactly the 11 of 18 questions where the claim count changed. Precision and recall rest on yes/no verdicts, and 5 of 148 chunk verdicts plus 3 of 22 reference-statement verdicts flipped between passes. Because 14 of the 18 reference answers were single sentences, recall on those questions reduced to one yes/no call, so a single flip moved a question from 1 to 0; one question scored 1, 0, 0 across identical passes. Relevancy stayed stable because the judge's generated questions are compared through an embedding model, which smooths out wording differences.

Different metrics, different noise sources

When the author regenerated answers instead, retrieval returned the same 8 chunks for every question every time, but the generated text changed a lot — most answers to content questions shared under a fifth of their wording with the previous generation.

That split the variance cleanly. The standard deviation of the run mean rose from 0.021 to 0.054 for faithfulness, meaning roughly 85% of its run-to-run variance came from the answers, and from 0.002 to 0.028 for relevancy, making that metric almost entirely answer-driven. Context recall and precision were different: RAGAS computes them from the question, the retrieved chunks and the reference answer, never the generated answer, so with deterministic retrieval all of their variance is judge noise. The author admits guessing the opposite split beforehand.

How big a difference must be to mean anything

From these measurements the post derives minimum gaps before a difference between two single runs is worth reporting: about 0.15 for faithfulness, 0.08 for relevancy, 0.09 for recall and 0.10 for precision, each acting roughly like a 95% interval on the difference. On a 9-question subset the bars widen by about 40%. The author cautions the figures rest on only three or four runs per metric and could plausibly be half or double; more runs were constrained by cost, with each full run consuming about 257 judge calls against a 500-per-day free-tier limit.

Applied retroactively to the earlier top-k sweep, the bar is damning: on the nine questions every sweep run scored, all seven runs fall inside the judge-noise band, so the sweep could not rank k values at all, and the reported optimum at k=8 came from single runs and uneven question coverage — one headline recall figure of 0.833 actually covered 9 of 18 questions.

Replicate the right thing

The fixes differ by metric, according to the post: for recall and precision, re-judge saved results, since the answers never enter the score; for faithfulness and relevancy, regenerate answers, since re-judging alone would make faithfulness look more than twice as stable as it really is.

Why it matters

Anyone tuning a RAG system by comparing single runs of LLM-judged metrics is likely reading noise. This post turns that risk into concrete thresholds: if two configurations differ by less than roughly 0.08 to 0.15 depending on the metric, the difference is not evidence of anything. It also shows the standard temperature-0 setting does not buy judge consistency, because the instability lives in claim segmentation and verdict flips rather than word choice. The practical takeaway is to know which half of the pipeline each metric actually reads, save intermediate results so the judge can be replayed alone, and report scores with their question coverage — that is the difference between a reproducible finding and a lucky run.