Korean eval of 330 LLMs auto-fails a third of answers on script alone

A four-line script check turned out to be the most revealing part of a new Korean-language evaluation of 330 language models: before a judge read a single answer, it disqualified a third of the corpus for not being cleanly written in Korean.

The evaluation comes from the team behind the ginigen-ai leaderboard, described in a write-up published on dev.to, with the methodology hosted on Hugging Face. Models were graded across seven axes — honorifics, terminology, knowledge of Korean institutions, summarisation, naturalness, register and format compliance — but the first filter was purely mechanical.

A four-line gate in front of the judge

The check counts characters. An answer fails if less than 25% of it is Hangul, if it contains more than three Hanja (Chinese characters, which Korean uses only sparingly), or if it contains any Japanese kana at all.

By that measure, 766 of 2,304 answers — 33.2% — were auto-failed. 171 of the 330 models (51.8%) produced at least one contaminated answer, and 54 models (16.4%) failed the script check on all seven tasks: across seven prompts, they never produced a single answer that passed.

The reasoning, per the write-up, is that an LLM judge asked to grade a partly Chinese answer will still return a plausible score about tone or register — a number that looks like data even though the answer was not in the requested language. So the team drew a line: whether an answer is in the right script can be counted, while tone cannot, and an answer only reaches the judge after passing the count. Saving 766 judge calls was a side effect, they note, not the goal.

Failures cluster on the hardest linguistic tasks

The auto-fails were not evenly spread: 142 models failed on honorifics, 125 on terminology, 113 on Korean institutions, 105 on summarisation, 99 on naturalness, 93 on register and 89 on format compliance. The task that demands the most from a model linguistically is also where answers fall out of the script most often.

The judge is calibrated before anything is measured

Before grading, the harness runs a fixed set of control answers with known expected grades and aborts the run unless it scores every one correctly. The judge also never sees model names, to strip out self-preference, and the rubric states that length is irrelevant because judges tend to reward verbosity.

The grades and vendor averages

The grade distribution across eight bands: A+++ (5 models), A++ (2), A+ (18), B+ (55), B (58), C (52), D (29) and F (111). Only 7.6% of models reached any A band; 33.6% scored F.

Per-axis A rates: honorifics 8.5% and knowledge of Korean institutions 9.4%, versus terminology 31.2%, format compliance 47.6%, summary accuracy 49.2%, freedom from translationese 53.1% and register 53.3%. Two axes sit at roughly a tenth of the field while the rest cluster near half.

Vendor mean scores on a 0–3 scale, for vendors with at least eight models measured: anthropic 2.29 (14 models), mistralai 2.07 (18), openai 1.96 (51), qwen 1.48 (51), google 1.48 (27), deepseek 1.33 (15), meta-llama 1.16 (8), nvidia 0.93 (10), minimax 0.70 (8) and z-ai 0.36 (15). The authors are explicit about the limits — one prompt set, one judge, one run — and frame the table as a snapshot of this particular measurement rather than a verdict on vendors.

Recency and size predict nothing

Five models scored a perfect 3.00: openai/gpt-5.4, openai/gpt-5.4-mini, openai/gpt-4o-2024-05-13, openai/gpt-3.5-turbo-16k — a 2023 model — and google/gemini-3.1-flash-image. Among the 28 models that earned an A on honorifics were mid-sized ones such as gpt-4o-mini, qwen-2.5-72b-instruct and llama-3.1-70b-instruct.

The takeaway for anyone picking a model for a Korean-facing product: release date, parameter count and position on English leaderboards carried no signal here.

Why it matters

Multilingual competence is routinely assumed from English benchmarks, and this measurement suggests that assumption fails at scale — over half the field could not keep seven consecutive answers in the correct script. The portable lesson for anyone building evaluations in any language is to write the countable check first and let it reject answers before a judge ever sees them: a judge will grade a wrong-script answer, and that grade will be indistinguishable from legitimate ones in your results. The leaderboard itself is public — 426 models tracked, 330 graded, refreshed daily — with an open API that returns per-answer reason strings, so specific grades can be contested rather than just the averages.