ClickBench removes pgrust entry over hand-written per-query kernels and cached answers

What happened

The maintainers of ClickBench, the open benchmark hosted by the ClickHouse project, have pulled a submission after concluding it was tuned to the test rather than to the workload. A pull request filed by Alexey Milovidov on the ClickHouse/ClickBench repository, titled "Remove pgrust: the entry is cheating (per-query kernels and result caching)," deletes the entry and its results. The filing surfaced on Hacker News's front page on October 23, 2026.

According to the pull request, the pgrust v0.4-preview binary ships a bespoke kernel for each individual benchmark query. The evidence sits in the source tree: files named q00.rs through q36.rs under crates/gravity/backend/irprobe/src, each corresponding to a ClickBench query. Alongside them are plan matchers named after the benchmark's queries — logic that recognises when an incoming query is one of the known test items and routes it to the matching hand-written path.

Cached answers, not computed ones

The second finding concerns a cache of filter verdicts shared across queries. According to the pull request, it allows repeated runs to reproduce the answer of an earlier run without touching the underlying data. In a benchmark that times how quickly a system can answer questions over a large dataset, a mechanism that replays stored verdicts on subsequent executions sidesteps the work being measured entirely.

How it was caught

The simplest evidence is also the most damning: the system fails on near-miss queries. According to the pull request, queries that differ only trivially from the benchmark's — such as SELECT MIN(URL) FROM hits — are rejected with an error. An engine that genuinely evaluates queries would shrug off a small change to an expression; one that pattern-matches against a fixed list of known queries cannot.

Tuning versus hard-coding

Benchmark gaming exists on a sliding scale. Every serious database tunes for the workloads it expects to face, and vendors have long optimised for prominent public benchmarks. The line the pull request draws is between an optimiser that makes a whole class of queries faster and a binary that special-cases exact query shapes. The first generalises; the second collapses the moment a query changes, which is precisely what the perturbation test exposed.

The source material does not claim anything about how the per-query kernels came to exist, only that they do. That distinction matters less than it might seem: whatever produced them, the structural problem is that ClickBench, like most public benchmarks, publishes its exact queries in advance, and any entrant can see precisely what will be timed.

Why it matters

This episode is a concrete demonstration of how fixed, transparent benchmarks can be gamed, and it carries three lessons. First, specialisation is indistinguishable from cheating until someone varies the input, so cheap audit tricks — nudging a query slightly and checking it still runs — are now essential due diligence for anyone reading a results table. Second, repeat-run timing is fragile: a cache that survives between runs can make "hot" numbers meaningless as a measure of query execution. Third, as generated code makes it trivial to produce dozens of bespoke per-query kernels, benchmark maintainers may need held-out or randomised queries to keep results honest. The removal is a small change in one repository, but the pattern it documents — hand-written kernels plus answer caching — is one any consumer of benchmark comparisons should now assume is possible.

The broader risk is erosion of trust. Benchmarks like ClickBench derive their value from being hard to game and actively policed; a caught case is mildly reassuring, an uncaught one is invisible. Readers comparing engines on analytics workloads should weight results from rigid, published query sets accordingly, and favour benchmarks that hold something back.