Text-to-SQL benchmarks ignore access control, and LLM table descriptions can hurt retrieval

Two findings aimed at the same blind spot

Two results surfaced on dev.to by Ashish Sinha converge on the same weak point in natural-language-to-SQL systems: the step that decides which parts of a schema reach the model. One is a paper accepted to SIGMOD 2027 showing that mainstream text-to-SQL benchmark scores were measured without access control. The other is a hands-on experiment showing that the common fix of enriching a schema with LLM-written descriptions can make retrieval measurably worse.

A benchmark that finally asks who is asking

According to the dev.to write-up, the paper "Benchmarking Text-to-SQL under Role-Based Access Control" by Yang Fei, Yangfan Jiang, Yin Yang and Xiaokui Xiao (arXiv, July 2026) takes the three benchmarks most systems are evaluated on — Spider, BIRD and LiveSQLBench — and adds what they all lack: roles and the policies attached to them. The augmented dataset covers 53 databases, 399 tables, 3,353 columns and 21,502 role-annotated query instances, with policies at column-operation granularity, meaning the question is not whether a role can read a table but whether it may SELECT a particular column. Roles were synthesized per database using an LLM-assisted pipeline, including scoped administrator roles.

When existing systems are scored against it, many high performers — open-weight LLMs especially — degrade sharply, which the authors attribute to frequent RBAC violations. Sinha adds caveats worth keeping: the benchmark has not been released yet, the paper promises a public repository for the pipeline, toolkit and datasets, and he has not run it himself.

RBAC-rejected successes

The paper's sharpest contribution is a failure category its metrics are built to expose: queries that return the right rows under ordinary grading while violating the access policy. Under Spider's or BIRD's scoring, those queries count as wins. The degradation is therefore sharp rather than gradual — not because writing SQL under constraints is intrinsically harder, but because a metric that never checked authorization was silently counting violations as successes.

The architectural reason, as Sinha describes it, is placement. Real schemas are too large for a prompt, so every stack first narrows candidate tables with a retrieval step. Database access control — grants, row-level security — acts later, at execution. The ranker is blind to who is asking, so a support agent's question can hand the model a compensation table, the model writes correct SQL against it, and row-level security then filters every row. The user sees an empty result with no way to distinguish "no such data" from "not permitted", and the agent confidently reports the former. Worse, the schema itself is information: a table name placed in a prompt discloses something even when zero rows return, and row filtering cannot retract a name.

The post argues the fix has to live at selection: restricted objects must be absent from the candidate set, not merely ranked low, and a restricted object should be indistinguishable from a nonexistent one so the schema cannot be probed. It also warns that scoping is not authentication — whatever component supplies the user's identity becomes a trust boundary.

When descriptions hurt retrieval

The second finding is a self-reported experiment on a real 1,245-object schema. Generating an LLM description for every table and indexing them alongside the names lowered recall. A table literally named contacts ranked third for the question "show the contacts of xmagnet" before cataloguing and dropped below rank 40 after. The descriptions were accurate — and correlated.

The mechanism is BM25 arithmetic. In a CRM, nearly every table is in some sense about contacts, so after cataloguing the token "contact" appeared in roughly 1,072 of 1,245 documents and its inverse document frequency fell to 0.15, against 4.27 when only names are indexed, where the term appears in 17 documents. Meanwhile BM25's length normalization penalizes long documents, and the central table — contacts has 55 columns — is reliably the longest. IDF collapse flattens the ranking; length normalization then actively favors peripheral tables. The queries that degraded most were the plain-English ones the descriptions existed to serve.

The fix that worked

Down-weighting description text inside a single index helped only slightly, and it cost the strongest case for descriptions, a question mapping "per member per month cost" to a table named v_pmpm, which no lexical path over names could bridge. What worked was separate BM25 indexes per field — identifiers in one, written prose in another — each with its own IDF and length statistics, fused by reciprocal rank fusion over ranks rather than scores. Per-field IDF restored the discriminating power of common terms by construction, and per-field length statistics stopped punishing wide tables. Sinha notes the same mechanism threatens any pipeline that enriches documents with generated summaries before indexing.

Why it matters

Both findings hit the layer between a user's question and the model's prompt. Benchmark numbers quoted from Spider, BIRD or LiveSQLBench were produced with unrestricted read access, so they say little about how a system behaves for real users with real permissions. And the widespread advice to catalogue a schema with an LLM can silently degrade the exact queries it targets while passing smoke tests. Both results carry caveats — one benchmark is unreleased, the other is a single-author, single-schema measurement, and Sinha maintains an open-source selection library with an interest in the outcome — but together they define concrete checks for anyone building AI over databases: evaluate under roles, and measure retrieval before and after enrichment.