pgvector HNSW filtering returned zero rows for 52 of 400 RAG queries

A grounded refusal that hid a retrieval failure

A developer recounts in a dev.to post how their documentation assistant confidently told a customer that nothing in the customer's docs covered API key rotation, when the answer was on page two. The relevant chunk existed in the database and its embedding was fine; Postgres simply never returned it. The culprit was an ordinary tenant filter combined with pgvector's HNSW index, and it produced zero-row results for 52 of 400 evaluation queries with no error, no warning, and nothing visible on dashboards.

The setup and the silent failure

The system served 40 customer workspaces from a 1.2 million-row chunks table with a vector(1024) embedding column, an HNSW index using cosine distance, and a plain btree index on tenant_id. The largest tenant held 410,000 chunks, about 34% of the table, while the smallest had 600.

Retrieval used the query pattern from every pgvector tutorial: filter by tenant, order by embedding distance, take 10 rows. The results went to Claude with instructions to answer only from the provided context. That instruction is what let the bug survive for weeks, because an empty context produced an honest "I don't know," which reads as correct grounded behavior on answer-quality metrics.

Why the filter breaks HNSW retrieval

According to the post, the HNSW scan collects hnsw.ef_search candidates (default 40) from the entire table first, and Postgres applies the WHERE clause afterwards. The expected number of returned rows is roughly ef_search multiplied by the filter's selectivity: a tenant owning 2% of rows yields about 0.8 rows on average, not 10. The author's rule of thumb is that with default settings and LIMIT 10, any filter matching under 25% of the table can come back short.

The uneven spread of chunks made it worse. Common questions such as password resets land in regions of the graph packed with the large tenant's auth documentation, so small tenants competed against near-identical chunks from another workspace that won the neighbor race and were then filtered away.

Measuring the damage

The author built a retrieval-only evaluation: 400 queries, 10 per tenant, each paired with a gold chunk known to exist, plus a warning log whenever the row count fell below the limit. With defaults, 253 queries returned the full 10 rows, 147 returned fewer, and 52 returned none. Every zero-row query came from a tenant holding under 3% of the table. For those 52 empty contexts, Claude refused 49 times, and three times it broke its instructions, answered from general knowledge, and invented a settings page that did not exist.

Raising ef_search is a partial, costly fix

A larger candidate list helps: at ef_search 100, short queries fell to 81; at 400, to 19 with four zero-row results, but p95 latency rose from 18ms to 61ms. Even at the maximum value of 1000, nine queries still came back short. The author also hit a second pitfall: setting ef_search at session level changed nothing because PgBouncer in transaction mode ran the SET on one backend and the query on another. The working approach is SET LOCAL inside the same transaction as the query, every time.

Iterative scans, then exact scans for tiny tenants

pgvector 0.8.0's hnsw.iterative_scan keeps walking the HNSW graph when filtering leaves too few rows, continuing until the LIMIT is satisfied or hnsw.max_scan_tuples (default 20,000) is reached. With relaxed_order and ef_search back at 40, short queries dropped from 147 to 3, zero-row queries dropped to zero, and p95 latency rose only from 18ms to 24ms. Relaxed ordering can return rows slightly out of distance order, so the author re-sorts through a materialized CTE; strict_order preserves exact ordering but is slower. Large-tenant queries saw no slowdown because they fill 10 rows on the first pass and never iterate.

The final three short queries came from the 600-row tenant: at that selectivity the 20,000-tuple scan cap yields roughly 10 matches on average, so the limit is missed about half the time. Tenants at or below 5,000 rows now skip the vector index entirely by disabling enable_indexscan, letting the tenant_id btree drive a bitmap scan followed by an exact sort, verified with EXPLAIN ANALYZE. The exact scan over 600 rows took 4ms with full recall.

Why it matters

This failure mode is invisible by design: no errors, no slow queries, clean dashboards, and an LLM whose grounded refusals look like the system working correctly. Any multi-tenant RAG deployment with skewed tenant sizes and post-filtered approximate vector search is exposed to it. The cheap defenses are logging retrieval counts alongside answer metrics, evaluating retrieval separately from answers, enabling iterative scans on pgvector 0.8 or later, and falling back to exact scans for very small tenants. The PgBouncer detail is its own lesson: session-level tuning can silently fail behind transaction pooling.