Timescale's pg_vectorsearch claims faster vector search than pgvector's HNSW and IVFFlat

What happened

Timescale has published pg_vectorsearch, an open-source PostgreSQL extension for approximate nearest neighbor (ANN) search, and the project's repository made it to Hacker News's front page. The extension introduces PRISM (Partitioned Routing Index for Similarity Matching), a new index access method implemented in C.

According to the project's README, PRISM is significantly faster than pgvector's IVFFlat and HNSW indexes for both queries and index construction. On a 100-million-vector dataset, the project claims PRISM beats graph-based HNSW at equivalent recall while running on an instance with a quarter of the memory. These are the project's own benchmarks rather than independent results, so the usual caution applies.

How PRISM works

PRISM is described as an IVF-style index with hierarchical routing. A centroid tree partitions vectors into clusters, and the tree is designed to stay warm in PostgreSQL's buffer cache. Vectors are compressed using RaBitQ, a binary quantization technique published at SIGMOD 2024 that comes with theoretical error bounds, and stored in posting lists laid out for SIMD-accelerated distance computation (AVX2, AVX512 and NEON are supported).

The error bounds are the interesting part. A search first estimates distances on the quantized vectors, then uses the bounds to discard any candidate that provably cannot be among the nearest neighbors, so only a small surviving set is reranked against the full-precision vectors stored in the table. The query flow is: traverse the centroid tree to find the closest clusters, scan posting lists with asynchronous I/O, apply RaBitQ two-stage filtering, rerank survivors in full precision, and return the k nearest neighbors.

The design draws on several research efforts the README credits explicitly: SPFresh (SOSP 2023) for incremental updates, SPANN (NeurIPS 2021) for billion-scale partitioned ANN, and Google's use of ScaNN in AlloyDB for hierarchical clustering. PRISM also supports dynamic updates — inserts, updates and deletes — with oversized posting lists split in place by a prism_rebalance procedure, since lists grow as rows arrive and never split on their own.

pgvector compatibility

The extension offers two paths. New users can store embeddings in its own vec32 (float32) or vec16 (float16) column types, which need no other extension. Existing pgvector users can install pg_vectorsearch alongside pgvector and build a PRISM index directly on existing vector or halfvec columns; the README states the column is neither rewritten nor copied, and existing queries keep working unchanged.

The types are binary-compatible with pgvector's despite having different names, so both extensions can coexist even in the same schema. The same distance operators (<->, <#>, <=>) and matching operator classes for L2, inner product and cosine are supported. Since PRISM is normally faster, the planner is expected to prefer it over an existing HNSW index while both exist — the README suggests verifying with EXPLAIN and then dropping the old index. Most deployments will only need to tune prism.nprobe, which defaults to an auto mode targeting roughly 0.95 recall.

Status and requirements

The project is explicitly alpha and intended for testing, not production. Upgrades between prereleases may not be possible, and the README warns users to plan on dropping and recreating the extension and its indexes. It requires PostgreSQL 18 or newer with development headers, GCC 11+ or Clang 14+, and the Meson build system; CBLAS is optional and speeds up RaBitQ encoding. The code is released under the PostgreSQL License.

Why it matters

Vector search is now core infrastructure for AI applications built on PostgreSQL — retrieval-augmented generation, semantic search and recommendation all depend on fast ANN queries over large embedding sets. pgvector's HNSW and IVFFlat indexes are the de facto default, and Timescale already ships pgvectorscale, a DiskANN-based alternative. PRISM adds a third option whose headline claim — better speed at equivalent recall with a quarter of the memory — targets exactly the pain point as embedding collections grow past tens of millions of rows.

The memory angle matters most: graph indexes like HNSW are known memory hogs, and a partitioned, quantization-heavy design that leans on Postgres's own buffer cache could make large-scale vector search viable on smaller instances. That said, the alpha status, the PostgreSQL 18+ requirement and the fact that the benchmarks come from the vendor itself mean most production users should treat this as a promising preview, not a migration target — though the pattern of folding research techniques like RaBitQ and SPANN into Postgres-native extensions signals where the ecosystem is heading.