deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Qdrant 1.20 adds BM25 keyword ranking directly to its full-text index

Qdrant 1.20 scores keyword matches with BM25 on the full-text index itself, removing the sparse-vector pipeline, and adds a code tokenizer, per-element indexes and two compressed vector types.

Qdrant 1.20 adds BM25 keyword ranking directly to its full-text index

BM25 scoring moves onto the full-text index

Qdrant, the open-source vector database, has shipped version 1.20, and the headline change is that keyword matches are now scored with BM25 on the full-text index itself, using the text already stored in payload fields. According to dev.to, which summarized the release post, the version went out on 22 October 2026.

Until now, BM25 in Qdrant meant a sidecar pipeline: documents were pushed through an embedding model to produce sparse vectors, those were stored beside the dense ones, and the corpus had to be re-encoded whenever it changed. The scoring statistics, average document length among them, were frozen into the stored vectors, so they drifted out of date as the collection moved.

Version 1.20 derives those statistics from the collection itself, which keeps scoring correct from the first query onward and as documents arrive or change. The sparse-vector path is still supported, and the migration described in the release keeps data where it is: create a full-text index with the bm25 scoring type, configure the same text processing the sparse model used, then swap the sparse queries for text queries.

The query surface sits inside the ordinary Query API, so a text query composes with dense-vector search in hybrid setups and can rescore points another query retrieved. The BM25 k and b parameters are set per query rather than baked into the index, so adjusting length normalization triggers no rebuild, and one index can serve several query profiles.

A tokenizer for source code

The default word tokenizer breaks text at spaces and punctuation, which suits prose and fails on identifiers: getUserName becomes one token, so a search for user name finds nothing. The new code tokenizer splits identifiers at underscores and case boundaries, acronyms included, while also indexing the whole identifier. Per the release, getUserName, get_user_name and user name now reach one another, and a query for hashmap finds HashMap. Paired with BM25 on a full-text index, ranked code search becomes possible directly from payload text, an option for a self-hosted documentation or repository search box.

Per-element indexes for nested filters

Nested filters are the awkward case in document stores: a payload holds an array of objects, and the filter asks whether one single element satisfies several conditions at once, such as one offer carrying both a particular shop and price. Indexes on the individual fields could only shorten the candidate list; Qdrant still had to load each candidate payload, often from disk, to confirm the pairing.

Version 1.20 adds a nested parameter to payload index types, producing a per-element index that stores each array element separately. When every condition in a nested filter targets a field with a per-element index on the same array, the filter is answered from the indexes alone. Qdrant's own measurements show 7 to 38 times faster execution on three nested filters over 100,000 points, with payloads already in memory, and the margin should grow when they are not.

Turbo16 and Turbo8

Two new datatypes compress vector storage. turbo16 applies the TurboQuant rotation first, spreading values across coordinates, then stores each vector on a 16-bit integer grid: 2 bytes per dimension, reconstruction error 7.5 times lower than float16's across 13 datasets by Qdrant's measurement, and recall@10 within 0.001 of float32 across the 12 search-benchmark datasets. turbo8 stores each dimension in a single byte; across an 11-dataset benchmark, searches completed 1.1 to 2.4 times faster than with float16, recall came close to float16 at half the size, and the gain over turbo4 was largest at low dimensions. The 8-bit encoding also exists as quantization through bits8, giving roughly fourfold compression against eightfold for bits4, which stays the default.

Upgrade notes

Two changes alter cluster behavior after the upgrade, according to dev.to: internal authentication secures the peer gRPC port by default, and inline storage takes over the memory tier of the HNSW index. Clusters configured with only a read-only key face a decision during the upgrade, making the upgrade plan the first document to read. The release also includes snapshots on GCS and Azure Blob Storage, audit logs written to stdout or stderr, collection creation times, safer replica removal, local-time log timestamps and substring matching.

Why it matters

Keyword ranking has usually lived beside vector search rather than inside it, with a sparse-vector pipeline to build, synchronize and re-encode. Moving BM25 onto the full-text index Qdrant already maintains removes that overhead and turns hybrid search into a native query rather than an integration project. Statistics that track the collection instead of sitting frozen inside vectors mean ranking quality holds up on corpora that change, and query-time k and b tuning strips another rebuild out of operations. Add per-element indexes for the nested-filter cases that previously forced disk reads, plus compressed vector types that hold recall near float32 at a fraction of the footprint, and Qdrant is leaning further toward being a general retrieval engine — one that remains open source and self-hostable.

  • #qdrant
  • #vector-database
  • #search
  • #open-source
  • #bm25

Related posts