Manticore Search adds built-in chunking to fix vector search on long documents

Manticore Search has added automatic document chunking to its vector search feature, moving work that normally lives in an external ingest pipeline — splitting documents, embedding each piece, merging chunk matches back into their parent documents — into the database's table definition. According to the company's blog post, which reached the front page of Hacker News, the change targets a failure mode that is easy to miss: long documents silently losing most of their content to an embedding model's input window.

The silent truncation problem

The post describes a typical setup: a table with automatic embeddings, where inserting text triggers the model and fills the vector column for you. Its example model, Xenova/all-MiniLM-L6-v2, has a 512-token input window. Load a 4,000-word document — around 5,000 tokens — and the insert succeeds and searches appear to work. But the model only read the first 380 words; everything after that can never be retrieved, and no warning appears anywhere. The resulting vector may not represent the document as a whole either. The post illustrates this with a backup-and-restore runbook whose final section, covering rotation of the TLS certificate on the replication port, sits far past the window and would be unretrievable even though the document looks fully indexed.

Chunking in the table definition

The fix is a new chunk_strategy setting on model-backed vector columns:

sql CREATE TABLE docs ( title text, content text, chunks float_vector_array knn_type='hnsw' hnsw_similarity='cosine' model_name='Xenova/all-MiniLM-L6-v2' from='title,content' chunk_strategy='sentence' max_tokens='256' overlap_tokens='32' );

With this in place, Manticore splits each document, embeds every chunk, and searches across all of them. Five strategies are offered: truncate (the old default), mean, fixed, recursive, and sentence. Truncate and mean produce a single vector per document and fit a float_vector column; fixed, recursive, and sentence yield many vectors and require a float_vector_array column. The tuning knobs are max_tokens for chunk size, overlap_tokens for tokens shared between neighbouring chunks, and max_chunks as a per-document ceiling.

Two design decisions stand out. A document stays a single search result: chunks compete individually, but Manticore returns the document once, with knn_dist() reporting the distance to its closest chunk, and the k parameter counts documents rather than chunks. Queries, meanwhile, are never chunked — they are short enough to embed as a whole, and only stored documents are split.

Measured gains and costs

The post benchmarks the feature against Manticore's own manual: 189 pages and roughly 298,000 words. For content sitting beyond the model's input window, recall@5 rose from 55.1 percent to 83.3 percent, and MRR improved from 0.44 to 0.70. The costs are concrete: about 2.5 times the RAM and roughly four times the ingest time, since every chunk receives its own embedding.

Why it matters

Chunking is among the fussiest parts of building retrieval over internal documentation — guides, runbooks, postmortems, precisely the corpus Manticore cites. Most implementations handle it with a splitter library, an extra table for chunks, and aggregation logic to merge chunk hits into document-level results. Folding all of that into a column definition collapses an entire pipeline into one CREATE TABLE statement, and it corrects a truncation default that many deployments will not have noticed. The trade-offs are real — memory and indexing time multiply — but the recall difference on long-document corpora is large, and the published benchmark gives teams a concrete expectation of both the gains and the price.