Manticore Search adds built-in chunking to fix silent truncation in vector search

Manticore Search moves chunking into the database

Manticore Search has added built-in chunking for its automatic vector search, fixing a problem where long documents were silently truncated before being embedded. According to a Manticore Search blog post republished on dev.to, a new chunk_strategy option on model-backed vector columns splits each document at INSERT time, embeds every chunk, and searches across all of them — without an external splitter library, ingest pipeline, or a separate table to hold chunks.

The failure it fixes

Embedding models accept a fixed number of input tokens, often around 512, and ignore whatever exceeds that limit. Manticore illustrates the problem with a roughly 5,000-token document fed to a 512-token model: about 380 words are read, the remaining 3,600 are discarded, and the insert still succeeds. Search appears to work, but nothing later in the document can ever be retrieved, and the resulting vector may not even represent the document as a whole.

The standard workaround until now has been to handle splitting in application code: chunk documents upstream, embed each piece, and merge chunk-level hits back into document results with a query-level aggregation.

Five strategies, two column types

The feature is configured in the table definition rather than in a pipeline. Adding chunk_strategy to a vector column that has an embedding model attached enables the behaviour, alongside options for max_tokens (chunk size), overlap_tokens (tokens shared between neighbouring chunks) and max_chunks (a per-document ceiling).

There are five strategies:

• truncate — the previous default; keeps a single vector and drops text past the model's window
• mean — a single vector that averages the embeddings of all chunks
• fixed, recursive and sentence — chunk-aware strategies that store many vectors per document

According to the post, truncate and mean produce one vector per document and work on a float_vector column, while the other three require a float_vector_array column.

A key design choice is that a document remains a single search result. Individual chunks compete inside the index, but Manticore returns the document once, with knn_dist() reporting the distance to its closest chunk, and the k parameter counts documents rather than chunks. Queries themselves are never chunked, since they are short enough to embed whole.

Measured gains and measured costs

The Manticore team benchmarked the feature against their own manual: 189 pages and roughly 298,000 words. For questions whose answers sit beyond the model's input window, recall@5 rose from 55.1% to 83.3% and MRR improved from 0.44 to 0.70. The trade-off is resource usage: roughly 2.5 times the RAM and about four times the ingest time, because every chunk adds its own vector to the HNSW index.

Why it matters

Silent truncation is one of the harder-to-notice failure modes in retrieval-augmented systems: nothing errors, search returns plausible results, and everything past the model's token window is simply invisible. Guarding against it has meant owning a chunking pipeline and result-merging logic in application code. By moving chunking into the engine — as it already did with running the embedding model itself — Manticore removes that machinery, at a price in RAM and ingestion speed that is now explicit and tunable. For teams running semantic search over long internal documents such as runbooks, guides and postmortems, the benchmark suggests a substantial recall improvement for content that previously could never be found.