Manticore Search has introduced a chunking feature for vector search, allowing users to process long documents that exceed the token limits of embedding models. Previously, documents longer than the model's window were truncated, leading to information loss.
The new functionality is enabled by adding the chunk_strategy parameter to a vector column within the CREATE TABLE statement. Supported strategies include truncate, mean, fixed, recursive, and sentence. This eliminates the need for external text-splitting libraries or complex data pipelines.
When using chunking, Manticore creates multiple vectors for a single document. Despite this, the system returns the document as a single result. Parameters such as max_tokens, overlap_tokens, and max_chunks allow for fine-tuning the fragmentation process.
Internal testing on documentation consisting of 189 pages and approximately 298 000 words showed significant performance improvements. The recall@5 metric increased from 55,1% to 83,3%, while the Mean Reciprocal Rank (MRR) rose from 0,44 to 0,70.
