The developer of the open-source Obsidian plugin Veynrel tested the impact of Rerank models on search quality. The study involved a synthetic repository of 64 Markdown notes and 28 search queries to evaluate whether a second-stage reranking process improves result relevance compared to standard semantic search.
Tests showed that for 24 queries with known answers, Rerank improved the nDCG@10 metric in 11 cases, while worsening it in 2 and leaving 11 unchanged. In 23 out of 24 cases, the initial semantic search already placed the relevant note first, and Rerank corrected the top result only once.
The author noted that in over half of the improved cases, the nDCG increase was due to reordering auxiliary materials rather than moving highly relevant documents. Consequently, the developer decided to implement Rerank as an optional feature triggered by a button rather than an automatic process.
The experiment also compared costs and latency. Using the OpenRouter API, the Voyage Rerank 3 Lite model was found to be approximately 392 times cheaper than Cohere for the tested volume of text. The developer also conducted a separate test on a demonstration repository containing 500 notes.
