Vector Index Evaluation: Measure Recall Before Speed

An approximate nearest-neighbor index is useful only if its faster results remain good enough for the application. Vector index evaluation should therefore start with a reference answer, not a latency chart. Measure how often the index returns the neighbors that an exact search would return, then decide what that loss means for your product.

New search extensions make this especially important. The maintainer's pgvectorsearch repository currently labels the project alpha and unsuitable for production. That is a reason to evaluate carefully, not a reason to assume poor quality or guaranteed superiority. An experimental index and an established index should both face the same workload and acceptance criteria.

Freeze the comparison inputs

Select a representative snapshot of vectors and queries. Record the embedding model, dimension, distance metric, normalization, and corpus version. If those change between runs, a result difference may come from the input rather than the index. Keep duplicates and near duplicates identifiable instead of silently removing them halfway through the experiment.

Use queries from the actual application where permission allows. Replace confidential text with approved fixtures before sharing results. Include short queries, ambiguous queries, and examples from important languages or customer segments. A single convenient synthetic distribution is rarely a complete description of real search traffic.

Do not use the tuning set as the final evaluation set. Tune parameters on one group and report performance on a separate group that remained unchanged. This reduces the temptation to optimize for a small collection of visible examples while claiming that the results generalize.

Build an exact reference

For each evaluation query, compute the nearest neighbors without approximation using the same metric and corpus. This can be expensive, which is why a bounded evaluation snapshot is useful. Store the query identifier and ordered reference IDs. If the application permits multiple equally good neighbors, define the tie policy before scoring.

Recall at k measures how many of the reference top k IDs appear in the approximate top k result:

recall@k = matching reference IDs / k

Suppose exact search returns ten IDs and the approximate search includes eight of them. Recall at ten is 0.8 for that query. This is an illustrative calculation, not a claim about a particular index. Compute the mean across queries and also inspect the worst-performing groups. A good average can hide a serious failure for one language or tenant.

Separate retrieval recall from answer quality

High neighbor recall does not guarantee a useful AI answer. The closest vector may point to an obsolete document, or the chunk may omit a necessary qualification. Conversely, several different retrieved passages may support the same correct answer. Evaluate the index as a search component, then evaluate the downstream system separately.

Keep a small set of manually reviewed failures. Classify them as index misses, embedding mismatch, filter mistakes, stale documents, or answer-generation errors. This makes the next action concrete. Raising a search parameter will not fix a document that was never indexed.

Our RAG retrieval evaluation checklist covers the broader retrieval question. This guide focuses on the approximate index tradeoff, not on replacing that application-level evaluation with a single numerical score.

Test filters as part of the search

Permission, tenant, language, and freshness filters can change the candidate population dramatically. An index that works well without filters may return too few eligible results when a filter is highly selective. Test the exact and approximate searches with identical filters and record the number of eligible documents.

Include cases with fewer than k eligible items. Do not call that an index failure if the corpus cannot supply k results. Define how your score handles those cases. Also verify that no result crosses a tenant boundary; relevance measurements never substitute for authorization tests.

Try a small, medium, and large eligible subset. Inspect both result completeness and elapsed time. If your implementation retrieves candidates first and filters later, make that behavior explicit. Increasing the candidate count may help recall but also increase work and memory.

Measure operational costs

Report median and tail latency, not just the fastest run. Include build time, index size, memory use, and behavior under concurrent queries. A warm index on an idle machine does not describe cold starts or a busy database that also handles writes.

After the initial run, insert, update, and delete a controlled set of documents. Verify that searches reflect those changes. Measure how maintenance affects query latency. Also test a restart and restore in a disposable environment, especially when an experimental extension warns that upgrade compatibility may be limited.

Keep the hardware and software versions in the results. Specify whether timings include network transfer and embedding computation. Comparing a server-side search measurement with an end-to-end application measurement produces an unfair speed comparison.

Make a decision table

For each parameter setting, store recall, tail latency, memory, and build cost. Choose a setting that meets the application's thresholds rather than declaring the fastest configuration universally best. A support search and a safety-sensitive retrieval workflow may reasonably choose different tradeoffs.

JSON Diff can help compare redacted experiment manifests so a hidden parameter change is visible. It does not compute recall or run a vector benchmark. Preserve raw query-level results alongside the summary in your own approved evaluation environment.

Conclusion

Evaluate an approximate vector index against a stable exact reference, realistic filters, and operational workloads. Then connect its misses to application consequences. A defensible decision explains what speed buys, what recall costs, and what evidence remains uncertain.