Retrieval quality depends on the task
The BEIR benchmark studies information retrieval across varied tasks and domains. Its broader lesson for builders is to evaluate across representative settings instead of assuming a method that works on one dataset will transfer unchanged to another. Historical benchmark rankings are not a substitute for measuring today's application.
A documentation search, a legal archive, and a product catalogue have different relevance criteria. Before comparing algorithms, define what a helpful result looks like for your users. A passage can share keywords with a question while failing to answer it.
Technical foundation: Thakur et al.: BEIR benchmark
Create a query set with evidence labels
Collect realistic questions, including short searches, full sentences, misspellings, and domain-specific identifiers. For each query, record one or more passages a reviewer considers relevant. Include questions with no valid source so the system can be evaluated on restraint as well as successful retrieval.
Keep a held-out set for later comparison. If every query is used to tune chunking and ranking, the measured improvement may simply reflect repeated adaptation to familiar examples. Record how the set was assembled and which user groups or languages it does not yet represent.
Inspect the failure before choosing a remedy
A missing result may come from an ingestion failure, a chunk boundary, a query mismatch, or ranking. Open the source and trace it through the index. If the necessary paragraph was never stored, changing the embedding model will not recover it.
If the passage is indexed but ranked too low, compare lexical and semantic retrieval on that case. A reranking step may improve the final order, but it also adds latency and cost. Measure the tradeoff with the same query set and retrieval budget. Do not add another model merely because the architecture looks more sophisticated.
Use metrics without losing the user task
Recall at k asks how much of the labeled relevant material appears in the top k results. Precision at k asks how much of that returned set is relevant. Ranking-sensitive metrics can further distinguish a useful first result from a useful result buried lower down. Labels and metric definitions must stay consistent between experiments.
Then test the final answer. More retrieved material may improve recall while distracting generation with contradictory or redundant passages. The winning retrieval configuration is the one that supports the desired user outcome within the allowed response time and resource budget.
A useful retrieval experiment
- Freeze a representative query set and relevance labels.
- Trace ingestion and chunk boundaries before changing models.
- Compare exact, lexical, and semantic cases separately.
- Measure quality alongside latency and context size.
- Verify end-to-end answers and legitimate no-answer behavior.
