Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Report
Data sources: ZENODO
addClaim

Optimizing a Hybrid Retrieval System for Accuracy, Speed, and Cost at Once

Authors: Mezo, Dezso;

Optimizing a Hybrid Retrieval System for Accuracy, Speed, and Cost at Once

Abstract

This practitioner's report describes how a production retrieval-augmented generation (RAG) system was optimized for accuracy, speed, and memory cost at the same time — a combination usually treated as a set of trade-offs. The system retrieves from a large corpus of long, hierarchically structured documents by fusing a dense (embedding) signal with a lexical (BM25-style) one, then generates cited answers with a large language model. The central finding is that the largest wins were cheap and structural rather than compute-heavy. One-bit (sign) vector quantization with an exact float rescore shrank the index by roughly 32× and removed the need for a dedicated vector database; an int8 encoder made the model about 4× smaller and 5× faster with byte-identical output; and a handful of lexical and structural ranking priors — title field weighting, home/sub-domain tier routing, and boilerplate demotion — improved accuracy at essentially no query-time cost. Equally important, several "obvious" heavy levers — a cross-encoder reranker, a second round of fine-tuning, and query rewriting — turned out neutral or negative once measured end to end. The report also covers evaluation methodology (semantic LLM judging versus brittle string matching), domain fine-tuning of the embedder, and the safety conditions for a semantic answer cache. All quantities are reported as ratios and relative changes rather than absolute, corpus-specific figures.

Powered by OpenAIRE graph
Found an issue? Give us feedback