Reranking

1 / Why Initial Retrieval Is Not Enough

Initial retrieval — whether through Semantic Search, BM25, or both — produces a ranked candidate list that is good but not optimal. Bi-encoder embeddings (used in semantic search) encode the query and document independently, which enables fast vector search but sacrifices the ability to capture fine-grained query-document interactions. Reranking addresses this by applying a more expensive but more accurate model to a smaller candidate set.

2 / Cross-Encoder Architecture

In the Hybrid RAG pipeline for Phoenix, the reranking stage used a Cross-Encoder model that processed the query and each candidate document as a single concatenated input. This joint evaluation captured interactions between query terms and document content that independent encoding misses — for example, whether a specific query term appears in a document's conclusion versus its introduction.

3 / Latency Management

The Cross-Encoder's joint evaluation is computationally expensive compared to vector similarity lookup. Processing hundreds of candidates through a Cross-Encoder would create unacceptable latency. The solution was limiting the reranking stage to the top-k candidates (typically 10-20) from the initial retrieval fusion stage. This kept total pipeline latency within interactive response times while applying the most accurate scoring to the most promising candidates.

4 / Impact

Reranking consistently improved the relevance of the final context delivered to the LLM's context window. The quality difference was observable: retrieval without reranking occasionally included tangentially relevant documents that diluted the LLM's answer quality. With reranking, the final context set was more focused, producing more accurate and grounded responses.