RAG Reranking Explained: Why Vector Similarity Isn't Enough

Key takeaways

Vector search uses bi-encoders which compress text into independent static embeddings, losing subtle token-level interactions and syntactic negation.
Rerankers use cross-encoders that process both the user query and candidate document simultaneously through self-attention layers.
A two-stage retrieval pipeline combines high-recall vector search (retrieving top 50-100 candidates) with high-precision reranking (returning top 3-5 to the LLM).
Modern rerankers like Cohere Rerank 3.5, BGE-Reranker-v2, and ColBERT late-interaction models increase Hit Rate by 20% to 35%.
Reranking adds 30ms-80ms of latency per query, but dramatically reduces prompt token costs and hallucination rates.

The compression penalty of vector search

Vector embeddings are a remarkable engineering shortcut. By converting sentences and paragraphs into floating-point vectors, systems can search through millions of documents in sub-10 millisecond latencies using approximate nearest neighbor (ANN) indexes.

However, compressing an entire 500-word passage into a single geometric vector comes with a severe compression penalty. A bi-encoder embeds the query and the document completely independently without ever allowing their tokens to interact. Consequently, vector search struggles with negation ("looking for Python, NOT Django"), precise numeric ranges, and domain-specific abbreviations.

This is where reranking in RAG transforms performance. Instead of feeding the top-5 raw vector matches directly to the generative LLM, production pipelines employ a two-stage retrieval architecture.

Bi-encoders process query and document independently, losing fine token interactions.
Negative filters and numeric conditions are frequently distorted by dense embeddings.
Two-stage retrieval repairs these blind spots by introducing deep cross-attention.

Bi-encoders vs Cross-encoders explained

To understand why rerankers outperform raw vector search, developers must understand the fundamental difference between Bi-Encoders and Cross-Encoders.

Architectural Comparison: Bi-Encoder vs Cross-Encoder
Dimension	Bi-Encoder (Vector Search)	Cross-Encoder (Reranker)
Input Handling	Query and Document encoded separately into vectors	Query and Document concatenated into single transformer input
Attention Scope	Self-attention within query only, within doc only	Full cross-attention between all query tokens and doc tokens
Computation Cost	Pre-computed offline; fast dot-product at runtime	Computed in real-time for each query-document pair
Latency	5ms - 15ms across millions of documents	30ms - 80ms across 50-100 candidates
Precision	Moderate (broad thematic match)	Extremely high (fine-grained relevance score 0.0 to 1.0)

How two-stage retrieval works in practice

A two-stage retrieval pipeline combines the complementary strengths of bi-encoders (speed and scale) and cross-encoders (deep semantic precision).

In Stage 1 (Candidate Generation), a hybrid search engine retrieves a wide candidate pool—typically between 50 and 100 documents—using a mix of vector similarity and BM25 keyword matching. This stage prioritizes high recall over precision.

In Stage 2 (Precision Scoring), cross-encoder reranking scores every candidate against the original user query, computing a relevance score. The pipeline sorts candidates by this score, strips away low-scoring distractors, and sends only the top 3 to 5 pristine chunks to the LLM prompt.

ColBERT and late-interaction reranking

For high-throughput systems where full cross-encoder scoring is too computationally expensive, ColBERT (Contextualized Late Interaction over BERT) offers an elegant intermediate solution.

ColBERT computes token-level vector representations for both the query and document, storing multiple vectors per chunk. At search time, it computes the maximum similarity (MaxSim) across token pairs.

This allows ColBERT to retain token-level cross-attention expressiveness while operating orders of magnitude faster than traditional cross-encoders, making it ideal for sub-50ms enterprise retrieval workloads.

Late interaction preserves token-level contextual representations without full cross-transformer cost.
MaxSim operators rapidly aggregate pairwise token matches across indexed document passages.
Delivers 95% of cross-encoder accuracy at a fraction of the computational overhead.

Benchmarking the latency and cost trade-off

Developers often ask: is reranking worth the additional latency? In production benchmarks across enterprise knowledge bases, passing 50 candidates through a modern reranker adds between 35ms and 65ms of latency.

However, because rerankers eliminate irrelevant context, you can safely decrease the final context window size from 10 chunks down to 3 chunks. This saves 2,000+ input tokens per prompt, accelerating LLM generation time and reducing overall end-to-end latency.

Frequently asked questions

What is the ideal candidate pool size to send to a reranker?

Between 30 and 100 documents is standard. Reranking fewer than 20 candidates risks missing items excluded by the first stage, while reranking more than 150 candidates introduces diminishing returns and unnecessary latency.

Can I use an LLM instead of a dedicated cross-encoder for reranking?

While frontier LLMs can rerank documents via zero-shot prompting, dedicated cross-encoder models (like BGE-Reranker or Cohere Rerank) are 20x faster, dramatically cheaper, and outperform LLM prompts on standard benchmarks.

Does reranking replace hybrid search?

No. Reranking complements hybrid search. Hybrid search gathers the top candidate pool from both vector and keyword indexes, and the reranker unifies and rescores them with deep cross-attention.

Check this against your own resume

Scan your CV against a real job description, or build a parse-safe one from scratch. Your first scan costs nothing.

Free ATS checker AI resume builder

Keep reading

AI EngineeringHybrid search vs vector search: why pure vector RAG quietly failsPure vector search is great for conceptual synonyms, but notoriously bad at finding product SKUs, error codes, and exact names. Hybrid search combines BM25 and vector embeddings to give you the best of both worlds.

AI EngineeringAdvanced RAG retrieval techniques: moving beyond basic chunkingNaive chunk-and-search reaches an accuracy ceiling quickly. Advanced techniques like sentence-window retrieval, parent-document linking, query rewriting, and cross-encoder reranking push retrieval accuracy past 95%.

AI EngineeringHow to reduce RAG latency and costs: an optimization guideRAG applications get expensive and slow at scale. Here are the 6 proven engineering optimizations that cut latency by 75% and slash API token costs by 80% in production.

Referenced in these guides

All career guides

RAG reranking explained: why vector similarity fails and how cross-encoders fix it