Context Retrieval: Lexical vs Semantic vs Hybrid
Your bar: explain how a system decides what to send to the model, and compare lexical, semantic, hybrid search, ranking and re-ranking by purpose, advantages, disadvantages, and failure modes.
Your bar: explain how a system decides which facts reach the model when it cannot send everything, walk a real query (“find the authentication bug”) through candidate generation → ranking → selection, and compare lexical, semantic, hybrid search plus ranking vs re-ranking by purpose, advantages, disadvantages, and failure modes. By the end you can answer:
given a fixed token budget, how do you find the smallest set that still contains the answer?1This lesson is one pipeline, then the engines that power it. Each chip is one thing it makes permanent:
Context Selection
Lesson 1 fixed the budget: the window is small and you don’t own all of it. Selection is the discipline of choosing the smallest set of text that still contains the answer — wide net, then ruthless trim.
The hero: a selection funnel
Is Selection is the pipeline that turns a huge corpus into the curated few that fit the window: candidate generation, ranking, then packing within budget.
Why it exists The corpus dwarfs the window (Lesson 1). Sending everything is physically impossible and actively harmful — noise buries signal and costs more.
Like (world) A librarian who can carry ten books to your desk from a million-book stack: first pull a wide shelf, then narrow to the ten that answer your question.
Like (code) A WHERE + ORDER BY + LIMIT
pipeline: filter to candidates, sort by a score, take the top N that fit the page.
Relevance signals — what each catches, what it misses
| Signal | What it catches | What it misses |
|---|---|---|
| Lexical match | Exact terms, identifiers, error codes | Synonyms & paraphrase (different words, same idea) |
| Semantic similarity | Meaning, synonyms, “asks the same thing” | Rare exact tokens (IDs, function names, codes) |
| Structural / graph | Imports, call sites, type/dependency links | Conceptual matches with no code edge between them |
| Recency | The just-changed file, the latest doc version | Stable-but-correct old code that never changes |
| Authority / trust | Canonical source, owned doc, signed record | The right fact when it only lives in a low-trust place |
No single signal is sufficient. Each is blind to a different class of right answer — which is exactly why hybrid retrieval (Module 2) exists.
“Relevant” means different things per system
| System | The query | What “relevant” means |
|---|---|---|
| Billing platform | “Why was account 4471 double-charged?” | The exact account records & ledger rows — lexical/ID match, authority. A “similar” account is worse than useless. |
| GitHub repository | “Where do we validate the JWT?” | The right symbols plus their dependencies — semantic concept + structural call sites & imports. |
| Documentation portal | “How do I rotate an API key?” | The right article at the right version — semantic match gated by recency/version, authority. |
Walk-through: “Find the authentication bug”
✓ More irrelevant files lower signal, raise cost, and push the right lines into the “lost in the middle” zone. Smaller-but-correct wins.
✓ It’s a pipeline: generate (recall) → rank (order) → pack (budget). Each stage can be the bottleneck independently.
📥 Memory rule: You can’t send everything — selection is choosing the smallest set that still contains the answer. Cast wide for recall, then trim to fit.
Memory check
- Name the three stages of the selection funnel. → candidate generation (recall) → ranking (order) → selection/packing (fit budget)
- Which stage, if it fails, can't be rescued downstream? → candidate generation — you can't rank or pack what was never retrieved
- What cheap context do you pack instead of full bodies? → signatures & docstrings of related code, plus structural links (imports, call sites)
Your code assistant keeps missing the actual buggy file. Where in the selection pipeline do you look first, and how do you prove it?
Hit these points: instrument the pipeline per stage, don’t guess → first check candidate generation recall: was the buggy file ever in the candidate set? if no, ranking/packing are irrelevant → if recall is the gap, add the missing signal — usually structural (imports/call sites) or lexical for an exact symbol the embedding model washed out → if recall is fine but the file ranked low, fix ranking / add a re-rank → if it ranked high but got dropped, it’s a packing/budget problem → measure retrieval hit-rate as a first-class metric, not just end-answer quality.
Retrieval
Module 1 said “cast a wide net, then rank.” This is the machinery that does it: lexical finds exact, semantic finds meaning, hybrid finds both, and re-ranking sharpens the top. Each one has a job it does well and a case where it falls down.
The hero: a two-stage retrieval pipeline
The four engines, side by side
| Technique | How it scores | Best at | Signature failure |
|---|---|---|---|
| Lexical (BM25) | TF-IDF / BM25 on exact terms | Identifiers, error codes, exact strings | Vocabulary mismatch → total miss |
| Semantic (vector) | Cosine of embeddings via ANN | Meaning, synonyms, paraphrase | Plausible-but-wrong; misses exact tokens |
| Hybrid | Fuse both (RRF / weighted) | Exact and semantic together | Bounded by both stages’ recall |
| Re-rank | Cross-encoder, query+doc jointly | Ordering the final top results | Can’t recover a missed document |
Search (lexical / keyword / BM25)
Is Exact-term matching: score documents by how well their words match the query words, weighted by term frequency and rarity (TF-IDF / BM25). 3
Why it exists Some queries are the exact tokens — an error code, a function name, an account ID. For those, meaning is irrelevant; the literal string is the answer.
Advantages Precise on exact terms & identifiers, fast, cheap, interpretable, needs no model — you can read why it matched.
Failure mode No synonyms or semantics: if the user phrases it differently than the doc (vocabulary mismatch), it returns a total miss.
Semantic search (embeddings / vector)
Is Encode query and chunks into vectors; rank by cosine similarity using an approximate-nearest-neighbor index (e.g. HNSW). 4
Why it exists Users rarely use the doc’s exact words. Embeddings match on meaning, so a paraphrase still finds the right passage.
Advantages Captures meaning, synonyms, paraphrase; robust to wording; finds the right idea even with zero shared words.
Failure mode Returns “topically similar but wrong”; weak on rare exact tokens — it can’t reliably match an exact error string or ID.
Hybrid search
Is Run lexical and semantic together, then fuse the two result lists into one ranking — commonly Reciprocal Rank Fusion or a weighted blend. 5
Why it exists The two engines fail on opposite cases: lexical whiffs on paraphrase, semantic whiffs on exact tokens. Run both and one usually catches what the other drops.
Advantages Covers both exact and meaning, so fewer right documents slip through. Anthropic’s Contextual Retrieval pairs embeddings with BM25 for exactly this reason. 2
Failure mode More infra and complexity, fusion weights to tune; still bounded by the recall of both first-stage retrievers.
Example · fusing the two rankings
const K = 60; // RRF damping: how fast rank 1's lead over rank 10 decays.
// Fuse on RANK, not score. BM25 and cosine live on different, unbounded
// scales, so adding or normalising the two scores compares nothing real.
function reciprocalRankFusion(lists: string[][], k = K): string[] {
const fused = new Map<string, number>();
for (const list of lists) {
list.forEach((chunkId, index) => {
const rank = index + 1;
fused.set(chunkId, (fused.get(chunkId) ?? 0) + 1 / (k + rank));
});
}
return [...fused.entries()]
.sort((a, b) => b[1] - a[1])
.map(([chunkId]) => chunkId);
}
async function retrieve(query: string, topK = 50) {
// Both retrievers run over the same corpus and return ordered chunk ids.
// Whichever one misses the passage, the other can still surface it.
const [lexical, semantic] = await Promise.all([
index.bm25(query, topK),
index.vector(embed(query), topK),
]);
// Fusion picks the candidates; the cross-encoder decides the final order.
const candidates = reciprocalRankFusion([lexical, semantic]);
return rerank(query, candidates.slice(0, 20));
}Fusion happens on rank, not score. A BM25 score of 14.2 and a cosine of 0.81 are not comparable quantities; their positions in two ordered lists are.
A chunk that only one retriever ranked well still surfaces — which is the entire reason for paying to run both.
kis the knob. A largerkflattens the curve so a chunk ranked first by one engine stops dominating a chunk ranked third by both.Fusion is a recall stage. The expensive cross-encoder runs after it, over a much shorter list — the two-stage shape from the top of this module.
RRF cannot recover a chunk neither retriever returned. Recall is capped by the two
topKvalues above it, and that ceiling is the first thing worth measuring.Both lists must address the same chunk ids. If the lexical and vector indexes chunk differently, fusion silently degrades into concatenation.
k = 60is the value from the original RRF paper 5, not a fact about your corpus — tune it against a labelled set, exactly like thetopKvalues.Running both retrievers doubles first-stage cost and adds an embedding call to every query.
Promise.allhides the latency; it does not hide the spend.
Ranking vs re-ranking — two different jobs
Ranking — is Order candidates by a cheap score (BM25, cosine). First stage. Purpose: optimize recall so the right document is somewhere in the list.
Re-ranking — is A second, expensive, accurate pass (cross-encoder) that re-scores the top-N jointly with the query. Purpose: optimize precision at the top.
Re-rank advantage Large precision lift on the final top-k the model actually sees — the items most likely to be packed into the window.
Re-rank failure Latency and cost; runs on top-N only; cannot recover a document the first stage already missed.
✓ Embeddings are weak on exact tokens (error codes, IDs, function names). Hybrid keeps BM25 precisely for those cases.
✓ Re-ranking only reorders what stage 1 returned. If recall missed the doc, no re-ranker can surface it — fix recall first.
📥 Memory rule: First-stage retrieval buys recall; re-ranking buys precision. Lexical finds exact, semantic finds meaning, hybrid finds both.
Memory check
- Which engine matches an exact error code, and which a paraphrase? → lexical/BM25 for the exact code; semantic/vector for the paraphrase
- What does the first stage optimize for vs the re-rank stage? → first stage = recall (don't miss it); re-rank = precision (best at the top)
- Why can't a re-ranker save bad retrieval? → it only re-scores the top-N it was given; a missed document never reaches it
Users complain semantic search returns “close but wrong” results and sometimes whiffs on exact error codes. What’s your fix and why?
Hit these points: name the root cause — embeddings score topical similarity, so they return plausible-but-wrong and wash out rare exact tokens like error codes → add hybrid: run BM25 alongside the vector search and fuse (RRF) so exact strings are caught by lexical and meaning by semantic → add a cross-encoder re-rank over the fused top-N to push the genuinely best result to the top (precision) → frame the split: hybrid fixes recall, re-rank fixes precision → verify with retrieval hit-rate on a labelled set, not just end-answer vibes; tune fusion weights against that set.
Retrieval practice — test the two modules
Q1. The three stages of the selection funnel, in order, are…
- Embed the corpus, store the vectors, then stream the chunks straight into the model
- Candidate generation (recall), then ranking (order), then selection and packing to budget
- Re-rank the corpus, drop the duplicates, then summarize everything into one digest
- Parse the schema, run the SQL query, then cache the response for the next caller
Q2. If the buggy file is never returned by candidate generation, the result is…
- Re-ranking promotes it because the cross-encoder reads the file body in detail
- A larger context window includes it automatically once the budget is increased
- The ranking stage recomputes the candidates and recovers the file on a retry
- No downstream stage can find it — you cannot rank or pack what was never retrieved
Q3. Lexical search (BM25) is the right tool, and semantic search the wrong one, when the query is…
- An exact error code or function name where the literal string must match exactly
- A vague description of a feeling the user has about how the product behaves today
- A paraphrase that shares no words at all with the document that answers it best
- A broad concept that appears across many documents written in different wordings
Q4. The signature failure mode of semantic (vector) search is that it…
- Cannot run at all unless the entire corpus first fits inside the context window
- Refuses any query that contains a synonym of a word found in the document set
- Returns plausible-but-wrong matches and struggles with rare exact tokens like IDs
- Always returns the single oldest document because recency is never a factor
Q5. Re-ranking (a cross-encoder over the top-N) primarily buys you…
- Recall, by widening the net so more candidate documents enter the first stage
- Precision at the top, by re-scoring candidates jointly with the query in detail
- Latency, because scoring query and document together is cheaper than BM25 alone
- Recovery of missed documents the first-stage retriever never returned at all
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
Name the three stages of the selection pipeline, in order, and what each one does.
What does lexical / BM25 search actually match on, and what is it good at?
What does semantic / embedding search match on, and why use it?
What is hybrid search, and what does fusion (e.g. RRF) do in it?
Users say semantic search returns “close but wrong” results and sometimes whiffs on exact error codes. What's your fix and why?
Your code assistant keeps missing the actual buggy file. Where in the pipeline do you look first, and how do you prove it?
Distinguish ranking from re-ranking. When is a cross-encoder worth its cost, and when is it a waste?
Someone proposes “just embed everything and drop BM25 — embeddings are strictly better.” Push back.
Design a retrieval strategy when you don't yet know the query mix — some lookups are exact IDs, some are vague conceptual questions, and the corpus keeps changing.
“Relevant” means different things per system. Contrast a billing platform, a code repo, and a docs portal — and say how that changes your retrieval.
You can raise recall (wider net) or precision (aggressive re-rank + tighter packing), but not both for free. How do you reason about the trade-off?
- Clarify the query mix & the cost of a miss (exact-ID lookups vs vague concepts; how bad is a wrong-but-plausible answer?).
- Define “relevant” for this domain and pick signals (lexical, semantic, structural, recency, authority).
- Stage 1 for recall: hybrid candidate generation (BM25 + vector) and how you fuse the lists.
- Stage 2 for precision: whether a cross-encoder re-rank on the top-N earns its latency.
- Packing within the token budget: top-k, full bodies vs signatures/docstrings, ordering.
- Freshness & failure modes: re-indexing the changing corpus, vocabulary mismatch, plausible-but-wrong.
- Measure: retrieval hit-rate and top-k precision on a labelled set; tune weights against it.