dineshbyte
  • Lesson 7 of 9
  • 12 min read
  • Lesson

Advanced RAG: HyDE, GraphRAG & Agentic Retrieval

Your bar: transform the query before you retrieve, match the retrieval structure to the question shape, and know when to graduate from a static pipeline to a self-correcting agentic loop.

Your bar: transform the raw query before you retrieve, match the retrieval structure to the shape of the question, and know exactly when to graduate from a static pipeline to a self-correcting agentic loop. By the end you can answer:

given a RAG system that retrieves the wrong thing, which lever — pre-retrieval, retrieval structure, or an agentic loop — do you pull, and what does each cost?1

Lesson 3 shipped the baseline RAG pipeline. This lesson covers the three changes that turn that baseline into a system. Each chip is one lever:

the raw query is a bad search queryso transform it first (rewrite · HyDE · decompose)flat top-k is not the only retrieval structureand retrieval can be an action in a loop, not a step
Baseline RAG (Lesson 3) — three places it levels upQuestionPre-retrievalM1 · fix the queryRetrievalM2 · structure itLLM → AnswerM3 · agentic / self-correcting loopgrade docs · re-retrieve · critique the draftretrieval becomes an ACTION, not one stepdraftre-searchrecall the question can't buyprecision + the right shape
Three independent upgrades to the same pipeline. M1 rewrites the input before it hits the index, M2 changes what "retrieval" returns, and M3 wraps the whole thing in a loop that checks its own work. You can adopt one, two, or all three.1
1

Query Transformation

The raw user query is often a bad search query: short, vague, full of pronouns, worded nothing like the documents that answer it. Fix it in a pre-retrieval stage, before it touches the index.

A stage that didn’t exist in the baseline

Insert a transform step BEFORE retrievalRaw query"why is it slow?"Pre-retrieval transformrewrite · expand · HyDEdecompose · step-backroute to the right index"latency p99 checkout API""db connection pool size"hypothetical answer doc→ retrieve each, union
One vague query in, several precise queries (or a hypothetical answer) out. The model that answers is unchanged; you hand the index a better search than the user typed.2

The pre-retrieval techniques, pinned down

TechniqueWhat problem it solvesCost
Query rewritingVague / pronoun-laden / misspelled input — clean, expand, disambiguate it.1 extra LLM call
Multi-queryOne phrasing misses; generate N paraphrases, retrieve each, union & dedupe → higher recall.N retrievals + dedupe
HyDEQuestions look nothing like answers; embed a drafted hypothetical answer to bridge the gap.1 LLM draft + embed
DecompositionA multi-part question no single chunk answers; split into sub-queries, retrieve each, combine.1 split + k retrievals
Step-backToo specific to find grounding; ask a broader question first, then the specific one.1 extra retrieval
RoutingWrong datasource entirely; classify the query and send it to docs vs code vs SQL.1 classify call

HyDE — the query/document vocabulary gap

Embed the ANSWER's shape, not the question'squestionanswerdocHyDE docfar apart in vector space (different vocab)LLM drafts a fake answer → embed THAT → lands next to the real doc
Questions and answers use different words, so the question's embedding sits far from the document that answers it. HyDE has the LLM draft a plausible answer and embeds that — even a partly-wrong draft has the right shape to retrieve the real source.2

Is Pre-retrieval is any transform applied to the query before it hits the retriever: rewrite, multi-query, HyDE, decomposition, step-back, or routing. The index and model are untouched.

Why it exists Recall lost at retrieval cannot be recovered downstream — no reranker, no bigger model can return a document that was never retrieved. The cheapest place to fix recall is the query.

Like (world) A good librarian doesn’t search your exact words. They restate “why is it slow?” as “checkout latency, connection pool” — the terms the shelves are actually organized by.

Like (code) Query normalization / canonicalization before a DB lookup: lowercasing, expanding synonyms, splitting a compound filter into sub-queries. Same input cleanup, applied to retrieval.

✗ “Just embed the user’s question and search.”
✓ The question is the worst search query you have — vague and worded unlike the answer. Transform it first; HyDE and multi-query routinely beat the raw query.
✗ “A better reranker will fix our low recall.”
✓ Reranking only reorders what was retrieved. If the right doc never entered the candidate set, no reranker can surface it — that’s a pre-retrieval problem.

📥 Memory rule: Fix the query before you fix the index. HyDE and multi-query buy recall that no reranker can recover.

Memory check
  • Why can't a reranker recover lost recall? → it only reorders the candidates already retrieved; a document never in the set can't be ranked up
  • What gap does HyDE bridge? → the query↔document vocabulary gap — questions look unlike answers, so it embeds a drafted hypothetical answer instead
  • Multi-query vs decomposition? → multi-query = N paraphrases of the SAME question (recall); decomposition = split a multi-part question into sub-questions (coverage)
M1 — Your RAG recall is poor on multi-part questions like “compare our refund policy in the EU vs the US.” Which pre-retrieval technique, and why?

Hit these points: this is two facts in one query, so a single embedding averages them and matches neither cleanly → decomposition: split into “EU refund policy” and “US refund policy”, retrieve each, then combine the chunks for the answer → optionally add multi-query per sub-question for recall → contrast: HyDE helps the vocabulary gap, not the multi-fact problem; a bigger window doesn’t help if neither fact was retrieved → verify by checking that chunks for both regions appear in the assembled context, not just the answer.

2

Advanced Retrieval Structures

Lesson 3’s baseline was flat chunk + top-k: one embedding per chunk, return the nearest k. That’s a floor. The shape of the question should pick the structure of the retriever.

Flat vs structured retrieval

Flat: match chunk, return chunkSmall-to-big: match small, return parentquery → nearest chunkmatched chunkprecise but no contextanswer often lacks surrounding contextquery → small precise chunksmall chunkreturn PARENT documentprecision of small + context of big
You can decouple what you match on from what you return. Match tiny chunks so the embedding is precise, but hand the model the larger parent so it has the surrounding context to reason with.1

Which structure for which question

StructureWhat it’s good atCost / complexityReach for it when…
Parent-doc / small-to-bigPrecision of small chunks + context of the parent.Low — extra parent lookupLocal facts that need surrounding context.
Hierarchical / summaryRetrieve over summaries, then drill into details.Medium — summary indexLarge docs; “which section, then what.”
Contextual retrievalPrepend chunk-situating context + add BM25 → fewer retrieval failures.Medium — re-embed + dual indexChunks ambiguous on their own (Lesson 3).
Multi-vector / ColBERTToken-level late interaction → higher precision.High — many vectors/chunkPrecision-critical, latency-tolerant search.
Metadata filteringPre-filter by date / type / tenant before vector search.Low — needs clean metadataRecency, access control, multi-tenant.
GraphRAGSubgraphs + community summaries for global themes.High — build & maintain graph”Connect-the-dots across the whole corpus.”

Local fact vs global theme — the GraphRAG line

Question shapeFlat chunk + top-kGraphRAG
”What is the retry limit?”✓ one chunk answers it✗ overkill, slower
”What themes recur across all incidents?”✗ no single chunk holds it✓ community summaries answer it

Read the diagonal. A local fact lives in one chunk, so flat retrieval finds it and GraphRAG is wasted complexity. A global theme is spread across the whole corpus — no chunk contains it, so only a graph that pre-aggregates relationships can answer.

Is Advanced retrieval structures change what you index and return — parents, summaries, situated chunks, token vectors, filtered candidates, or graph subgraphs — instead of one flat vector per chunk.

Why it exists Different questions have different shapes. Flat top-k optimizes one trade-off (chunk-level similarity); it can’t serve precision-with-context, recency, or corpus-wide synthesis at the same time.

Like (world) A library uses a card catalog (summaries) to find the shelf, the full book (parent) to read context, and a citation graph to answer “what influenced what.” Different tools, different questions.

Like (code) Choosing an index for a query pattern: a B-tree for point lookups, a covering index for range scans, a graph DB for traversals. You match the data structure to the access pattern.

✗ “GraphRAG is a strictly better RAG — use it everywhere.”
✓ It wins on global, theme-spanning questions and loses on local facts: it’s costlier to build and run. For “what is X?” flat top-k is cheaper and just as good.
✗ “Smaller chunks just mean worse context for the model.”
✓ With small-to-big you match on the small chunk for precision but return the parent for context — you get both, not a trade-off.

📥 Memory rule: Match the retrieval structure to the question shape — local fact → small-to-big; global theme → GraphRAG.

Memory check
  • What does small-to-big decouple? → what you MATCH on (small precise chunk) from what you RETURN (the larger parent for context)
  • When does GraphRAG beat flat top-k? → global "connect-the-dots / themes across the whole corpus" questions no single chunk can answer; not for local facts
  • What two things does contextual retrieval add? → a short chunk-situating context prepended before embedding, plus a BM25 lexical index alongside the vectors (Anthropic)
M2 — A teammate proposes rebuilding the whole RAG stack on GraphRAG because “it’s the state of the art.” How do you respond as the senior in the room?

Hit these points: first ask what questions users actually ask — pull the distribution → if most are local lookups (“what is the limit for X?”), flat top-k or small-to-big already answers them at a fraction of the cost → GraphRAG earns its keep only on global, theme-spanning questions, and it carries real cost: building the entity graph, generating community summaries, keeping it fresh on every reindex → propose it as an added retriever routed to for global queries, not a rip-and-replace → name the trade-off: complexity and maintenance vs a capability you only need for a slice of traffic.

3

Agentic & Self-Correcting RAG

Modules 1–2 still ran retrieval once, as a fixed step. The last shift makes retrieval an action inside a loop: the model decides whether to retrieve, grades what it got, and critiques its own draft. This is the read-on-demand pattern from Lesson 3 and the ai-agents track.

The self-correcting loop

Retrieve → grade → (good? answer : re-retrieve) → critique → finalizeRetrieveGrade docsrelevant enough?Draft answergoodRe-retrieve /web fallbacklow qualityCritique draftgrounded? revise?Finalizenot grounded → retrieve again
Static RAG runs left-to-right once. Agentic RAG adds the red and grey return paths: bad docs trigger re-retrieval or a web fallback, and an ungrounded draft is sent back for another pass before anything reaches the user.3

Three named patterns

PatternThe control loopWhat it buys
Agentic RAGLLM decides whether / what / when to retrieve; can issue many searches, use tools, iterate. (This is what Claude Code does.)Handles open-ended, multi-step questions.
Self-RAGModel retrieves on demand, then reflects: critiques its own draft and revises before answering.Catches ungrounded / unsupported claims.
Corrective RAG (CRAG)Grade retrieved docs; if quality is low, fall back (e.g. web search) or re-retrieve before answering.Robustness when the index comes up short.

The trade-off you must name

Static pipelineAgentic loop
Quality / robustnessFixed — one shot, no recovery✓ checks & corrects itself
Latency✓ one pass, predictable✗ multiple round-trips
Cost✓ few LLM calls✗ grading + re-tries cost more
Complexity✓ simple to reason about✗ loops, budgets, stop conditions

Read the columns. The agentic loop wins exactly one row — quality — and pays in latency, cost, and complexity for it. Adopt it only where correctness justifies the bill; a static pipeline is the right default for simple, well-covered questions.

Is Agentic / self-correcting RAG turns retrieval from a fixed pipeline step into an action the model invokes in a loop — deciding when to search, grading results, and critiquing its own draft before finalizing.

Why it exists A static pipeline answers once from whatever it retrieved, right or wrong. A loop lets the system detect a bad retrieval and recover — re-search, fall back, or revise — instead of confidently answering from junk.

Like (world) A researcher who checks whether their sources actually support the claim, goes back to the library when they don’t, and proofreads the draft before submitting — rather than writing from the first book they grabbed.

Like (code) A retry-with-validation loop versus a single fire-and-forget call: validate the response, on failure refine the request and retry, with a max-attempts budget so it terminates.

✗ “Agentic RAG is the upgrade — always loop.”
✓ Loops add latency, cost, and failure modes (runaway re-tries, no stop condition). For simple, well-covered questions a static pipeline is correct and far cheaper.
✗ “Self-correction means the model can’t ever be wrong now.”
✓ The critic is the same fallible model; it reduces ungrounded answers, it doesn’t eliminate them. You still need eval and stop conditions, not blind trust.

📥 Memory rule: Static RAG answers once; agentic RAG checks its own work — pay the extra calls only when correctness justifies them.

Memory check
  • What does CRAG do when retrieved docs grade poorly? → falls back (e.g. web search) or re-retrieves before answering, instead of answering from low-quality docs
  • What is the core trade-off of agentic loops? → higher quality / robustness vs more latency, cost, and complexity — only worth it when correctness justifies it
  • What does Self-RAG add over plain retrieval? → reflection — the model critiques its own draft for grounding and revises before finalizing
M3 — When would you NOT add an agentic loop to a RAG system, and how do you decide?

Hit these points: when the question distribution is simple and well-covered by the index — a static pipeline already answers correctly, so a loop only adds latency and cost → when latency budget is tight (interactive UX) and the extra round-trips break the SLA → when there’s no reliable grading signal, the loop can’t tell good from bad and just burns calls → decide from eval data: measure static-pipeline accuracy first; add the loop only on the slice where correctness is high-stakes and static accuracy is provably insufficient → always cap attempts and define a stop condition so it terminates.

Retrieval practice — test the three modules

Q1. The core argument for pre-retrieval query transformation is that…
  • A larger context window removes the need to ever transform the user's raw query at all
  • The reranker can always recover any relevant document the retriever happened to miss
  • Recall lost at retrieval is unrecoverable downstream, so fix the query before the index
  • The model answers better when it is shown the user's exact wording with no changes
Q2. HyDE (Hypothetical Document Embeddings) works by…
  • Drafting a hypothetical answer and embedding that to bridge the query/document gap
  • Caching every prior answer and replaying it when a similar question is seen again
  • Splitting one compound question into several smaller sub-questions before retrieval
  • Ranking candidates with a cross-encoder after the first vector search has finished
Q3. The "small-to-big" (parent-document) retriever gives you…
  • A way to store one document across many shards so each query hits every shard
  • A graph of entities whose community summaries answer corpus-wide theme questions
  • A lexical BM25 index bolted onto the vectors to catch exact keyword and ID matches
  • Precision of a small matched chunk plus the surrounding context of its larger parent
Q4. GraphRAG most clearly beats flat chunk + top-k when the question is…
  • A single local lookup such as "what is the configured retry limit for the job?"
  • A global "connect-the-dots" theme that no single chunk in the corpus contains
  • A latency-critical request that must return its answer in well under one second
  • A request that must be filtered by tenant and date before any vector search runs
Q5. Corrective RAG (CRAG) differs from a static pipeline because it…
  • Embeds the question and returns the nearest k chunks exactly once with no checks
  • Fine-tunes the base model on retrieved documents so it memorizes them for later
  • Grades the retrieved docs and falls back or re-retrieves when their quality is low
  • Prepends a short situating summary to each chunk before it is embedded and indexed

Interview — pick your bar

Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).

What is pre-retrieval query transformation, and name the main techniques.
Hit these points: it's any transform applied to the raw query before it hits the retriever — the index and model are left untouched → rewriting cleans up vague / pronoun-laden / misspelled input → multi-query generates N paraphrases, retrieves each, unions and dedupes → HyDE drafts a hypothetical answer and embeds that → decomposition splits a multi-part question into sub-queries → step-back asks a broader question first → routing classifies the query and sends it to the right source → the point: the raw user query is usually a bad search query.
Explain how HyDE works and what gap it bridges.
Hit these points: HyDE = Hypothetical Document Embeddings → the problem it fixes is the query↔document vocabulary gap — a question is worded nothing like the document that answers it, so its embedding sits far from the real source → HyDE has the LLM draft a plausible (hypothetical) answer and embeds that, not the question → even a partly-wrong draft has the right shape and lands near the real document in vector space → cost: one extra LLM draft plus an embed call → the answer-generating model itself is unchanged.
What does the small-to-big (parent-document) retriever do?
Hit these points: it decouples what you match on from what you return → you embed and match on small, precise chunks so similarity is sharp → but you hand the model back the larger parent document so it has the surrounding context to reason with → you get the precision of small chunks and the context of big ones, not a trade-off between them → cost is low — just an extra parent lookup after the match.
Define agentic RAG, Self-RAG, and Corrective RAG (CRAG) — what's the control loop in each?
Hit these points: baseline RAG = retrieve → assemble → LLM → answer, once; an agent = an LLM using tools → agentic RAG: the LLM decides whether / what / when to retrieve in a loop, issuing many searches and iterating → Self-RAG: the model retrieves on demand, then reflects — critiques its own draft for grounding and revises before answering → CRAG: grade the retrieved docs; if quality is low, fall back (e.g. web search) or re-retrieve before answering → common thread: retrieval becomes an action inside a loop, not a fixed pipeline step.
Your RAG recall is poor on multi-part questions like "compare our refund policy in the EU vs the US." Which pre-retrieval technique, and why?
Hit these points: this is two facts in one query, so a single embedding averages them and matches neither cleanly → reach for decomposition: split into "EU refund policy" and "US refund policy", retrieve each, then combine the chunks for the answer → optionally add multi-query per sub-question for recall → contrast: HyDE addresses the vocabulary gap, not the multi-fact problem; a bigger window doesn't help if neither fact was retrieved → verify by checking that chunks for both regions appear in the assembled context, not just the final answer.
Routing and step-back are both pre-retrieval. Contrast what they do and when each is the right call.
Hit these points: routing classifies the query and sends it to the right datasource / index / tool — code vs docs vs SQL — so you don't search the wrong corpus at all → step-back generalizes a too-specific question to pull broader grounding first, then retrieves for the specific one → routing is about where to look; step-back is about how broad to look → use routing when sources are heterogeneous; use step-back when the specific query is too narrow to match useful context → they compose: route first, then step-back within the chosen index → each is one cheap extra call, so weigh that latency against the recall it buys.
A teammate proposes rebuilding the whole RAG stack on GraphRAG because "it's the state of the art." How do you respond as the senior in the room?
Hit these points: first ask what questions users actually ask — pull the distribution → if most are local lookups ("what is the limit for X?"), flat top-k or small-to-big already answers them at a fraction of the cost → GraphRAG earns its keep only on global, theme-spanning "connect-the-dots" questions, and it carries real cost: building the entity graph, generating community summaries, keeping it fresh on every reindex → propose it as an added retriever, routed to for global queries, not a rip-and-replace → name the trade-off: build and maintenance complexity vs a capability you only need for a slice of traffic.
Explain contextual retrieval and when its added cost is worth paying.
Hit these points: the failure it targets is chunks that are ambiguous on their own — a chunk stripped of its heading/section is hard to match and hard for the model to use → contextual retrieval (Anthropic) prepends a short chunk-situating context before embedding, so each chunk self-describes → it also adds a BM25 lexical index alongside the vectors, catching exact IDs and keywords that embeddings miss → together they cut retrieval failures versus naive embedding-only chunking → the cost: re-embedding every chunk with its generated context plus maintaining a dual index → worth it when chunks are short/fragmentary and exact-match terms matter; skip it when chunks are already self-contained.
Walk through how you match retrieval structure to the shape of the question across a mixed query workload.
Hit these points: the principle: the shape of the question should pick the structure of the retriever, and flat chunk + top-k is the floor, not the ceiling → local fact in one chunk → flat top-k or small-to-big (match small, return parent for context) → "which section, then what" over large docs → hierarchical / summary retrieval → recency / tenant / access scope → metadata pre-filter before vector search → precision-critical, latency-tolerant → multi-vector / ColBERT late interaction → global theme spread across the whole corpus → GraphRAG community summaries → operationalize with a router that classifies the query and dispatches; the staff move is not picking one structure but composing several and routing by question shape.
When is an agentic / corrective loop worth its latency and cost, and when is a static pipeline the right default?
Hit these points: the loop wins exactly one axis — quality / robustness — and pays for it in latency (multiple round-trips), cost (grading + re-tries), and complexity (loops, budgets, stop conditions) → default static for simple, well-covered questions where one pass already answers correctly → reach for CRAG-style grade-and-fallback on the long tail where the index comes up short → reach for Self-RAG reflection where an ungrounded answer is high-stakes → decide from eval data: measure static-pipeline accuracy first, add the loop only on the slice where correctness is high-stakes and static accuracy is provably insufficient → you also need a reliable grading signal or the loop just burns calls → always cap attempts and define a stop condition so it terminates.
Make the case that pre-retrieval is the highest-leverage place to spend — then steelman the opposite.
For: recall lost at retrieval is unrecoverable downstream — no reranker reorders a document that was never in the candidate set, and no bigger model reasons over text it never received; the cheapest place to fix recall is the query, and HyDE / multi-query are a few extra calls. Steelman against: if the right doc is being retrieved but buried or context-poor, the bottleneck is retrieval structure (small-to-big, reranking) or assembly, not the query; and if questions are global-theme, no query rewrite helps — you need GraphRAG. Synthesis: attribute the failure by class first — is the right doc absent (pre-retrieval), present but mis-ranked (structure/rerank), or non-local (graph)? — then spend on the lever that matches, starting with the query because it's the common cause and the cheapest.
Design-round framework — drive any advanced-RAG prompt through these, out loud:
  1. Characterize the question distribution — local facts vs global themes, single- vs multi-part, recency-sensitive.
  2. Pre-retrieval — rewrite / multi-query / HyDE / decompose / step-back, and route to the right source.
  3. Retrieval structure — match it to question shape: small-to-big, hierarchical, contextual, multi-vector, GraphRAG.
  4. Rerank & assemble within budget — precision first, parents for context, order for primacy/recency.
  5. Decide static vs loop — add CRAG grade-and-fallback / Self-RAG critique only where correctness justifies the cost.
  6. Bound the loop — cap attempts, define stop conditions, set a latency/cost budget.
  7. Observability & failure modes — log assembled context, measure retrieval hit-rate; handle missing / low-quality / stale / conflicting docs.
Design an advanced-RAG pipeline for a question type that flat top-k fails on — e.g. "summarize how our incident response process evolved over the last two years."
A strong answer covers: name why flat top-k fails — this is a global, multi-document synthesis question; no single chunk holds "how it evolved," so nearest-k returns scattered fragments → pre-retrieval: decompose into time-bucketed sub-questions and add metadata filtering on date so each retrieval is scoped → structure: layer GraphRAG or hierarchical summary retrieval so community/section summaries supply the cross-document themes, with small-to-big underneath for the specific facts that anchor each claim → assemble: rerank, then pack summaries plus a few grounding chunks within budget → consider an agentic loop that grades whether each time period is covered and re-retrieves the gaps before drafting → observability: log which periods and themes were retrieved; failure modes — missing year → flag the gap rather than smooth over it → name the trade-off: the graph/summary build and the loop add cost and latency, justified because flat retrieval simply can't answer this class.
Design retrieval for a corpus that must serve both local facts ("what's the retry limit?") and global synthesis ("what themes recur across all our incidents?").
A strong answer covers: start by recognizing two question shapes need two structures — don't force one retriever to do both → put a router up front that classifies the query as local-lookup vs global-theme → local path: small-to-big / flat top-k with metadata filtering — cheap, fast, one chunk answers it → global path: GraphRAG (or hierarchical summaries) so community summaries pre-aggregate relationships no single chunk holds → share one ingestion pipeline that produces both the chunk/parent index and the entity graph + summaries, so the corpus is indexed once into two structures → freshness: re-embed chunks and rebuild affected graph communities on change; the graph is the costlier thing to keep fresh, so name that → assemble within budget per path and rerank → observability: track hit-rate separately per path, since they fail differently → name the trade-off: maintaining a graph alongside the vector index is real cost and operational complexity, paid only because a meaningful slice of traffic is global synthesis that flat retrieval can't serve.
Sources
Anthropic, "Introducing Contextual Retrieval" — anthropic.com/news/contextual-retrieval. Chunk-situating context, hybrid embeddings + BM25, and why retrieval structure cuts failures. Also informs small-to-big / parent-document framing.
HyDE (Gao et al., 2022, "Precise Zero-Shot Dense Retrieval without Relevance Labels," arXiv:2212.10496), multi-query, step-back prompting, and query decomposition — established pre-retrieval / query-transformation techniques. Building on Lewis et al., 2020, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401), the RAG baseline this lesson extends.
Anthropic, "Building effective agents" — anthropic.com/engineering; and Claude Code documentation (docs.anthropic.com) on retrieval as a read-on-demand action inside an agent loop. Named self-correcting patterns: Self-RAG (Asai et al., 2023, arXiv:2310.11511) and Corrective RAG / CRAG (Yan et al., 2024, arXiv:2401.15884). Numbers in this lesson are illustrative.