RAG, Chunking & Codebase Context Explained
Your bar: explain how Cursor and Claude Code understand large repos, why chunking exists and how it fails, and how RAG works from first principles — and why most RAG bugs are retrieval bugs.
Your bar: explain how Cursor and Claude Code make sense of a repo too large for any window, why chunking exists and how it breaks, and how RAG works from first principles. Then argue, against an interviewer pushing back, why most “the LLM is dumb” bugs are retrieval bugs. By the end you can answer:
given a 2M-token codebase and a 200k window, how does the system surface the right symbols — and where does that pipeline silently break?1This lesson is the productization of Lesson 1’s “context is the new database query.” Three steps build the query engine:
Codebase Context
A repo is far bigger than any window. So the tool must discover the relevant files and symbols, follow their dependencies, and assemble the smallest sufficient slice — the way a human navigates an unfamiliar codebase.
Two strategies, side by side
The three discovery steps, pinned down
| Step | What it finds | Classic-tooling analogue |
|---|---|---|
| File discovery | Which files are even relevant (by path, glob, directory tree). |
|
| Symbol discovery | Definitions & references of the functions/classes in play. | An LSP index or |
| Dependency traversal | What those symbols import & call — the surrounding graph. | The import / call graph. |
| Context assembly | Pack the slice within budget — prefer signatures over full bodies. | Building a query result set. |
Is Codebase context is the slice of a repo a tool surfaces for one task: the right files, their symbols, and the dependencies that make them make sense — assembled within the token budget.
Why it exists The repo is ~10× any window1, so the tool can never load it all. It must reproduce, automatically, how a human finds the few relevant places before reading.
Like (world) A new hire who can’t read the whole wiki — they search, open the few right pages, follow links between them, then answer. Pre-index = they read a summary deck first; read-on-demand = they search fresh each time.
Like (code) A search index vs a live SELECT: a materialized view (embeddings)
is fast but can lag the source; querying the table directly (grep) is always current but slower
per hit.
✓ Cursor leans on a precomputed embedding index; Claude Code searches the live filesystem with tools. Different freshness/latency trade-offs, same end goal: the right symbols in budget.
✓ Embeddings shine for fuzzy/semantic recall, but can miss an exact identifier that lexical search nails — and a stale index returns deleted code. Each tool wins different queries.
📥 Memory rule: Pre-index = fast but can go stale; read-on-demand = fresh but costs round-trips. Same goal: surface the right symbols within the budget.
Memory check
- Name the three discovery steps before assembly. → file discovery, symbol discovery (definitions/references), dependency traversal (imports/call graph)
- One downside each of pre-index vs read-on-demand? → pre-index can go stale / miss exact IDs; read-on-demand pays in tool round-trips (latency + tokens)
- Why prefer signatures over full bodies when assembling? → they convey the interface at a fraction of the tokens, leaving budget for more relevant code and the answer
Your team’s embedding-indexed code assistant keeps citing a function that was deleted last week. What’s happening and how do you fix it?
Hit these points: a pre-built embedding index is a snapshot — if it isn’t reindexed on change, it serves stale vectors pointing at code that no longer exists → the fix is incremental reindexing on commit/save, plus a freshness check or TTL on the index → pair embeddings with a lexical/grep pass so exact current identifiers are verified against the live tree → or move latency-tolerant queries to read-on-demand so they always hit fresh files → root cause is index/source skew, not the model.
Chunking
You can’t embed or retrieve a whole file — it’s too big and too coarse. Chunking splits documents into pieces that each fit the embedder and are granular enough to return as a precise hit. How you cut decides what you can ever retrieve.
The size trade-off
Overlap & the fragmentation failure
Common mistakes → the better move
| Common mistake | Why it hurts | Better |
|---|---|---|
| Fixed-size character splitting | Cuts mid-function / mid-sentence; chunks are incoherent. | Structure-aware: split on function / class / heading bounds. |
| Zero overlap | Boundary-spanning facts fragment and become unretrievable. | Modest overlap so cross-boundary facts survive. |
| Ignoring document structure | A chunk lacks the heading/section it belongs to; ambiguous alone. | Contextual chunking: prepend a short doc/section summary to each chunk so it self-describes. |
Is Chunking is the rule for splitting a document into retrievable units. A good chunk is complete (answers on its own) and retrievable (its embedding matches the questions it should serve).
Why it exists Files exceed embedding input limits, and retrieval needs granularity — you want the relevant passage back, not a whole file. Chunking is the unit of both storage and recall.
Like (world) Index cards from a book. Cut them on chapter/section lines and each card stands alone; cut every 500 letters and you get cards that start mid-sentence and answer nothing.
Like (code) Choosing a primary key / row grain in a table: too coarse and a query returns bloated rows; too fine and you must join many rows to answer. Chunk grain is the retrieval grain.
✓ Fixed-size splits cut mid-thought and fragment facts. Split on meaning (function/heading bounds) and add overlap so each chunk is complete and retrievable.
✓ Too small strips the surrounding context that made the passage meaningful — and multiplies fragments. Precision needs the right grain plus enough context, not minimum size.
📥 Memory rule: Chunk on meaning, not character count. A chunk should be retrievable and complete on its own.
Memory check
- What does overlap protect against? → context fragmentation — a fact spanning a boundary that would otherwise be split across two chunks and retrieved by neither
- Too-big vs too-small chunks fail how? → too big = diluted/averaged embedding, imprecise hits, wasted tokens; too small = lost surrounding context, many fragments
- What is contextual chunking? → prepend a short summary of the doc/section to each chunk so it's self-describing and matches better (Anthropic Contextual Retrieval)
A RAG bot answers most questions but always whiffs on “what is the rate limit?” even though the docs clearly state it. Where do you look first?
Hit these points: suspect context fragmentation — the fact (“limit is set to 5/min”) likely straddles a chunk boundary, so neither chunk is a complete, retrievable answer → inspect the actual chunks around that sentence, not the model output → fixes: add overlap so the fact survives whole; switch to structure-aware splitting so the sentence isn’t cut; apply contextual chunking so the chunk carries its section heading → verify by checking whether the right chunk now appears in top-k for that query — a retrieval check, not an answer check.
RAG — From First Principles
Retrieval-Augmented Generation is Lesson 1’s idea, shipped: take a question, retrieve the relevant chunks, assemble them into the window, and let the LLM answer — grounded in your data instead of its frozen weights.
The pipeline
Why RAG exists vs the alternatives
| Need | Stuff it all in the window | Fine-tune the model | RAG |
|---|---|---|---|
| Private / current data | Doesn’t fit; truncated | Stale at next data change | ✓ fresh, pulled per query |
| Citations / provenance | Buried in noise | ✗ weights can’t cite | ✓ returns the source chunk |
| Cost to update | Pay per token every call | Retrain run each time | ✓ reindex changed docs only |
Is RAG is the pattern of retrieving relevant external text at query time and putting it in the context window so the model answers from it — the productized form of “context is the new database query.”
Why it exists To ground the model in private, current, authoritative data; cut hallucination; enable citations; and do it cheaper and fresher than fine-tuning — without stuffing everything into a fixed window.
Like (world) An open-book exam. The student (LLM) is fixed, but pulling the right page before answering changes the grade. RAG is the “go fetch the right page” step.
Like (code) A read-through cache in front of a database: on each request you fetch the few relevant records and hand them to the handler. Retrieval is that fetch; the LLM is the handler.
Why so many RAG systems are poor
| The naive default | What it costs | The fix (maps to Module 2) |
|---|---|---|
| Fixed-size chunking | Fragmented, incoherent chunks | Structure-aware + contextual chunking |
| Embedding-only retrieval | Misses exact keywords/IDs | Hybrid: embeddings + BM25 lexical |
| Top-k, no reranking | Right doc buried below the cut | Rerank so the best chunk rises to top |
| No retrieval evaluation | You measure the answer, never the recall | Eval retrieval hit-rate, not just vibes |
Read the middle column. Every failure is upstream of the model: bad chunks, blind retrieval, no rerank, no measurement. Teams grade the answer and never check whether the right document was even retrieved — so they “fix the LLM” and nothing improves.
✓ Usually retrieval missed: the model got nothing or the wrong passage and answered confidently anyway. Garbage in → fluent garbage out. Fix recall before model.
✓ That’s the naive baseline. Without hybrid search and reranking, the relevant doc often sits below the cut (and “lost in the middle” even when included) — so it never reaches the model usefully.
📥 Memory rule: RAG is only as good as its retrieval. Most “the LLM is dumb” bugs are retrieval bugs.
Memory check
- Name the five stages of the RAG pipeline. → Question → Retrieval → Context assembly → LLM → Answer
- Why RAG over fine-tuning for current/private data? → fresh per query, can cite the source chunk, cheaper to update (reindex changed docs vs retrain), no stuffing the window
- Three reasons real RAG systems disappoint? → naive fixed-size chunking, embedding-only retrieval with no rerank, and no evaluation of retrieval quality (they measure the answer, not recall)
Sketch a docs-Q&A or support assistant as RAG, then name the three places it most often breaks in production.
Hit these points: pipeline — chunk the help center / docs, embed + index, then per question retrieve top chunks, assemble into the window with the question, LLM answers with citations → break 1: chunking destroyed the fact (fragmentation / fixed-size cuts) → break 2: retrieval missed — embedding-only with no hybrid/rerank, so the right doc is buried below top-k or lost in the middle → break 3: stale index — the doc changed but wasn’t reindexed, or conflicting passages confuse the model → in all three the model is innocent; the fix is upstream, and you must measure retrieval hit-rate, not just answer quality.
Retrieval practice — test the three modules
Q1. The key difference between Cursor's and Claude Code's codebase strategy is that…
- Cursor reads files live with grep while Claude Code precomputes one big static embedding index
- Both rely on the same persistent vector index and differ only in which LLM they each call
- Claude Code caches every answer it gives while Cursor recomputes the answer from scratch each time
- Cursor pre-indexes embeddings for semantic recall; Claude Code reads the live filesystem on demand
Q2. A known downside of a pre-built embedding index for code is that…
- It can go stale and miss exact identifiers, returning code that may no longer exist in the repo
- It is unable to search files larger than a single screen of code without a special configuration
- It always requires a paid GPU at query time and cannot be run on standard developer laptops
- It can only index one programming language per repository and ignores all the other source files
Q3. "Context fragmentation" in chunking refers to…
- Storing chunks across many database shards so a single query must contact every shard at once
- The model forgetting earlier chunks of a long conversation once newer turns push them out
- A fact split across a chunk boundary so no single chunk is complete and retrieval misses it
- Compressing each chunk so aggressively that its embedding vector no longer matches any query
Q4. The five-stage RAG pipeline, in order, is…
- Question → LLM → Retrieval → Context assembly → Answer returned to the calling user
- Question → Retrieval → Context assembly → LLM → Answer returned to the calling user
- Retrieval → Question → LLM → Context assembly → Answer returned to the calling user
- Question → Context assembly → Retrieval → LLM → Answer returned to the calling user
Q5. A RAG bot confidently gives a wrong answer. The most likely root cause is…
- The base model simply lacks the reasoning ability and must be replaced by a far larger one
- The token budget was exceeded so the API silently refused to generate any answer at all here
- Retrieval surfaced the wrong passage or nothing, and the model answered from bad input anyway
- The user phrased the question politely, which biased the model toward an agreeable wrong reply
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
What are the three discovery steps a tool runs before it assembles a codebase-context window?
Contrast Cursor's and Claude Code's strategy for understanding a large repo.
What is chunking, and what makes one chunk "good"?
Name the five stages of the RAG pipeline in order, and say what RAG is.
Your embedding-indexed assistant keeps citing a function deleted last week. What's wrong and how do you fix it?
Walk through chunk-size trade-offs, and explain how zero-overlap fixed-size splitting silently loses facts.
A RAG bot answers most questions but always whiffs on "what is the rate limit?" though the docs state it. Where do you look first?
An embedding-only RAG returns the right document for paraphrased questions but misses when users type an exact error code. Why, and what do you add?
At scale, how do you decide between a pre-built embedding index and read-on-demand — and when do you combine them?
A team says "our RAG is bad, the model hallucinates — we need a smarter model." Diagnose this as the senior in the room.
Make the case that retrieval quality matters more than model quality for a code assistant — then steelman the opposite.
- Clarify scale: corpus size vs window, query volume, freshness needs, latency & cost budget.
- Ingest & chunk: structure-aware splitting + overlap; contextual chunking so each chunk self-describes.
- Candidate generation: hybrid — semantic (embeddings) + lexical (BM25) for exact IDs/keywords.
- Rerank the top-N for precision; assemble within budget, ordered against lost-in-the-middle.
- Freshness: incremental reindex on change, TTL/invalidation, or read volatile/exact data live.
- Evaluation: measure retrieval hit-rate / recall@k as the gate, not just answer vibes.
- Failure modes & guardrails: retrieval miss, fragmentation, stale index, conflicting passages, no citation.