What Is Context? Context Windows & Tokens for LLMs
Your bar: explain why an LLM cannot see your whole codebase, what actually fills the context window, and why context retrieval — not the model — is where modern AI systems win or lose.
Your bar: explain why an LLM cannot see your whole codebase, name what actually competes for the context window, and argue, against an interviewer pushing back, why context retrieval usually beats model choice. By the end you can answer the north-star question: why does one AI system feel far smarter than another when both run the same LLM? 1
The whole answer fits in one sentence. Each chip is one thing this lesson makes permanent:
An LLM only sees its context windowa fixed token budget, not your dataso the system must retrieve & assemble what to show itand that retrieval — not the model — decides how smart it looks
1
What Is Context?
Context = the model’s entire world for one call. It has no other access to your data — no DB connection, no filesystem, no memory of last time. Only what you pack into the window.
The window is a fixed budget — and you don’t own all of it
The four words, pinned down
| Term | Simple definition | Unit / shape | One-line analogy |
|---|---|---|---|
| Context | Everything the model can see for one call: prompt + history + retrieved data + tools + the question. | text | The model’s entire RAM at runtime. |
| Context window | The fixed maximum tokens per call (input + output together). | tokens (a hard ceiling) | Total installed RAM — physical limit. |
| Token | The sub-word chunk the model counts in (~4 chars / ~¾ of a word). | ~0.75 words | The byte the budget is measured in. |
| Working context | The curated subset you actually assemble for this turn — the relevant few, not all that exists. | text (selected) | The pages you put on the desk, not the whole library. |
Is Context is the complete set of text passed to the model for a single inference. The model is a pure function:
answer = f(context). Change nothing but the context and you change the answer.Why it exists Models are stateless and have no I/O. They can’t open a file, hit a DB, or recall yesterday. The only channel into the model is the window — so anything it must “know” has to be put there, every call.
Like (world) A consultant in a windowless room who answers one question from only the documents you slide under the door. Slide the wrong docs and you get a confident, wrong answer.
Like (code) A stateless request handler with no database connection — all inputs must arrive in the request body. The window is the request body, capped in size.
Why an LLM can’t just “see the whole codebase”
✗ “Bigger context windows make this problem go away.”
✓ They raise the ceiling, not the relevance. A 1M window still can’t hold a 10M-token system, costs more, and suffers position bias — selection still decides quality.
✗ “The model remembers our earlier files / last session.”
✓ It remembers nothing. If it isn’t re-packed into this call’s window, it does not exist to the model.
📥 Memory rule: The model only knows what’s in the window right now. Context = a fixed token budget you must spend wisely, not a pipe to your data.
Memory check
- What are the four budget tenants of the window besides retrieved context? → system prompt, tool/function definitions, conversation history, and the reserve for the answer
- Why can't a bigger window fully fix retrieval? → real systems exceed any window; bigger costs more + slower; models recall the middle of long contexts poorly (position bias)
- A token is roughly how many words? → ~¾ of an English word; ~4 characters
An engineer says “let’s just use the 1M-token model and paste the whole repo in.” Where is this wrong?
Hit these points: real repos exceed even 1M tokens, so you still choose a subset → cost and latency scale with tokens sent, so you’d pay 5× for mostly-irrelevant text → long-context recall is uneven — relevant lines in the middle get “lost in the middle” → noise lowers signal: irrelevant code actively degrades answers, it isn’t free padding → the right framing is “select the smallest sufficient context,” not “fit more in.”
2
Context Is the New Database Query
In classic software, the query fetches from the DB and the engine answers. In AI, the model is fixed — so the context-retrieval step is the new query, and that’s where the engineering value (and the bugs) moved.
The inversion that matters
| Classic software | AI system | |
|---|---|---|
| Fixed / commodity part | The query language (SQL is SQL) | The model (everyone can rent the same LLM) |
| Where the value lives | Schema + indexes + the DB’s data | Retrieval + selection + assembly of context |
| Your competitive moat | Your data & how you model it | Your data & how you retrieve and pack it |
| A “bad query” causes | Wrong rows / slow scan | Wrong answer — silently, with full confidence |
Why retrieval quality usually beats model quality
| answer quality | Right context | Wrong context |
|---|---|---|
| Best model | ✓✓ excellent | ✗ confidently wrong |
| Smaller model | ✓ surprisingly good | ✗ wrong |
Read the columns, not the rows. Moving across columns (context) swings the outcome from right to wrong. Moving down the rows (model) only nudges it. The column you control is context.
Is “Context as query” means the retrieval pipeline that selects and packs the window plays the exact role SQL plays against a database: it decides what the answering engine ever sees.
Why it exists The model is rentable and roughly the same for every competitor. So the differentiator shifts to the one thing you own and design: which facts reach the model, and how cleanly.
Like (world) Two lawyers, same law degree. The one who pulls the three relevant precedents wins; the one who dumps the whole library on the judge loses. Same brain, different brief.
Like (code) A read-model / query layer in CQRS: the store is generic, but the projection you build for a screen determines what the UI can show. Context retrieval is that projection for the LLM.
✗ “To get smarter answers, upgrade the model.”
✓ Past a baseline, upgrade the retrieval. Right context on a mid model usually beats wrong context on the flagship — and costs less.
✗ “A wrong answer means the model is dumb.”
✓ Usually it means the wrong context arrived. The failure is silent: the model can’t tell you it got the wrong files — it just answers from what it has.
📥 Memory rule: Context retrieval is the new SQL query. The model is the commodity engine; what you feed it is your product.
Memory check
- In the pipeline mapping, SQL query corresponds to what? → the context-retrieval step (it decides what the answering engine sees)
- Why is a context bug more dangerous than a SQL bug? → it fails silently — the model answers confidently from wrong input instead of erroring
- One sentence: why does retrieval often beat model choice? → the model only reasons over what it's given; the model is rentable and shared, the context is yours to design
Your AI feature gives wrong answers. A teammate wants to swap to the most expensive model. How do you push back as the senior in the room?
Hit these points: first ask “is this a retrieval problem or a reasoning problem?” → instrument what context was actually sent for the failing cases — usually the right document wasn’t retrieved, so a bigger model just reasons better over the wrong input → “right context + cheaper model often beats wrong context + flagship,” and it’s cheaper and faster → only after retrieval is verified good is model capability the bottleneck worth paying for → name the trade-off: model upgrade is a flat tax on every call; fixing retrieval compounds across every feature.
Frame “context is the new database query” for a VP who knows backend but not LLMs.
Hit these points: in our stack the DB is the source of truth and the SQL query decides what a request sees → with LLMs the model is a rented, commodity engine — the same one our competitors use → so the leverage moves to the “query” that feeds it: which of our data we retrieve and how we pack it into a fixed token budget → that pipeline is where our differentiation, our cost, and most of our bugs now live → therefore we invest in retrieval & context quality the way we’d invest in schema and indexing, not in chasing model versions.
Retrieval practice — test the two modules
Q1. "Context" for an LLM call is best described as…
- A persistent connection the model keeps open to your application's primary database
- The model's long-term memory of every conversation it has previously had with users
- The complete set of text passed in for one call — the model's entire world for that inference
- A cache of recent answers the model replays when it sees a similar question again later
Q2. Besides retrieved data, what else competes for the same fixed window budget?
- System prompt, tool definitions, conversation history, and reserved room for the answer
- The model's weights, the GPU memory, the network buffer, and the API rate limiter
- The embedding index, the vector database, the reranker, and the chunk overlap setting
- Only the user's current question — everything else is stored outside the window entirely
Q3. A bigger context window does NOT fully solve retrieval because…
- Larger windows are only available on open-source models that teams cannot deploy in production
- The model refuses to read more than a few thousand tokens regardless of the window size
- Token budgets are a billing concept only and have no effect on the quality of the answer
- Real systems still exceed it, cost and latency rise, and middle-of-context recall degrades
Q4. In "context is the new database query," the context-retrieval step plays the role of…
- The database engine itself — the durable store that holds the source-of-truth records
- The SQL query — it decides what the answering engine ever gets to see for this request
- The connection pool — it throttles how many requests reach the model concurrently
- The response cache — it stores prior model outputs so identical questions skip the model
Q5. Two products use the identical LLM, yet one feels far smarter. The most likely reason is…
- The smarter one secretly fine-tuned the model on a large private proprietary dataset
- The smarter one pays for priority API access that returns higher-quality model responses
- The smarter one retrieves and assembles better context into the window before each call
- The smarter one simply uses a much larger context window for every single request made
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
Define context, context window, token, and working context, and how they relate.
Hit these points: token = the unit (~¾ word) everything is counted in → context window = the fixed max tokens per call, a hard ceiling like total RAM → context = the actual text you pass for one call (prompt + history + retrieved + tools + question) → working context = the curated subset you assemble for this turn, the relevant few not all that exists → relationship: working context is what you choose to spend the window's budget on, and context engineering is the act of choosing it.
Besides the documents you retrieve, what else competes for the context window?
Hit these points: the system prompt and instructions → the tool / function definitions (grow with the number of tools) → the running conversation history (grows every turn — the quiet budget eater) → reserved space for the model's own answer → only what's left is yours for retrieved context, which is why selection, not window size, is the real constraint.
What is a token, and why are both the window limit and the bill measured in tokens rather than words or characters?
Hit these points: a token is the sub-word chunk the model actually processes — roughly ¾ of an English word, ~4 characters (illustrative) → the model never sees raw text; the tokenizer splits input into tokens and the model reads and emits those → so the natural unit for "how much fits" is tokens, which is why the window is a token ceiling, not a character or word count → cost follows the same unit because you pay per token of input and output processed → practical consequence: code, JSON, and rare words tokenize less efficiently than plain prose, so a "small" file can spend more budget than you'd guess from its character count.
What's the difference between the context window and the working context?
Hit these points: the context window is the fixed physical ceiling — the max tokens the model accepts for one call, the same for everyone on that model → the working context is the curated subset you actually assemble for this specific turn: the relevant few documents, not everything that exists → the window is capacity; the working context is the choice you make inside it → you can have a huge window and a terrible working context (dump the whole repo) or a small window and an excellent one (the three right files) → context engineering is the work of building a good working context, and the window only bounds how much room you have to do it.
Why can't an LLM "just read" a 5-million-line codebase to answer a question about it?
Hit these points: the model has no I/O — it can't open files; it only sees the window → the codebase is orders of magnitude larger than any window → even the largest windows cost more, run slower, and recall the middle poorly → so the system must select a small relevant subset (files, symbols, dependencies) and pack it → the problem is "which slice," not "read it all" → this is why Cursor and Claude Code build retrieval rather than leaning on window size.
A context bug and a SQL bug both return wrong data. Why is the context bug operationally scarier?
Hit these points: a SQL bug usually surfaces loudly — wrong rows, an error, a slow query you can profile → a context bug is silent: the model gives a fluent, confident answer from the wrong input and nothing throws → there's no stack trace pointing at "wrong document retrieved" → so you have to add the observability a DB gives you for free: log exactly what context was assembled per call, make retrieval auditable, and track retrieval hit-rate, not just end-answer vibes.
"Just use a bigger window and the problem goes away." Why isn't that a real fix?
Hit these points: a bigger window raises the ceiling, not the relevance — selection still decides what the model reasons over → cost and latency scale with tokens sent, so a 5× bigger context is roughly a 5× bigger bill and a slower response on every call (numbers illustrative) → recall is uneven across a long window: facts in the middle get "lost in the middle" while the model over-weights the start and end → irrelevant text is not free padding — it dilutes signal and actively degrades the answer → and real corpora still exceed any window, so you're choosing a subset regardless → the fix is a better working context, not more raw capacity; reserve the big window for genuinely large single inputs.
An answer is wrong. How do you tell whether the context was wrong versus the model being incapable?
Hit these points: reproduce the failing case and dump exactly what context was assembled and sent — if the needed fact isn't in the window, it's a retrieval/context problem and a bigger model won't help → control test: paste the correct context in by hand and re-ask the same model; if it now answers correctly, retrieval was the cause, not reasoning → conversely, if the right context is present and it still fails, that points at a reasoning/capability ceiling (multi-step logic, long synthesis) → check for "lost in the middle" — try moving the relevant chunk to the start/end before blaming the model → the durable version is instrumentation: log assembled context per call and track retrieval hit-rate so you can attribute failures by class instead of guessing.
Make the case that retrieval quality matters more than model quality — then steelman the opposite.
For: the model only reasons over what it's given; wrong context → wrong answer at any model size; the model is a shared commodity while context is your design; retrieval fixes compound across features and cost less than model upgrades. Steelman against: below a capability floor a weak model can't reason over even perfect context (multi-step logic, long synthesis), so there model choice is the binding constraint. Synthesis: fix retrieval first — it's the common cause and the cheapest lever; pay for model capability once retrieval is verified good and reasoning is provably the bottleneck.
A teammate wants to "fix" wrong answers by switching to a 1M-token model and pasting the whole repo in. Where does this break at scale?
Hit these points: real repos still exceed even 1M tokens, so you're back to choosing a subset → cost and latency scale with tokens sent, so you pay several-fold for mostly-irrelevant text on every call → long-context recall is uneven — the relevant lines get lost in the middle → noise actively lowers answer quality; irrelevant code is not free padding → the durable fix is selection + retrieval evaluation, not a bigger window; reserve the large window for genuinely large single inputs, not as a substitute for retrieval.
A strong engineer wants to spend the quarter upgrading the model to fix quality. Make the principal-level call on model-vs-retrieval investment.
Hit these points: don't decide by opinion — decide by evidence: instrument the failing cases and attribute them by class (wrong context retrieved vs right context but bad reasoning) before committing a quarter → in most products the dominant cause is retrieval, not reasoning, so a model upgrade pays a flat tax on every call while leaving the real defect in place → framing the trade-off: a model swap is a recurring per-token cost increase with a one-time quality bump that any competitor can also rent; retrieval investment is an asset you own that compounds across every feature and lowers cost → sequence it: fix and evaluate retrieval first because it's the common cause and cheapest lever, then — only if data shows reasoning is the binding constraint — spend on model capability with a clear before/after metric → redirect the engineer's energy into retrieval evals and observability, which also tells you when the model genuinely is the bottleneck → the principal move is making it a measured decision with a kill criterion, not a quarter-long bet on a hunch.
Design-round framework — drive any context-system prompt through these, out loud:
- Clarify scale: window size vs total corpus tokens, latency & cost budget.
- Candidate generation — lexical (BM25) + semantic (embeddings), hybrid.
- Rank, then re-rank the top-N for precision.
- Assemble within budget — signatures over full bodies; order for primacy/recency.
- Freshness — index invalidation; fetch volatile/exact data live, don't embed it.
- Observability — log the assembled context per call; measure retrieval hit-rate, not just answers.
- Failure modes & guardrails — missing / wrong / stale / conflicting / excessive.
Design the context-retrieval layer for a coding assistant over a 2M-token monorepo with a 200k window.
A strong answer covers: you can show ~10% of the repo at most, so retrieval is the product → index symbols + a dependency graph, not just flat text chunks → on a query, gather candidates by exact symbol match (lexical) and semantic similarity, then walk imports/call-sites for structural context → re-rank, then pack within budget: signatures and docstrings over full bodies, the files named in the query first → reserve fixed budget for history + the answer → keep the index fresh (re-embed/ctags on change) or read on demand for always-fresh precision → instrument what was retrieved so you can measure and improve hit-rate → name the trade-off: pre-indexed embeddings (fast, can go stale) vs read-on-demand (fresh, more round-trips).
Design the context assembly for a customer-support assistant — knowledge base plus per-customer history — under a fixed token budget.
A strong answer covers: start from the budget — partition the window into fixed slices (system prompt & policy, KB passages, customer history, the current question, reserved answer) so no source can starve the others → two retrieval sources with different shapes: the KB is semi-static, so pre-embed it and retrieve top-N passages by hybrid lexical + semantic match on the question; the customer's history (tickets, orders, entitlements) is volatile and account-scoped, so fetch it live by customer ID rather than embedding it → re-rank candidates and pack the highest-precision few — summarize or truncate older history rather than dumping the full transcript → treat exact, high-stakes facts (current plan, balance, open-ticket status) as live look-ups, never stale embeddings → freshness & correctness: invalidate KB chunks on article edits; scope every retrieval to the authenticated customer so you never leak another account's data → observability: log the assembled context per call and measure KB hit-rate and "had-the-right-fact" rate, not just CSAT → failure modes: missing KB article → say "I don't know / escalate" rather than hallucinate; conflicting KB vs history → prefer the customer-specific fact; budget overflow → drop lowest-ranked KB passages before touching identity/policy → name the trade-off: pre-indexed KB (fast, can lag edits) vs live history (fresh, exact, more latency), and the privacy line that history must always be retrieved per-customer.