Context System Architecture-Review Checklist

Architecture-review checklist — the ten questions for any AI design

Product teardown — name the strategy, the failure falls out

Product specifics are version-dependent / at time of writing — review the architecture, not the version.

Production design — match retrieval to the data type

The senior reflex: exact & financial → structured query; fuzzy & textual → hybrid + rerank; volatile → fetch live; long-running → compress + remember.

The five failure modes — symptom → cause → fix

ingest -> chunk -> index -> retrieve
      -> rerank -> assemble -> LLM -> log

MISSING      at retrieve   (recall gap)
WRONG        at rerank     (bad ranking)
OUTDATED     at index      (freshness)
CONFLICTING  at assemble   (no policy)
EXCESSIVE    at assemble   (over budget)

Each stage gets its own metric.
A wrong answer with the RIGHT context
= a reasoning bug.
With the WRONG context
= a retrieval bug.  Separate them.

Evaluation & observability — what to probe L6

• Is retrieval measured on its own? recall@k / hit-rate (did the right doc make the top-k at all) — not just end-answer quality. A high end-to-end score masks low recall on the hard slice; the model bluffs from parametric memory on easy queries.
• Reranker / order graded? precision@k, MRR, nDCG judge the final order on top of high recall — separate metric, separate stage.
• Answer half graded against the context? faithfulness / groundedness (every claim supported → catches hallucination), answer relevance, context precision/recall. A grounded answer over bad context is a retrieval bug; a hallucination over good context is a generation bug — don’t fix the wrong half.
• Versioned golden set + offline regression in CI? query · relevant doc IDs · ideal answer, replayed pre-deploy to catch regressions before users do — not “we evaluate by trying it.”
• Online signals tracked? thumbs up/down, deflection / escalation, citation-clicks, follow-up / rephrase rate — curated back into the golden set so the next offline run catches them.
• Is the assembled context logged & traceable per call? query (+ any rewrite), retrieved IDs+scores, packed chunks, final answer — so “the bot was wrong” becomes “doc X wasn’t retrieved.” A wrong answer with the RIGHT context is a reasoning bug; with the WRONG context it’s a retrieval bug.
• LLM-as-judge calibrated? if used to scale faithfulness/relevance grading — anchored to human labels on a sample, watched for judge bias (verbosity, position, self-preference). Use it to scale, not to define truth.

Pre-retrieval & advanced RAG — decision points L7

Red flag: “a better reranker will fix our low recall” — reranking only reorders what was already retrieved; a doc never in the candidate set can’t be surfaced. That’s a pre-retrieval problem.

Embeddings, indexing & cost — what to probe L8

• Is the ANN index choice justified? HNSW (graph — fast, high recall, memory-heavy) vs IVF (cluster — tune speed via #probes) vs PQ (compress — big memory savings, lossy → lower recall; often IVF+PQ) — picked against the recall ↔ latency ↔ memory triangle, not by default. Don’t reach for a vector DB before you need one.
• Is index freshness / invalidation handled? incremental upserts / CDC on edit, deletes on removal, invalidation on source change. A stale index is the outdated-context failure (L4), just upstream — monitor oldest-chunk age as an SLO so staleness is observable, not silent.
• Is volatile data fetched live, not embedded? prices / availability change constantly → TTL or fetch via a structured tool at query time; never bake fast-changing fields into the index.
• Is a re-embedding plan in place? changing the embedding model = re-embed the whole corpus behind a shadow index + cutover (old and new vectors are incomparable) — quantify tokens-per-chunk × chunk count × price, validate the win on an eval set first.
• Cost levers identified & measured? input tokens usually dominate → fewer-but-better chunks (also more accurate), trim the prompt, rerank only the top-n, cheaper / Matryoshka-truncated embedding model, batch corpus embedding, prompt caching of the stable prefix. Validate each cut against an eval set so you don’t trade accuracy for dollars.

Caching, ordering & security — what to probe L9

VP / leadership review questions

• Build vs buy? Buy when the data is generic and the vendor’s strategy fits; build when our differentiation is retrieval over proprietary data. The model is a commodity; the pipeline is the moat. grounding in live systems
• Retrieval investment vs a model upgrade? Retrieval fixes compound across every feature; a model upgrade is a flat tax on every call. Better retrieval lifts accuracy, cuts cost, and reduces hallucination at once.
• How do we measure retrieval quality? recall@k + rerank precision on a labeled set; production hit-rate; deflection/escalation for support, audit-pass for billing — tracked separately from end-answer accuracy.
• Cost levers? Cost scales with tokens sent — tighter selection + rerank, compress long sessions, trim history, cache hot retrievals. Right context on a cheaper model often beats wrong context on the flagship.
• Risk & compliance? PII in retrieved context → redaction + access control at retrieve; injection via poisoned content → guardrails before the model; auditability → log assembled context per call; finance → deterministic retrieval + no-synthesis rule.
• Upgrade the model, or fix retrieval? Instrument first — was the right context retrieved for the failing cases? If no → fix retrieval (cheaper, compounds). If yes and multi-step reasoning is the bottleneck → pay for model capability. Never upgrade to paper over a retrieval gap.
• How do we measure retrieval quality? recall@k / hit-rate on a versioned golden set, graded separately from the answer; reranker order via nDCG; production signals (deflection, citation-clicks, rephrase rate). If the only answer is “we try it,” there is no measurement. L6
• What’s our eval / observability maturity? Is there an offline regression gate in CI, online signals fed back into the golden set, and a per-call trace of the assembled context — so any wrong answer is debuggable to a stage (retrieval vs generation)? “We’ll add observability later” is the red flag.
• What’s our multi-tenant data-isolation guarantee? Access control enforced in the retrieval query (scoped by tenant_id / ACL at the index), never via a prompt instruction — the prompt is not a security boundary. Plus injection defense on untrusted retrieved content and PII handling per OWASP LLM Top 10. ACL in the query