Reviewing an AI Feature Like a Distributed System
Review an AI feature the way you review any production service: reliability, scalability, observability — with one new variable, non-determinism, changing every lens.
Your bar: walk into a design review for an AI feature and run it through the same lenses you’d run on any production service — reliability, scalability, availability, observability, cost, failure modes — then name the one new variable AI adds, non-determinism, and show exactly how it bends each lens. By the end you can answer the question a VP actually asks: what happens when the model is slow, wrong, rate-limited, or down — and can you see it? 1
An AI feature is not a new kind of system. It’s a service calling a remote dependency. Four moves frame the review:
Reliability
For a normal service, reliable means “it responded, within budget.” For an AI feature you carry a second axis: was the response correct-enough? A confident wrong answer is an outage the health check never sees.2
Two axes, not one — availability AND quality
The graceful-degradation ladder — shed quality before availability
Is Reliability for AI = the probability of a useful, correct-enough answer within budget, under real load and partial failure. Two axes: availability/latency and quality.
Why it exists The model can be “up” yet slow, rate-limited, or confidently wrong. Each of those is an outage to the user even though the endpoint returns 200.
Like (world) A specialist who’s always at their desk but sometimes gives wrong advice. “Available” isn’t the bar; “reliably useful” is.
Like (code) SRE’s SLI/SLO/error-budget discipline — but you spend a budget on two axes, latency and correctness, not one.
📥 Memory rule: Reliable AI means useful and correct-enough, within budget — not “the model is up.” Set an SLO on quality, not just availability.
Memory check
- What are the two axes of AI reliability? → availability/latency (responded in budget) and quality (correct-enough)
- Why is "200 OK" insufficient? → a fast, confident wrong answer passes the health check but harms the user — quality is invisible to availability monitoring
- What's the degradation ladder? → fallback model → shorter context → cached/templated answer → honest decline — shed quality before availability
M1 — What does “reliability” mean for an AI feature, and why isn’t “200 OK” enough?
Hit these points: reliability for AI = the probability of a useful, correct-enough answer within budget under real load and partial failure — not “the model process is up” → a model that returns a confident wrong answer, or hangs for 30s, or is rate-limited, is an outage to the user even though the endpoint is technically healthy → so you carry two axes: availability (did it respond in budget) and quality (was the response correct/grounded/safe) → set an SLO on both, spend an error budget against both, and degrade gracefully — shorter context, cheaper fallback model, cached/templated answer, or an honest “can’t do that right now” — rather than hanging or shipping a wrong answer.
Scalability
A normal service scales in requests per second against your own pool. An AI feature scales in tokens per second against a provider rate limit you don’t control — and the cost of one request grows with how much it reads and writes.3
The capacity unit is the token, and the ceiling is upstream
What changes vs a normal stateless service
| Dimension | Normal service | AI feature |
|---|---|---|
| Capacity unit | Requests / sec | Tokens / sec (input + output) |
| The ceiling | Your own pool / CPU | A provider rate limit you don’t control |
| Per-request cost | Roughly flat | Scales with input + output length |
| Self-hosted bottleneck | CPU / connections | KV-cache memory (context len × concurrency) |
| Saturation symptom | CPU pegged, queue grows | Provider throttling, latency cliff |
Is Inference capacity planning = sizing for tokens/sec and concurrent requests against a provider rate limit (or self-hosted GPU memory), with queueing, batching, and admission control.
Why it exists The binding constraint moved off your box. You bill and provision in tokens; the upstream limit and output length, not your CPU, decide when you saturate.
Like (world) A toll booth with a fixed cars-per-minute throughput. You can’t widen it; you meter who enters and queue the rest so nobody gridlocks.
Like (code) A bounded connection pool against a downstream you don’t own — admission control and back-pressure keep one fat query from starving the rest.
📥 Memory rule: Provision and bill in tokens, not requests; the ceiling is an upstream limit you don’t own. Add admission control before the wall, not after the cliff.
Memory check
- What's the capacity unit for an AI feature? → the token (input + output) against a provider rate limit, not RPS against your own pool
- Why won't autoscaling pods fix saturation? → the ceiling is the provider's token limit, not your CPU; more pods don't raise it — you must queue, shed, batch, or fall back
- What's the self-hosted bottleneck? → KV-cache memory, which grows with context length × concurrency — long-context requests starve short ones
M2 — How is capacity planning different for an AI feature than for a normal stateless service?
Hit these points: the capacity unit is the token, not the request — you provision and bill in input+output tokens the way you provision CPU and bill in RPS → the ceiling is usually a provider rate limit (tokens/min) you don’t control, not your own thread pool → on self-hosted GPUs the binding constraint is KV-cache memory, which grows with context length × concurrency, so a few long-context requests can starve many short ones → a long output costs latency and money linearly, and queueing plus admission control decide whether you degrade gracefully or fall off a latency cliff → model it as a queue with a token budget and a hard upstream limit, and add load shedding before you hit it.
Observability
Logs, metrics, and traces still apply — but AI adds a fourth axis classic systems don’t have: a quality signal (was the output correct, grounded, safe?). And you must be able to reconstruct, after the fact, exactly what one request saw and did.4
Three signals + a quality axis + auditability
Why this lens is heavier for AI
| Question to answer | Normal service | AI feature |
|---|---|---|
| Is it up / fast? | Metrics + traces | Same — plus token & cost metrics |
| Was it correct? | Mostly implied by 200 | A separate eval/quality signal — the new axis |
| What did it do & why? | Request log | Full call tree: prompt, context, model+version, tools |
| Did behavior drift? | Rare (you ship code) | Common — provider can update the model under you |
Is AI observability = answering “what did the system do and why” from telemetry across the trace/span call tree, an eval signal for quality, and cost/usage — logs/metrics/traces plus a quality axis.
Why it exists The output can be wrong while every classic signal looks green. Without a quality signal and per-request auditability you can’t detect or debug a wrong answer.
Like (world) A flight recorder: you don’t just log that the plane landed — you can replay every input and decision to explain a bad outcome afterward.
Like (code) Distributed tracing with a correctness assertion bolted on — the span tree shows what ran; the eval shows whether the result was right.
📥 Memory rule: Logs, metrics, traces plus a quality axis. If you can’t reconstruct what a request saw and whether it was right, it isn’t observable.
Memory check
- What's the fourth axis AI adds? → a quality/eval signal — was the output correct, grounded, safe — scored offline on a golden set and sampled online
- What must auditability let you reconstruct? → the prompt, retrieved context, model+version, tool calls, and output for a given request id, retained
- Why prefer OpenTelemetry GenAI conventions? → a standard attribute set (model, token counts, tool calls) keeps telemetry from being vendor-locked
M3 — What’s the minimum observability you’d require before letting an AI feature into production?
Hit these points: classic three signals plus a fourth axis → traces: the full agent-loop call tree — model calls, retrievals, tool calls, latency, token counts per span, ideally on the OpenTelemetry GenAI conventions so you’re not vendor-locked → metrics: latency (with tails), error and rate-limit rates, tokens/request, cost/request → the quality axis: an eval signal scoring whether output was correct, grounded, and safe, offline on a golden set and sampled online → auditability: reconstruct exactly what a request saw and did — prompt, retrieved context, model+version, tool calls, output — tied to a request id and retained → if you can’t answer “what did it do for ticket #4471 and was it right,” you’re not ready.
★ Principal-engineer view: Resist the urge to invent “AI architecture.” You already own the playbook — SLOs, error budgets, graceful degradation, back-pressure, tracing. Run it. Then spend your scrutiny on the three deltas non-determinism creates: correctness becomes part of reliability, tokens become the capacity unit, and output quality becomes a monitored signal that can drift. The provider and its token bill are now in your critical path — review them like any other hard dependency, with a fallback and a kill switch. 5
This review is the methodology the whole track runs on. For where these concerns land inside a multi-agent runtime, see AI Agents · Lesson 4 — Agent systems & production; for the context-pipeline review angle, seeContext Engineering · Lesson 5 — Architecture reviews & production design. This track is the dedicated production-review methodology.
Retrieval practice — test the three lenses
Q1. What is the single new variable an AI feature adds over a normal remote-dependency service?
- It requires far more compute, so the only real change is provisioning bigger machines
- It always responds slower, so latency budgets simply need to be set much higher
- The dependency is non-deterministic, so the same input can return a wrong output
- It cannot be cached at all, so every single request must be recomputed from scratch
Q2. An AI endpoint returns 200 with a confident but wrong answer. How does the reliability lens treat it?
- As an outage on the quality axis — availability looks fine but the user was harmed
- As a healthy response, because returning a 200 status proves the system is reliable
- As a capacity event, since wrong answers are caused only by hitting the token limit
- As a billing problem, because correctness is purely a function of how much you spend
Q3. Your AI traffic is unchanged but requests start getting throttled. What is the binding constraint?
- Your own CPU is pegged, so adding more application pods will immediately resolve it
- The database connection pool is exhausted by the volume of writes from the feature
- The network egress quota was hit, which only ever affects outbound file transfers
- A provider token rate limit you don't control — more pods won't raise that ceiling
Q4. What does AI observability require that classic logs, metrics, and traces do not cover?
- A faster dashboard refresh rate so operators notice latency spikes a little sooner
- A quality signal scoring whether the output was correct, grounded, and safe to ship
- A second copy of every metric stored in a different region for disaster recovery
- A larger log retention window so old request logs are kept for a much longer time
Q5. The model is rate-limited at peak. What is the right reliability response?
- Hold every request open and wait, since a late correct answer is always acceptable
- Retry the same call immediately and repeatedly until the provider finally accepts it
- Degrade gracefully — fall back to a faster model, shorten context, or decline honestly
- Return a confident guess so the latency dashboard and health check both stay green
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
Why is an AI feature "just a distributed system," and what's the one thing that's different?
What does "reliability" mean for an AI feature, and why isn't "200 OK" enough?
What's the capacity unit for an AI feature, and where's the ceiling?
Name the fourth observability axis AI adds beyond logs, metrics, and traces.
Your AI feature's p99 latency is 9s and users are abandoning. Where do you look first?
Name three failure modes an AI dependency adds that a normal HTTP dependency doesn't, and contain each.
Why does the same input sometimes give a different output, and what does that change about review?
You inherit an AI feature with no SLOs. Define a reliability target and an error budget.
Walk me through how you'd review a proposed AI feature the way you review any new service.
A team says "it's just an API call to the model, treat it like any other microservice." Where's that right, where does it break?
- Frame it: a service calling a remote, rate-limited, non-deterministic dependency — run the usual lenses, then ask what non-determinism does to each.
- Scope & blast radius: who sees it, cost of a wrong vs slow vs down answer, is the model in the critical path or can it be async / advisory?
- Reliability: SLOs on availability AND quality; graceful-degradation ladder (fallback model → shorter context → cached/templated → honest decline); timeouts / retries / circuit breaker around the provider.
- Scalability: capacity in tokens/sec against the provider rate limit; queueing, admission control, output caps, per-tenant budgets; KV-cache limits if self-hosted.
- Observability: OTel GenAI traces, latency / cost / token metrics, an eval signal for quality, request-level auditability.
- Failure modes: hallucination, rate-limit, version drift, latency cliff, cost blowup — each with a containment.
- Cost & control: tokens/request, caching, model tiering, per-tenant metering, and a kill switch — then close on the VP's question.