dineshbyte
  • Lesson 1 of 1
  • 12 min read
  • Lesson

The AI Infra Stack End-to-End: Embeddings, Vector DBs & Serving

Trace one request through the AI infra stack: text to embeddings to vector DB (HNSW/IVF) to reranker to model gateway to inference server. Where latency, cost, and failure concentrate.

Your bar: trace one user query through the whole stack —

text → embedding → vector DB → reranker → model gateway → inference server

— and say, at each hop, what it does and where the latency, cost, and failure concentrate. By the end you can answer the foundational interview question:

what actually happens to a request between “user hits enter” and “tokens stream back,” and which hop do you instrument first?1

The stack is one straight line. Every hop transforms the request and hands it on:

Text becomes a vector (embedding model)the vector DB finds near neighbors via ANNa reranker sharpens precision on the top-Kthe gateway and inference server serve the answer
One request, every hopQuery textuser inputEmbeddingtext → vectorVector DBANN searchHNSW / IVFRerankercross-encoderre-score top-KGatewayauth · routecache · limitInference serverKV cache · continuous batchingGPU forward passResponse cachehit → skip inferenceTokens streamanswer to userWhere each pain concentratesLATENCYinference forward pass & queue-wait dominate; ANN search is fast unless over-tunedCOSTper-token inference; the gateway is the one place to see and cap total spendFAILUREANN recall gaps · gateway as blast-radius chokepoint · KV memory capping batch size
The whole track in one picture. Every later lesson zooms into one of these boxes — this lesson is the map: what each hop does, and where latency, cost, and failure live.1
1

The Stack Map

The request path is a pipeline of single-purpose hops. Each one transforms the request and hands it forward, and each one is where some specific kind of pain — latency, cost, or failure — concentrates.

The hop-by-hop request path

Each hop: one job, one riskEmbedding modeljob: text → vector · risk: model/version driftVector DB (ANN)job: find near neighbors · risk: recall vs latencyRerankerjob: precision on top-K · risk: can't recover missesModel gatewayjob: auth/route/cache/limit · risk: chokepointInference serverjob: forward pass · risk: KV memory caps batchFirst two stages select WHAT to send; the last two govern HOW it's served. Different SLOs, different teams.
Two halves: a retrieval half (embedding, vector DB, reranker) that decides whatgoes into the prompt, and a serving half (gateway, inference) that decides howthe answer is produced. This track is the ops deep dive on both.1

What each hop does — and what it costs you

HopIts one jobWhere the pain lands
Embedding modelTurn query text into a dense vectorA model/version swap re-indexes the whole corpus
Vector DB (ANN)Return the top-K nearest vectors fastRecall vs latency vs memory — the core ANN trade
RerankerRe-score the top-K for precisionFixed cost per candidate; cannot recover a missed doc
Model gatewayAuth, route, rate-limit, cache, accountCost & reliability chokepoint — the blast radius
Inference serverRun the GPU forward pass, stream tokensKV-cache memory caps batch size, caps throughput

Is The AI infra stack is the chain of services a request crosses from raw text to streamed answer: embed, retrieve (ANN), rerank, gateway, infer, cache.

Why it exists No single service does all of it well. Retrieval needs a vector index; serving needs GPUs and a control plane. Splitting the work lets each tier scale and fail independently.

Like (world) A library request: the catalog narrows millions of books to a shelf (retrieval), a librarian picks the best few (rerank), and a reading room serves them (inference). Each step has its own line and bottleneck.

Like (code) A classic web request path: DNS → LB → app → cache → DB. You profile per hop and optimize the one that dominates p95 — same discipline here.

✗ “The LLM is the system — everything else is glue.”

✓ The model is one hop. Most production latency, cost, and failure live in retrieval and the serving control plane around it, not inside the model call.

✗ “If it’s slow, it’s the model — make the prompt shorter.”

✓ Instrument per hop first. The dominant cost is often queue-wait at the inference server or an over-tuned ANN search, not the forward pass itself.

📥 Memory rule: The stack is a pipeline of single-purpose hops. Retrieval picks what; serving governs how — instrument every hop before you optimize one.

Memory check
  • Name the hops in order. → text → embedding → vector DB (ANN) → reranker → model gateway → inference server → response cache
  • Which half decides what's in the prompt vs how it's served? → embed/retrieve/rerank decide WHAT; gateway/inference decide HOW
  • Which hop usually dominates latency? → the inference forward pass (sequential, GPU-bound), with queue-wait next under load — but measure, don't assume
M1 — Walk me through what happens to one user query as it travels through a production RAG stack.

Hit these points: the query text is embedded into a vector by an embedding model → that vector hits the vector DB, which runs approximate nearest-neighbor (HNSW or IVF) over millions of stored vectors to return a top-K candidate set → an optional reranker (a cross-encoder) re-scores that small set jointly with the query for precision → the chosen passages plus the prompt go to the model gateway, which authenticates, rate-limits, routes, and may serve a response-cache hit → on a miss the inference server runs the forward pass with KV cache and continuous batching, streaming tokens back → name where each hop concentrates risk: ANN trades recall for latency, the gateway is the reliability and cost chokepoint, and the inference server is GPU-bound on KV-cache memory and batch occupancy.

2

Embeddings

An embedding is a dense vector that captures meaning: vectors close together mean similar things. You compute them once at ingest, then similarity at query time is just a distance computation.2

Bi-encoder: embed once, compare cheaply

Meaning becomes position; closeness becomes distanceAt ingest (once)documentsembedding modelvectors → indexedAt query (per request)the querysame modelone query vectorvector space"refund""money back"query"GPU spec"near = similar meaning · far = unrelated
A bi-encoder embeds each document independently, so the corpus is pre-computed and indexed offline. At query time you embed only the query and compare — that asymmetry is what makes vector search feasible at scale.2

Why pre-compute? The cost of the alternative

ApproachWhen the work happensCost per query
Bi-encoder (embed once)Documents at ingest; query at searchEmbed 1 query + a distance scan — cheap, scales
Cross-encoder (joint)Every query×doc pair, at query timeO(corpus) forward passes — accurate but unscalable
Hybrid (this stack)Bi-encoder first, cross-encoder on top-NCheap recall, then precision only where it pays

Is An embedding is a fixed-length dense vector representing a chunk of text such that semantic similarity maps to vector proximity (cosine or dot product).

Why it exists Keyword match misses meaning (“car” vs “automobile”). Embeddings let you search by what text means, and reduce search to a fast distance computation.

Like (world) Placing books by topic on a map, not alphabetically. Once shelved by meaning, finding “things like this one” is just looking nearby.

Like (code) A pre-built index vs a full table scan: you pay the indexing cost once at write time so reads stay cheap. The embedding model is the index function.

✗ “More dimensions always means better retrieval.”

✓ Dimensions cost storage, memory, and ANN latency at scale. Match the model’s dimension to your recall need and budget — bigger is not free and rarely linear in quality.

✗ “I can swap the embedding model anytime.”

✓ Query and corpus must share the same model. Swapping it means re-embedding and re-indexing the entire corpus — a migration, not a config change.

📥 Memory rule: Embeddings turn meaning into position. Pre-compute the corpus once; embed only the query at runtime — and never mix embedding models across query and index.

Memory check
  • What does proximity in embedding space mean? → semantic similarity — near vectors mean similar things, so similarity becomes a distance computation
  • Why is a bi-encoder used for first-stage retrieval? → it embeds each doc independently, so the corpus is pre-computed; query time is one embed + a cheap scan
  • What breaks if you change the embedding model? → query and corpus must match — a swap forces re-embedding and re-indexing the whole corpus
M2 — What is an embedding, and why do you pre-compute them at ingest instead of at query time?

Hit these points: an embedding is a dense vector that captures meaning, so vectors close in the space mean similar things and similarity becomes a distance computation → a bi-encoder embeds each document once, independently, so you can pre-compute and index the whole corpus ahead of time → at query time you only embed the query and compare vectors, which is cheap → the alternative, scoring every query-document pair jointly (a cross-encoder), is far more accurate but O(corpus) per query and impossible to run over millions of docs at interactive speed → so the architecture splits the work: cheap pre-computed bi-encoder for first-stage retrieval, expensive cross-encoder only on the small reranked top-N.

This track is the infra/ops view of embeddings — serving, indexing cost, and scale. For the context-engineering angle (how embeddings shape what enters the prompt), see Embeddings, Vector Indexes (HNSW/IVF/PQ) & Cost, and for how you measure retrieval quality, RAG Evaluation: recall@k, RAGAS & Observability.

3

Vector Databases & ANN

A vector DB indexes embeddings for approximate nearest-neighbor search at scale. Exact k-NN is O(n) per query and collapses; ANN trades a little recall for large speed and memory wins.3

Exact vs approximate — why ANN exists

Exact scans everything; ANN searches a fraction✗ Exact k-NNcompare query to every vectorO(n) per queryperfect recall, does not scale✓ ANN (HNSW / IVF)visit only a few via the index~log(n) or cluster-boundedtiny recall loss, scales to billionsThe whole reason a vector DB exists: make billion-vector search interactive by skipping most of the corpus.
Exact search is correct but linear — fine for a prototype, fatal at scale. ANN indexes turn the scan into a guided traversal, accepting that they may occasionally miss a true neighbor.3

The two ANN families and their knobs

IndexHow it searchesThe recall knob & trade

HNSW (graph)

Layered proximity graphs; greedy descent, logarithmic hops

ef ↑ / M ↑ raise recall; cost is latency + memory

IVF (clustering)

Partition into cells; probe only the nearest few

nprobe ↑ raises recall; cost is latency, lighter on memory

Exact (flat)

Brute-force scan of all vectorsNo knob — perfect recall, O(n), small corpora only

The one trade you always tune

Recall ↔ latency ↔ memory: pick your pointRECALLfound the true neighbors?LATENCYwider search = slowerMEMORYgraph degree = RAMef / nprobeslide right: more recall, more latency
Every ANN deployment is a chosen point on this triangle. The search-breadth knob (effor HNSW, nprobe for IVF) slides you along recall↔latency; build-time degree (M) and quantization trade memory. There is no setting that maximizes all three.3

Is A vector DB stores embeddings and serves ANN similarity search with metadata filtering, CRUD, sharding, and replication — the serving substrate retrieval reads from.

Why it exists A flat array + brute-force similarity works as a demo but collapses on latency, freshness, and multi-tenancy. A vector DB makes search sub-linear and operable.

Like (world) A library’s card catalog: you don’t walk every shelf, you follow an index that jumps you near the right section — accepting it occasionally points you one shelf off.

Like (code) A B-tree index on a SQL column: it converts a full scan into a guided lookup. ANN is the same idea for high-dimensional vectors, but approximate by design.

✗ “ANN gives the same results as exact search, just faster.”

✓ ANN is approximate — it can miss a true neighbor. You buy recall back with the search knob, paying latency for it. Measure recall@k; don’t assume it.

✗ “I always need a dedicated vector database.”

✓ If embeddings fit beside your OLTP data and volume is modest, pgvector in Postgres keeps one store you already operate. Reach for a dedicated system when scale, QPS, or hybrid search force it.

📥 Memory rule: A vector DB trades a little recall for large speed & memory wins. The search knob (ef/nprobe) is the dial — and the reranker can only re-order what it returned.

Memory check
  • Why doesn't exact k-NN scale? → it's O(n) per query — every query scans the whole corpus, which is fatal past a few hundred thousand vectors at interactive latency
  • What does the HNSW ef / IVF nprobe knob trade? → recall against latency — a wider search finds more true neighbors but costs more time
  • Can a reranker fix low first-stage recall? → no — it only re-orders the top-K it was given; a doc the ANN search missed is gone
M3 — Why can’t you just run exact nearest-neighbor search over your embeddings, and what does ANN trade away?

Hit these points: exact k-NN compares the query against every stored vector, so it is O(n) per query and does not scale past a few hundred thousand vectors at interactive latency → ANN indexes (HNSW, IVF) build a structure that searches only a fraction of the corpus, turning O(n) into roughly logarithmic or cluster-bounded work → the trade is recall: ANN may miss a true neighbor, so you tune a knob (HNSW ef, IVF nprobe) that buys recall back at the cost of latency and compute → there is no free lunch: higher recall means a wider search means more latency, and the reranker can only re-order what the first stage returned, never recover a doc it missed.

Retrieval practice — test the three modules

Q1. In the standard RAG request path, the hops occur in which order?
  • Reranker first, then embed the query, then run the vector search over the corpus
  • Inference server first, then gateway, then embed, then a vector similarity search
  • Embed the query, vector-DB ANN search, rerank top-K, gateway, then inference
  • Gateway first, then inference, then embedding, then a final reranking of results
Q2. Why do you pre-compute document embeddings once at ingest time?
  • A bi-encoder embeds each doc alone, so the corpus indexes offline and queries stay cheap
  • Because embeddings expire quickly and must be refreshed before every single user query
  • So the reranker can skip the cross-encoder stage and score every pair at query time
  • To let the model gateway cache the vectors and avoid calling the inference server later
Q3. Exact k-NN search over a large embedding corpus does not scale because it…
  • Loses recall as the corpus grows, so it misses most of the true nearest neighbors
  • Needs a graph index that becomes too large to hold in memory past a few thousand rows
  • Returns approximate results that drift further from correct with every new insert
  • Compares the query to every stored vector, making it O(n) per query at any size
Q4. Raising the HNSW ef (or IVF nprobe) search knob will…
  • Shrink memory use and latency together while leaving the recall rate exactly the same
  • Raise recall by widening the search, at the cost of higher per-query latency and compute
  • Let the reranker recover documents the first-stage ANN search had already missed
  • Convert the approximate index back into an exact scan with perfect recall and no cost
Q5. The model gateway is best described as the stack's…
  • The component that turns query text into the dense vector used for similarity search
  • The GPU process that runs the forward pass and manages the KV cache during decoding
  • Cost and reliability chokepoint: auth, routing, rate limits, caching, and spend control
  • The cross-encoder that re-scores the candidate set returned by the vector database tier

Interview — pick your bar

Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).

Name the hops a query crosses from text to streamed answer, in order.
Hit these points: query text → embedding model (text becomes a vector) → vector DB (ANN search returns the top-K candidates) → reranker (a cross-encoder re-scores the top-K for precision) → model gateway (auth, routing, rate limits, response cache) → inference server (KV cache + batching, the forward pass) → tokens stream back → the first half decides what goes in the prompt; the second half governs how it's served.
What is an embedding, and what does proximity in the vector space mean?
Hit these points: an embedding is a dense vector that captures meaning → vectors that are close together represent similar things, so similarity becomes a distance computation (cosine or dot product) → you pre-compute the corpus embeddings once at ingest with a bi-encoder, then embed only the query at runtime → that's what lets keyword-blind semantic search work: "car" and "automobile" land near each other even with no shared tokens.
What does a vector database give you over a flat array of vectors plus a similarity loop?
Hit these points: ANN indexing for sub-linear search over millions of vectors instead of a full O(n) scan → metadata filtering so you can constrain by tenant_id, date, or type in the same query → CRUD and incremental index maintenance so inserts/deletes don't force a rebuild → the operational layer: sharding, replication, persistence, backups → the flat-array version is a fine prototype but collapses on latency, freshness, and multi-tenancy as it grows.
Why is exact nearest-neighbor search a non-starter at scale, and what replaces it?
Hit these points: exact k-NN compares the query to every stored vector, so it's O(n) per query → that's fine for a few hundred thousand vectors but fatal at millions or billions at interactive latency → ANN (approximate nearest neighbor) builds an index — a proximity graph (HNSW) or clusters (IVF) — that visits only a fraction of the corpus → the trade is a small recall loss for large speed and memory wins → you tune recall back with a search knob, accepting more latency.
Where does latency concentrate in this stack, and how do you find the dominant hop?
Hit these points: instrument every hop with a span — embed, ANN search, rerank, gateway, inference (queue + forward pass), response — and read per-hop p95, not just the total → on most stacks the inference forward pass dominates because generation is sequential and GPU-bound, with queue-wait next under load → ANN search is usually single-digit ms unless you over-tuned recall; the reranker adds a fixed cost proportional to candidate count → the gateway is a thin hop unless it's retrying or failing over → the discipline: measure per-hop first — the hop you assume is slow rarely is.
Why is the model gateway the reliability and cost chokepoint, and what belongs in it?
Hit these points: every request to every model flows through it, so it's the single place to see and cap spend and the single thing that can take the feature down → it owns auth, per-tenant rate limits and quotas, routing across providers/tiers, retries with backoff and failover, response caching, and cost accounting per request → that makes it the natural place to enforce a budget and degrade gracefully when a provider is down → it's also a blast-radius risk: it must be horizontally scalable and stateless where possible, never a hidden single instance → it's the API-gateway pattern applied to LLM traffic.
A reranker can only re-order what vector search returned. Why does that constrain the whole pipeline?
Hit these points: retrieval is two stages — a cheap high-recall first stage (ANN over embeddings) and an expensive high-precision second stage (the cross-encoder reranker) → the reranker re-scores only the top-K it was handed, so if the right doc isn't in that K, no reranking recovers it → recall is set by the first stage; precision is improved by the second → so tune the first stage for recall (wider ANN search, larger K), accept the latency, then let the reranker sharpen order → failure mode: a too-small K or under-tuned ANN silently caps answer quality and reranking hides none of it.
What is the KV cache, and why does it dominate GPU memory during inference?
Hit these points: during generation the model reuses the attention key/value tensors for every token already processed, caching them instead of recomputing → the KV cache grows with sequence length and concurrent requests, and at long contexts it dwarfs the model weights as the memory consumer → fragmented or over-allocated KV cache wastes GPU memory and caps how many requests you can batch, which caps throughput → this is exactly what PagedAttention addresses — paging KV memory like an OS pages RAM to cut fragmentation and pack more sequences → so KV-cache management is the lever that sets serving throughput and cost per token.5
What does continuous (in-flight) batching buy you over static batching, and what's the trade?
Hit these points: GPUs are efficient on big matmuls, so one request at a time leaves the hardware idle — batching fills it → static batching waits to assemble a fixed batch and holds fast requests hostage to the slowest, hurting tail latency → continuous batching schedules at the token step: it admits new requests and evicts finished ones each iteration, keeping the GPU full and not blocking short requests behind long ones → the trade is scheduler complexity and a memory ceiling set by the KV cache — you can only batch as many sequences as KV memory holds → net: higher throughput and steadier latency, which is why production inference servers default to it.5
The same retrieval corpus, but answer quality differs between two index configs. How do you reason about it before touching the model?
Hit these points: separate recall (did the right doc make the candidate set?) from precision (was it ranked high enough to use?) → measure recall@k against a labeled set for both configs — if the doc isn't in the top-K, it's a first-stage problem: under-tuned ANN (ef/nprobe too low), too-small K, or chunking that split the answer → if recall is fine but ranking is poor, it's the reranker or the assembly order, not retrieval → only after retrieval is sound do you look at the prompt or model → the principle: don't tune the expensive end of the pipeline to compensate for a cheap-end recall gap — fix it where it's caused.
"We'll just add a bigger model and a longer context — that fixes quality." Tear this apart as a system design.
Hit these points: a bigger model and longer context don't fix a retrieval gap — if the right doc never enters the window, no model size recovers it → longer context inflates the KV cache, shrinking how many requests batch, raising latency and cost per token at the inference server → a bigger model raises per-token cost and latency everywhere, hitting the gateway budget hardest → the cheaper, higher-leverage fixes live upstream: better embeddings/chunking for recall, a reranker for precision, response caching for repeat queries → the framing: spend where the bottleneck is measured to be, and the bottleneck is usually retrieval quality or serving throughput, not model capacity.
Design-round framework — narrate the stack in request order so the interviewer hears each hop's job, then its bottleneck:
  1. Clarify scope: QPS, corpus size, latency SLO, freshness, tenant isolation, and the cost of a wrong answer vs a slow one.
  2. Retrieval half: embed the query (cache popular embeddings) → ANN search with a tenant_id metadata pre-filter → rerank the top-K with a cross-encoder sized to the budget.
  3. Serving half: assemble prompt + passages → model gateway (auth, per-tenant rate limit, route, response-cache check) → inference server (KV-cache paging, continuous batching, streaming).
  4. Name the bottleneck at each hop: ANN recall vs latency, gateway as cost/reliability chokepoint, KV memory capping batch size.
  5. Build-vs-buy: pgvector vs a dedicated vector DB, hosted vs self-served inference — decide from measured scale, not preference.
  6. Measure & degrade: per-hop p95, cost per query, recall@k, cache hit ratio; graceful degradation when a provider fails.
Design the request path for a multi-tenant RAG search feature: thousands of QPS over tens of millions of documents.
A strong answer covers: clarify scope first — QPS, corpus size, latency SLO, freshness, tenant isolation, and the cost of a wrong vs slow answer → map the path: embed the query (cache popular query embeddings), ANN search with a tenant_id metadata pre-filter so isolation is enforced at the index, return top-K → rerank the top-K with a cross-encoder sized to the latency budget → assemble prompt + passages and send through the model gateway (auth, per-tenant rate limit, route, response-cache check before spending on inference) → on a miss the inference server runs with KV-cache paging and continuous batching, streaming tokens → name the bottlenecks: ANN recall-vs-latency, the gateway as cost/reliability chokepoint, KV memory capping batch size → measure per-hop p95, cost per query, recall@k, cache hit ratio; degrade gracefully when a provider fails.
Build-vs-buy: when do you reach for a dedicated vector DB versus pgvector in your existing Postgres?
A strong answer covers: start from the data, not the hype — if embeddings fit alongside your OLTP data and retrieval volume is modest, pgvector keeps everything in one store you already operate, back up, and join against, a real operational saving → reach for a dedicated vector DB when scale or traffic breaks that: very large corpora needing sharding, high QPS needing an independently scaled retrieval tier, advanced hybrid search and reranking modules, or serverless cost models → the trade is operational surface — another system to run, secure, and pay for, against Postgres's familiarity and weaker ANN tuning → deciders: corpus size, QPS, freshness, metadata-filter complexity, team operational appetite → default to the store you already run until a measured limit forces the move.4
Sources
Request-path framing follows the standard RAG retrieve → assemble → serve architecture, grounded in vendor serving docs (vLLM — docs.vllm.ai) and provider API docs (e.g. platform.openai.com/docs, docs.anthropic.com) rather than a single paper.
Reimers & Gurevych, 2019, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks". Bi-encoder architecture: pre-compute document embeddings once, compare cheaply at query time. Khattab & Zaharia, 2020, "ColBERT", covers the late-interaction middle ground.
Malkov & Yashunin, 2016, "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" — arXiv:1603.09320 (HNSW: ef/M knobs). Johnson, Douze & Jégou, 2017, "Billion-scale similarity search with GPUs" — arXiv:1702.08734 (FAISS: IVF + product quantization, recall vs memory).
pgvector documentation — github.com/pgvector/pgvector (exact + HNSW/IVFFlat in Postgres). Pinecone documentation — docs.pinecone.io (managed serverless vector DB: sharding, replication, metadata filtering) for the build-vs-buy framing. Product specifics are version-dependent.
Kwon et al., 2023, "Efficient Memory Management for Large Language Model Serving with PagedAttention" — arXiv:2309.06180 (vLLM, SOSP '23). KV cache as the serving bottleneck; continuous batching and OS-style paging for throughput. Throughput numbers are hardware/model-specific; treat as illustrative.