System Design for AI in Production

Two problems wear one costume here: how you would have caught this before full rollout, and how you debug it after the fact. Cover both and you show senior judgment. This walks through the staged-rollout and root-cause playbook.

The most common ML system design round. Interviewers reward the funnel structure: candidate generation, then ranking, then re-ranking, with the right model at each stage plus an honest plan for cold start, freshness, and feedback loops. This lays out that structure.

Severe class imbalance, a tight latency budget, and an adversary who keeps adapting. What interviewers watch for: treating imbalance honestly, setting the operating point from costs, and building for the feedback loop. This covers the end-to-end design.

Models fail silently, so the real question is whether you would notice. What interviewers look for: watching the right layers (operational, data, prediction, outcome) and alerting on drift without drowning in false pages. This covers the system and the metrics that count.

Multimodal search checks whether you grasp shared embedding spaces and the retrieve-then-rank pattern under a real catalog's scale and freshness. What interviewers reward: CLIP-style joint embeddings, hybrid retrieval, and honest relevance evaluation. This lays out the design.

Text-to-SQL is deceptively hard because correctness is all-or-nothing and the failure mode is a confident wrong number. What interviewers reward: schema grounding, query validation, and a safety layer, not merely 'prompt an LLM with the schema.' This lays out the production design.

Moderation is a multi-stage classification problem with harsh tradeoffs: false negatives cause real harm, false positives silence legitimate users, and the adversary keeps adapting. What interviewers reward: the tiered pipeline, per-severity precision/recall calibration, and human-in-the-loop. This lays out the design.

The moment a company runs LLMs in more than one place, it needs a gateway. The signal is the cross-cutting concerns (cost, reliability, observability, governance) that a gateway consolidates, not merely 'proxy the API.' Here is the design.

Ads ranking is where calibrated probabilities collide with tight latency and money. The signal is knowing that CTR has to be calibrated (not just ranked), the feature and serving design, and the auction context. Here is the design that goes past 'train a classifier.'

Anomaly detection seems simple until seasonality, cold start, and alert fatigue arrive. The signal is modeling the expected baseline (including weekly and daily cycles), picking unsupervised methods when labels are scarce, and tuning so you do not bury users in false alarms. Here is the design.

The canonical AI system-design question. The signal is covering both the model lifecycle and the serving stack at real scale, without rambling. Most candidates design only half and lose the points.

The modern agentic system-design question. The signal is the plan, search, read, synthesize, verify loop with per-claim citations, plus keeping cost and latency bounded. The hard parts are not the writing.

Cross-session memory is what makes an assistant feel personal, and it is mostly a retrieval and state-management problem rather than a bigger context window. The signal is the tiered architecture plus what to store, forget, and protect.

Support is the flagship applied-AI use case, exercising agents, RAG, tools, and the critical human handoff. The signal is knowing when to answer, when to take action, and when to escalate, safely.

Limited GPUs, dozens of models, and every team demanding low latency for little money. The signal is whether you can shape that into a single governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can genuinely attribute.

AI code review succeeds or fails on precision: a handful of noisy comments and the team silences the bot for good. The signal is grounding in the diff plus repo context, unforgiving false-positive control, and winning developer trust one accepted comment at a time.

Summarize, draft, and triage, all across the most sensitive PII a person holds. The signal is grounding in the real thread, matching the user's voice, and a firm human-in-the-loop rule on anything that gets sent.

A GPU-heavy generative system where sampling steps drive the bill and a single bad image becomes a headline. The signal is the diffusion serving pipeline, the cost levers on per-image GPU time, and a required two-sided safety layer.

Screening resumes is both high-volume and high-stakes: a legally sensitive, bias-prone judgment about people. The signal is trading off throughput against fairness, human oversight, and explainability, not an LLM ranking resumes. Here is the design.

Transcribing in real time at scale probes streaming ASR, latency budgets, and GPU fleet management under heavy concurrency. The signal is chunked streaming inference with partial results, not batch transcription. Here is the design.

Document extraction (IDP) is a massive enterprise use case with a steep correctness bar: a wrong extracted number beats having none. The signal is the parse, extract, validate, human-review pipeline plus confidence-based routing. Here is the design.

A voice assistant links STT, an LLM, and TTS under a punishing latency budget where each stage piles on delay. The signal is streaming and pipelining the stages plus turn-taking, not three blocking calls. Here is the design.

Healthcare is the highest-stakes AI domain: a wrong answer can injure a patient, and regulators are paying attention. The signal is designing decision support with grounding, human oversight, and guardrails, never an autonomous diagnoser. Here is the design.

Legal review is high-stakes and precision-critical: a missed clause or hallucinated citation carries real consequences, and lawyers have been sanctioned over fabricated cites. The signal is grounding in the actual documents, citing exact passages, and lawyer-in-the-loop. Here is the design.

Dynamic pricing combines demand forecasting, optimization, and real-time serving, under fairness and trust constraints. The signal is the predict-then-optimize structure plus guardrails against perverse outcomes. Here is the design.

E-commerce search is where retrieval, ranking, and business goals meet, and it carries a strict exact-match requirement (brand, size, SKU). The signal is query understanding plus hybrid retrieval plus business-aware ranking. Here is the design.

A GPU-heavy generative-audio system with text/genre/lyric conditioning, minutes-long coherence, and copyright constraints. The signal is the generation pipeline plus the async, cost, and safety layer that gets it shippable.

Video generation is image generation plus the harsh constraint of temporal consistency, and it is the most GPU-intensive generative task around. The signal is diffusion over spacetime plus a serving design that holds up under multi-minute jobs.

Code migration is a high-value agentic task with a strict correctness bar: the migrated code has to still work. The signal is grounding in the real repo, transforming in verifiable chunks, and gating on tests, never one huge LLM rewrite.

Notification systems fail by spamming users until they mute, and a mute is a permanently lost channel. The signal is treating it as a per-user send/hold/batch/suppress decision optimized for long-term trust, not for clicks or volume.

Two models in a chain (ASR then summarization), a long-context problem buried inside, and a faithfulness bar where a fabricated action item carries real consequences. The signal is the pipeline plus how you keep a 3-hour transcript honest at thousands a day.

On-device AI gives up raw capability for privacy, offline use, and latency, under tight memory and battery limits. The signal is the small-quantized-model stack plus a hybrid router that sends the hard queries to the cloud without leaking private context.

The trap is swapping the classifier for an LLM. The real-time, imbalance, and adversarial constraints stay put. The signal is a hybrid: a fast calibrated model scores inline, LLMs investigate the gray zone off the hot path.

How large recommenders and search draw candidates from millions of items in milliseconds. The signal is why the user and item towers stay separate, how that enables precomputed embeddings plus an ANN index, and where ranking takes over.

Every keystroke has to return ranked suggestions in tens of milliseconds, at search-engine QPS. The interesting part is the data structure and what you precompute, not the query path. Here is the design that holds up under the latency budget.

ETA is spatiotemporal regression served in real time over conditions that shift by the minute. The signal is feature freshness, segment or stage decomposition, and the asymmetric cost of being wrong. Here is the design.

PYMK is graph recommendation at billion-node scale. The interviewer is testing for one instinct: do you generate candidates from the social graph, or naively try to score every pair? Here is the design that holds up through the follow-ups.

Feed ranking is multi-objective ML at massive scale with a harsh serving constraint. The interviewer wants to see whether you optimize a value model toward long-term satisfaction, or fall into the CTR trap that breeds clickbait. Here is the design.

The service under every RAG stack and semantic search box. The signal is your ANN index choice and how you deal with the three things that quietly break it: metadata filtering, live updates, and scale.

Predict, value, and bid on one impression in roughly 10ms, billions of times a day. The signal is the pCTR-to-bid pipeline under a tight latency budget, plus why calibration and budget pacing decide whether you make or lose money.

Image search is embedding retrieval at billion-image scale. A strong answer shows when a vision-only encoder is enough and when you need a CLIP-style shared space, plus how the ANN index actually serves the query. Here is the design.

Spam detection is adversarial, imbalanced classification under a low-latency bar. The interviewer is watching whether you set the threshold by cost asymmetry and rely on signals spammers cannot fake. Here is the design.

Query understanding is the front end of search that converts a raw query into structured intent. Get it wrong and even a perfect index returns garbage. The signal is the pipeline and how you measure it. Here is the design.

Online models need features that are fresh and computed the same way training computed them. The classic failure is training-serving skew that quietly degrades the model. The signal is the online/offline split built on one shared definition. Here is the design.

New users and new items carry zero interaction history, so collaborative filtering has nothing to work with. The candidates who pass treat it as a lifecycle, not a single trick. Here is how.

Rank items by relevance alone and you ship ten near-identical results and a filter bubble. What interviewers want is a re-ranking stage that scores the list as a set, plus the metric that proves it works.

Some questions need precise multi-hop facts that vector retrieval cannot chain. The signal is entity linking plus query translation over a graph (or GraphRAG), and knowing when the graph is worth its maintenance cost. Here is the design.

AV perception is safety-critical, real-time, multimodal ML: detect and track everything around the vehicle across multiple sensors. The signal is sensor fusion plus detection/tracking under a hard latency budget and a fail-safe bar. Here is the design.

Machine translation is seq2seq generation delivered across hundreds of language pairs under strict latency and quality budgets. What matters is sidestepping the quadratic pair blowup, saving low-resource languages, and serving it cheaply. Here is the design.

Detecting walking, driving, or a fall from a sensor stream is windowed time-series classification, typically on a battery-bound device. What matters is windowing, temporal smoothing, and the on-device constraints most candidates overlook. Here is the design.

Matrix factorization is the classic collaborative-filtering method and the conceptual seed of modern embedding-based recsys. What matters is the latent-factor idea and how it leads to two-tower/neural models.

Recommending events (or jobs) breaks ordinary recsys in two spots: items expire and items are local. What matters is how you handle perishability, geography, and the fact that every item is a cold-start item. Here is the design.

Tagging articles at scale is bread-and-butter ML, but the naive version fails three ways: it is multi-label, the taxonomy is hierarchical and keeps growing, and rare classes hide behind aggregate accuracy. What matters is the model choice plus how you handle all three.

A recommender trains on data its own past recommendations produced, so it learns to confirm its own beliefs. What matters is spotting the loop, naming the biases it breeds, and knowing the exploration and debiasing fixes that break it.

Labeled data is the fuel for ML, and a labeling platform succeeds or fails on quality control. The signal is the workflow plus the quality math: consensus, gold honeypots, inter-annotator agreement, and active learning to spend the budget where it counts.

Detecting and boxing objects at scale reduces to one driving tradeoff plus a handful of CV specifics most candidates fumble: the detector family, NMS, focal loss, and why mAP rather than accuracy is the metric. Here is the design.

Two forces shape this design: documents longer than the context window, and summaries that must not lie. The strong answer pairs hierarchical map-reduce with claim-level grounding, then serves it cheaply at scale.

A high-AUC churn model that changes no behavior is worthless. The strong answer nails the label definition, chooses classification vs survival deliberately, and is judged on retention uplift, not accuracy.

Reps only have so many hours, so the model exists to steer them toward the leads worth calling. Interviewers listen for value-weighted prioritization, carefully built conversion labels, and evidence of revenue uplift measured against a holdout.

Forecasting a single series is textbook work; forecasting a million SKU-stores that have to reconcile, where stockouts cost more than overstock, is what the interview actually tests. The strong answer is global, hierarchical, and cost-aware.

A Devin-style agent that carries an issue all the way to a merged PR is the hardest kind to make reliable, since every step can fail and errors pile up. Here is the architecture and the loop that keeps it honest.

Exact-match caching does little when no two prompts match. Semantic caching reuses answers across similar queries, and its entire danger is handing back a near-match that is subtly wrong. Here is how to build it safely.

GPUs are scarce and costly, so multi-region HA for LLMs is not simply web-app HA with bigger machines. Capacity, routing, and failover all bend around the GPU constraint. Here is the design.

Batch scoring a billion items is a throughput-and-cost problem, the reverse of low-latency serving. Every choice that harms latency works in your favor here. Here is how to design for dollars-per-million-documents.

When every team evaluates prompts in one-off notebooks, an org ships regressions and argues about vibes. A shared eval platform turns quality into something measurable and comparable. Here is what it must provide.

When prompts live inside source code, every wording tweak becomes a full deploy and nobody knows which prompt is running. The fix is to treat prompts as managed, versioned configuration. Here is the platform.

An agent that clicks through a real website is slow, brittle, and a single wrong click away from a costly mistake. The architecture comes down to perception, action, and guardrails around irreversible steps. Here is how to build it.

RLHF and evals can only match the preference data behind them, and that data comes from humans whose quality swings wildly. The platform that yields trustworthy labels is a serious system in its own right. Here is its design.

'What was revenue by region last quarter?' looks like text-to-SQL, yet production analytics agents break on ambiguity, wrong joins, and confidently wrong numbers. Here is the architecture that makes the answers trustworthy.

Hand every team an LLM API key and your bill turns into a mystery capped by a six-figure surprise at month end. Attributing, capping, and optimizing spend is a platform feature. Here is how to build it.

An AI support agent working through millions of chats will occasionally be wrong, rude, or unhelpful, and no one can read them all. Watching quality at that scale is a design problem in itself, with its own failure modes.

Classic distributed-systems tradeoffs appear in ML infra with an ML twist: stale features and eventually-consistent reads carry model-accuracy consequences, not just correctness ones. Here is how to reason about it.

Anthropic's signature opener. The naive answer (one server pushes to N clients) is capped by the source uplink and scales linearly. The answer interviewers reward turns receivers into senders so the swarm's capacity grows with its size. Here is that reasoning.

The Dynamo question. Interviewers want three decisions made cleanly: how you partition keys so adding nodes does not reshuffle everything, how you replicate for durability, and where you sit on the CAP spectrum. Here is how to reason through all three with concrete quorum math.

Every API platform needs one. Interviewers reward the candidate who picks the right algorithm (token bucket vs sliding window) for the burst behavior, then cracks the genuinely hard part: holding one shared limit across many servers without a per-request round trip to a central store.

Looks simple until the follow-ups: which layer do you balance at, how do you keep traffic off a dead server, and how does the load balancer itself avoid being a single point of failure? Interviewers want the algorithm, the health-check loop, and the high-availability story.

A deceptively rich warm-up. The interesting calls are how you mint short, collision-free, unguessable keys at scale, and how a read-heavy 100:1 workload shapes storage and caching. Here is the clean design with capacity math.

The observability infrastructure question. Interviewers want you to split the three telemetry types, choose storage that withstands massive write volume, and design alerting that fires fast without burying on-call in noise. Here is the pipeline with retention and cardinality math.

The distributed ordering problem that sits at the heart of Kafka, replicated logs, and consensus systems. Interviewers look for you to face the uncomfortable fact that physical clocks lie, then pick the right instruments: a single sequencer, consensus, or logical clocks. This walks that progression.

The real-time messaging staple. Interviewers look for the delivery path (persistent connections, not polling), a fan-out strategy that survives large groups, and a clear story on ordering and delivery guarantees. This lays out that design, including the fan-out tradeoff that marks senior candidates.

This asks you to design the scoreboard, not the player. A strong answer keeps offline gates (AUC, calibration, NDCG) apart from the online verdict (A/B with revenue, user, and advertiser guardrails), and layers in counterfactual replay so you can trust a model before it ever serves a live auction.

A news feed differs from a generic recommender. Time is a first-class signal: a story that mattered this morning is noise by tonight. The interviewer expects recency decay, an engagement-versus-quality value model, and a plan for filter bubbles, not merely retrieval plus ranking.

Truth is not a label you can gather cheaply, and adversaries adapt as soon as you ship. A strong answer combines content, graph, and behavioral signals, places humans in the loop where precision matters, and treats adversarial drift as a permanent operating condition rather than a one-time training problem.

Detecting the language of a string seems trivial until the input is three words, blends two languages, or is half code. A strong answer chooses the model by latency budget, handles short text and code-switching, and returns calibrated confidence rather than a brittle single guess.

Cleaning noise out of speech is a tradeoff between how good it sounds and how fast it runs. A strong answer weighs spectrogram masking against waveform models under a real-time budget, names PESQ and STOI as the metrics, and stays honest about the artifacts each approach brings.

Video search is image search plus a time axis, and the time axis is the whole problem. A strong answer embeds frames, pools them into scene vectors, indexes with ANN at billion scale, and explains how a multimodal query (clip, text, or both) locates the right moment, not just the right video.

Inline completion lives or dies on tail latency: a suggestion that lands after the developer keeps typing is useless. Learn how to assemble the context window, hide model latency behind speculation and caching, and keep a tight feedback loop on acceptance rate.

Enterprise search lives or dies on permissions, freshness, and wiring into thirty messy SaaS sources; retrieval quality is table stakes. Learn how to fan out across connectors, enforce per-user access at query time, and blend lexical with vector search for results people trust.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context wins out, and how to evaluate summaries at scale.

Moderating billions of items a day with an LLM on each one is too slow and too expensive. Learn the tiered funnel (cheap classifier then LLM then human), how to tune thresholds for precision versus recall, and how to stay ahead of adversaries.

Personalizing an LLM is a retrieval and memory problem, not a per-user fine-tune. Learn how to assemble user context at request time, keep long-term memory from bloating the prompt, and respect privacy and the right to be forgotten.

A writing assistant has to feel instant as the cursor moves through a live document. Learn how to separate fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker while the user types.

Auto-resolving support tickets means grounding answers in your knowledge base, safely calling real APIs (refunds, account changes), and knowing when to hand off to a human. Learn the agent loop, the guardrails, and how to measure resolution without eroding trust.

Agents that loop for minutes, invoke tools, and spawn sub-tasks demand durable execution rather than one request handler. See how to checkpoint state, recover after failures, cap runaway loops, and trace every move a non-deterministic agent makes.

Running experiments on LLM features is tough because outputs are open-ended and quality is fuzzy. See how to assign traffic, choose metrics beyond engagement, tame variance from non-determinism, and dodge the traps that let a winning variant lose once it ships.

Selling RAG as a product means wrangling messy ingestion, multi-tenant isolation, and per-customer index freshness, all behind a simple API. See the ingestion and query planes, how to keep tenants isolated, and how to hand customers eval and observability.

Live speech-to-speech translation is a latency race across three models (recognize, translate, synthesize) as audio keeps streaming in. See the streaming pipeline, how to commit partial results without flip-flopping, and the tradeoff between latency and translation quality.

A celebrity posts and 50M followers need a notification. The tricky parts are not sending one message; they are fan-out strategy, deduplication, channel routing, and not flooding users. Here is the architecture that handles both the long tail and the viral spike.

A rider taps request and within seconds a nearby driver gets assigned. The interesting problems are geospatial indexing of moving drivers, the matching objective (nearest is not always best), and handling the race where two riders want the same car. Here is how to build it.

In payments, a duplicate charge or a lost credit is not a bug, it is a financial incident. The design turns on idempotency keys, double-entry bookkeeping, and an append-only ledger you can audit and reconcile. Here is how money systems stay correct.

A cache is easy until you spread it across many nodes. Then come consistent hashing, eviction policy, the thundering herd on a cache miss, hot keys, and how much staleness you can tolerate. Here is the design that holds up under real traffic.

Metrics, traces, and IoT data are append-heavy, time-ordered, and seldom updated. A general database handles this poorly. The wins come from columnar layout, compression tuned for timestamps, downsampling, and retention. Here is the time-series design interviewers want.

Raw clickstream lands in object storage and somehow turns into clean tables, dashboards, and ML features. The design questions are table format, the bronze-silver-gold layering, batch versus streaming, and how you handle schema drift and late data. Here is the lakehouse blueprint.

Crawling a few pages is trivial. Crawling the web means a URL frontier with priority, per-host politeness, dedup across billions of URLs, and defenses against spider traps and infinite content. Here is the crawler architecture that scales without getting your IPs banned.

Showing the top 10 is easy. Showing a player their global rank among 50M others, updated live, is the real challenge. The answer turns on the right data structure, sharding, and approximate ranks for the long tail. Here is how leaderboards scale.

Find the 20 nearest restaurants within 5km, fast, for millions of users. A naive distance scan over every place is hopeless. The design centers on spatial indexing, geohash or quadtree partitioning, and coping with dense cities versus empty regions. Here is how proximity search works.

Background jobs fail, retry, and get redelivered. If a job sends an email or charges a card, running it twice is a real problem. The design combines at-least-once delivery with idempotent handlers, visibility timeouts, and dead-letter queues. Here is how to make jobs safe.

When a page loads, an ad slot goes to auction across dozens of bidders and a winner is picked before the page finishes rendering, all in under 100ms. The design covers the auction mechanism, the brutal latency budget, budget pacing, and fraud. Here is how an exchange runs millions of auctions per second.

Moving a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards in one stroke. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

SSE versus WebSocket is the easy half. The half that separates candidates: a buffer-everything output filter throws away the time-to-first-token you paid a GPU to deliver, and you cannot send an HTTP error after the 200 has already flushed.

The LLM is the easy part. The system is a learner model: a skill graph, a per-student mastery estimate, and an item selector that keeps the student at the edge of their ability. Here is the design, including the pedagogical constraint most candidates miss.

Five thousand businesses, five thousand assistants, one platform. The hard parts are tenant isolation you can defend, noisy neighbors, onboarding a tenant with zero data, and evaluating assistants you can never manually QA. Here is the control plane and the data plane.

You cannot review a live stream after publishing, because it already reached the audience. The broadcast delay buffer is the design, and everything else (frame sampling, the audio channel, the cascade) is built to fit inside it. Here is how it fits together.

Every AI system spends quality to buy speed, whether or not the team admits it. What scores is knowing the exchange rate for each knob, taking the free wins first, and having already decided what you will sell when the system saturates.

A provider outage should cost you quality, not the feature. The answer that scores is a pre-decided fallback ladder, a timeout policy that treats slow as down, and the discipline to test the path you hope never runs.

The root cause is not a weak system prompt. A model trained to be agreeable is dangerous to a user in crisis, because the objective and the safety requirement point in different directions. The fix has to live outside the model. Here is what that architecture looks like.

A classifier trained on one culture's annotations does not generalize to another culture's speech, and the aggregate metric hides it. The signal is recognizing that the ground truth itself is the bug, then designing the policy and eval stack around that.

Trust is an interface and reliability problem, not a messaging one, and the goal is not maximum trust. Here is the mechanism list that actually moves it, the recovery path after a public failure, and the metric most teams optimize in the wrong direction.

Every AI product that runs generated code (notebooks, agents, code interpreters) ends up building this, and the interview turns on two numbers: how strong your isolation is and how fast a session starts. The answer is microVMs plus snapshot restore, and knowing why the VM boot is not the slow part.