AI System Design Under Real Constraints: Latency, Cost, Privacy, Quality

AI system design under real constraints is not the generic distributed-systems interview. The hard part is rarely raw scale. It is four constraints that pull against each other: a latency budget the product cannot exceed, a cost per request that has to survive real traffic, a privacy boundary the data cannot cross, and a quality bar you have to prove you hit. Your job is to design a working system, often a retrieval pipeline or an agent, inside those walls, and to treat evaluation as a gate rather than an afterthought. The most common failure is designing the happy path and bolting the constraints on at the end. Here is how to design from the constraints outward.

The constraints are the problem, not the footnotes

A frontier model that performs in a demo is the easy part, and it has been for a while. The question that matters is how it holds up under production conditions: a p95 someone will page you about, a bill someone will question, data someone is legally responsible for, and answers someone will trust.

So open with the constraints. Before you draw a single box, ask what they are, because they reshape every decision that follows.

• Latency budget. A hard number, often in the low seconds, that caps how many model calls and retrieval hops you can afford.
• Cost per request. Tokens in plus tokens out, times price, times volume. It decides model size, context size, and whether an agent loop is even viable.
• Privacy and data boundary. Where the data may live, who may see which document, what gets logged, what a security reviewer will check.
• Quality bar. What score, on what eval, means this is good enough to ship.

Treat these as footnotes and you have misread the question. The system design questions are the main practice set here, and the must-know set is the fastest way to surface your gaps before the loop.

Design against the latency budget

The latency budget is usually the first genuine constraint, because it decides the shape of the pipeline. Every model call and retrieval hop draws down a fixed account. Say the budget out loud, then spend it deliberately.

A rough accounting keeps you honest. An embedding call and a vector search are cheap, typically tens of milliseconds. A generation call dominates, and it scales with the output length rather than the input, so a long answer costs you more than a stuffed context. A cross-encoder reranker buys precision and spends latency. An agent loop multiplies your entire budget by the number of turns, which is why a five-step agent almost never fits a two-second interactive budget and belongs in an async or background flow instead.

The moves worth naming: stream the first token so perceived latency drops even when total latency does not, cache the head queries, cap reranking depth, and know when one well-grounded call beats a multi-step loop. Say which of these you would reach for and why. The RAG and agent design questions map most closely to these tradeoffs.

Design against cost per request

Cost is the constraint candidates handle worst, because it is the one nobody drills. Do the multiplication out loud. Tokens in plus tokens out, times the price per token, times requests per day. You do not need current prices memorized, and the interviewer does not want a quote. They want to see that you know which knobs move the number.

Retrieval inflates the input side, so stuffing twenty chunks into the prompt to be safe is a cost decision, not a free one. Model choice moves the number by roughly an order of magnitude, which is why the pattern of routing easy queries to a small model and escalating only the hard ones is worth proposing by name. Caching helps most when the query distribution has a fat head. Fine-tuning a smaller model can beat prompting a large one on cost at high volume, at the price of a training and serving pipeline you now own.

The signal is that you notice cost at pilot volume looks nothing like cost at ten times that, and you say so before the interviewer does. The ML infrastructure questions go deeper on serving, batching, and the GPU economics behind those numbers.

Design against the privacy boundary

You are not a lawyer and the panel knows it. What they want is a design that behaves as if the data cannot cross its boundary, with the controls a reviewer will hunt for. Three things carry most of the signal.

Residency by construction. If the data has to stay in a region or inside a boundary, every component that processes it lives there too, including the index, the caches, and any logs that could capture content. Do not let a convenience feature quietly copy data out of bounds, and be specific that prompts and completions are data.

Identity and least privilege. The system should honor the existing permission model instead of reading everything as a superuser. A retrieval system that returns documents a user is not permitted to see is a security incident, not a bug. Carry the user's access scope into retrieval so results are filtered by what that user may actually read, and filter at query time rather than hoping the model declines.

Auditability. Keep a record of what was accessed and why, apply retention rules someone can verify, and show you would bring in security and compliance experts instead of guessing. The AI security and governance questions go deep on the access, audit, and data-handling concerns that surface here.

Design against the quality bar

This is where strong candidates pull ahead: they treat quality as a constraint with a number, not a vibe. Set the metric from the product goal, not a generic benchmark. Hold back a slice of real traffic and queries, fix a bar that means good enough to ship for this use case, and measure against it before launch.

Include the unhappy cases: the queries that should return nothing, the ambiguous ones, the ones that have already gone wrong. Then keep the eval running after deployment so quality regressions surface before users notice. A model provider updates a model, your corpus drifts, a prompt change ripples somewhere unexpected. An engineer who describes the offline eval set and the online guardrail (a groundedness check, a sampled human review, a canary on a slice of traffic) is describing a system they have actually operated.

The tradeoffs are the answer

The four constraints do not cooperate, and the panel wants to watch you negotiate them. Reranking buys quality and spends latency. Caching buys latency and cost and spends freshness. A smaller model buys cost and spends quality on the hard tail. Keeping data in a restricted environment buys privacy and can cost you access to the strongest model.

Do not present a balanced menu. Name the constraint that dominates for this product, choose to match it, and say what would make you reverse the call. "This is an interactive feature with a two-second budget, so I get one retrieval hop and one generation call, which means I spend my quality budget on better chunking and hybrid retrieval instead of a reranker. If it became an async workflow, I would add the reranker immediately."

Budget your time and lead with constraints

A workable split: roughly a third on constraints and high-level architecture, a third on the retrieval and evaluation pipeline, and a third on cost, privacy, and operability. Spend the whole window on chunking strategy and you will fail on the parts that set this round apart.

The through-line: begin from the budget, the bill, the boundary, and the bar. Design inside them on purpose, and prove the result works on real data. A slicker pipeline that quietly assumes infinite latency and free tokens loses to a plainer one that honors the constraints and can say what it costs.