serving

The per-token sticker price masks the real decision. Self-hosting only pays off past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

In naive LLM serving most GPU memory bleeds away to KV-cache fragmentation. PagedAttention adapts virtual memory paging to win it back. What matters is explaining the fragmentation problem and how blocks resolve it.

MoE trims compute but is awkward to serve: experts have to be sharded, tokens routed across devices, and batches balanced. What matters is the all-to-all communication and the load-imbalance problem, not the training story.

Both work, so the decision is made by four numbers: how many examples you have, how far the capability has to move, how many variants you must serve, and how many GPUs you own. The serving axis is the one candidates never mention.

Merging one adapter is a single matrix add. Merging four is where careers get interesting: the low-rank updates were each trained as if the others did not exist, and they collide. Here is what actually works and when to route instead.

Limited GPUs, dozens of models, and every team demanding low latency for little money. The signal is whether you can shape that into a single governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can genuinely attribute.

Tagging articles at scale is bread-and-butter ML, but the naive version fails three ways: it is multi-label, the taxonomy is hierarchical and keeps growing, and rare classes hide behind aggregate accuracy. What matters is the model choice plus how you handle all three.

Detecting and boxing objects at scale reduces to one driving tradeoff plus a handful of CV specifics most candidates fumble: the detector family, NMS, focal loss, and why mAP rather than accuracy is the metric. Here is the design.

GPUs are scarce and costly, so multi-region HA for LLMs is not simply web-app HA with bigger machines. Capacity, routing, and failover all bend around the GPU constraint. Here is the design.

Classic distributed-systems tradeoffs appear in ML infra with an ML twist: stale features and eventually-consistent reads carry model-accuracy consequences, not just correctness ones. Here is how to reason about it.

Interviewers here want concrete figures, not 'grab a bigger GPU.' Weight memory stays constant, the KV cache scales with traffic, and the order you pull levers in settles the outcome. This walks through the napkin math and the serving stack.

The KV cache is what makes LLM serving hard, and the interviewer wants the mechanics: what it holds, why it caps concurrency, and the tricks that shrink it. This is the answer that shows you grasp decode-time economics.

Decode runs sequentially and is memory-bound, so latency tricks count. What they grade is whether you can explain speculative decoding's draft-and-verify mechanism (and why it stays exact) along with the other levers and when each fits. This is the latency toolkit.

Knowing the algorithms is only half the job. The rest is the serving stack that actually delivers throughput within a latency budget. What they grade is whether you match the server to the workload and name the four knobs that move the needle.

Continuous batching is the biggest single throughput lever in modern LLM serving. The signal is explaining why static batching leaves the GPU idle on variable-length generation, and why PagedAttention is what makes the fix practical.

Under load, a shared inference service has to decide whose request runs now. The signal is queuing with priorities, backpressure, and fairness wired into batching, not first-come-first-served until the service falls over.

A PyTorch model in eager mode is not the fastest form for inference. The signal is knowing that ONNX buys portability while TensorRT-style compilers buy hardware-tuned speed, and precisely which optimizations get you there. Here is the answer.

A newly booted replica is sluggish on its opening requests while weights load and kernels compile, which fights with reactive autoscaling. The signal is identifying where cold start comes from plus the warmup and provisioning remedies.

Running one full fine-tuned model per customer grows GPU count linearly and drains the budget quickly. A method exists to pack hundreds of variants onto a single GPU without sacrificing batching. The interviewer wants to hear how.

Users experience the p99, not the median, and the tail is where serving systems quietly break. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Pinning one GPU per model leaves most of a fleet stranded on idle silicon. Sharing safely is a genuine systems problem with four distinct mechanisms, each suited to a different traffic shape. Here is how to choose.

Launching an LLM service on a guessed GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

One long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill carves that prompt up so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt is re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks align exactly. Here is the hashing and eviction machinery underneath.

Round-robin routing throws away a feature that doubles throughput: cache locality. Designing the router means picking a replica by KV state, not just least-loaded. Here is the two-layer design.

Three batching strategies, three very different latency profiles. Choosing wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

GPU memory runs out well before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can turn into the new bottleneck. Here is the bandwidth math.

FP8 serving is fast and usually fine, until it quietly degrades on the one workload your benchmark did not cover. What counts is knowing what FP8 breaks and how to validate it. Here is the recipe.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to strip it out.

Most models do not fill a GPU, so one model per GPU wastes money. The three sharing mechanisms (MIG, MPS, swapping) differ widely in isolation and overhead. Here is how to pick.

Autoscaling reacts in minutes, but a burst lands in seconds, and the gap is where your tail latency dies. Absorbing bursts is about buffers and shedding, not just adding replicas. Here is the playbook.

When each request thinks for 30,000 tokens, serving swings from compute-bound prefill to memory-bound decode, and the KV cache turns into the resource you genuinely schedule. The levers that governed chat traffic stop being the ones that count.

Anyone can list the engines. The signal is naming the one mechanism that distinguishes each (paged KV blocks, a radix prefix tree, ahead-of-time kernel compilation, quantized CPU inference) and picking from your traffic shape rather than a leaderboard.