ML Infrastructure & GPUs

Interviewers here want concrete figures, not 'grab a bigger GPU.' Weight memory stays constant, the KV cache scales with traffic, and the order you pull levers in settles the outcome. This walks through the napkin math and the serving stack.

The interviewer is checking that you understand how a model too large for any one GPU still gets trained, and can run the optimizer-state memory math that justifies sharding. This covers the parallelism taxonomy and the 16-bytes-per-parameter figure.

Quantization is the opening move for shrinking and accelerating models, and the interviewer expects more than 'use fewer bits.' What they grade is whether you know what each precision level gains you, why outliers wreck naive quantization, and how you demonstrate quality survived. This lays that out.

A staple at hardware-aware teams. What they look for is grasping that attention's cost is memory traffic rather than FLOPs, and that FlashAttention is an exact, IO-aware reordering rather than an approximation. This is the answer that shows you reason about the memory hierarchy.

The KV cache is what makes LLM serving hard, and the interviewer wants the mechanics: what it holds, why it caps concurrency, and the tricks that shrink it. This is the answer that shows you grasp decode-time economics.

Decode runs sequentially and is memory-bound, so latency tricks count. What they grade is whether you can explain speculative decoding's draft-and-verify mechanism (and why it stays exact) along with the other levers and when each fits. This is the latency toolkit.

Mixed precision is routine at scale, and the interviewer wants the numerics: why FP16 needs loss scaling, why BF16 mostly does not, and what remains in FP32. What they grade is whether you understand dynamic range vs precision. This is that answer.

A hardware-literacy question, particularly at NVIDIA and the labs. What they grade is whether you understand throughput vs latency hardware, the memory hierarchy, and why matrix multiplication maps onto GPUs and TPUs. This is the architecture-aware answer.

Knowing the algorithms is only half the job. The rest is the serving stack that actually delivers throughput within a latency budget. What they grade is whether you match the server to the workload and name the four knobs that move the needle.

Distillation gives you a small fast model that retains most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

Kernel-level questions separate people who have written CUDA from people who have read about it. The signal is commanding the SIMT execution model and the three classic throughput killers, with a concrete fix for each. Here is the low-level answer.

Continuous batching is the biggest single throughput lever in modern LLM serving. The signal is explaining why static batching leaves the GPU idle on variable-length generation, and why PagedAttention is what makes the fix practical.

On-device AI is a genuine product surface (privacy, offline, latency), and it imposes hard constraints. The signal is the quantization plus format plus runtime stack and the tradeoffs you accept under tight memory and battery budgets.

Picking GPUs is a genuine applied decision, and 'get the biggest one' misses the point. The signal is knowing which spec binds your workload (VRAM, bandwidth, interconnect) and the train-versus-serve difference that flips the answer.

Autoscaling GPUs is nothing like autoscaling web servers: GPUs are scarce and expensive, model loading is slow, and the right signal is not CPU. The signal is scaling on queue and GPU metrics, taming cold starts, and the scale-to-zero economics.

Under load, a shared inference service has to decide whose request runs now. The signal is queuing with priorities, backpressure, and fairness wired into batching, not first-come-first-served until the service falls over.

FSDP and ZeRO let you train models too large for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

Distributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data travels. The signal is what each one does and which parallelism strategy relies on it. Here is the answer.

Activations, not just weights, can dominate training memory, and gradient checkpointing is the standard remedy. The signal is the precise trade: recompute activations in the backward pass rather than storing them. Here is the answer.

When a model is too large for one GPU you split the model itself, not just the data. The signal is telling tensor parallelism (split within a layer) apart from pipeline parallelism (split across layers) and matching each to the interconnect. Here is the answer.

LLM serving has its own metrics, and a single latency number hides the real bottleneck. The signal is separating prefill from decode and treating GPU utilization as a clue, not a verdict. Here is the diagnostic toolkit.

A PyTorch model in eager mode is not the fastest form for inference. The signal is knowing that ONNX buys portability while TensorRT-style compilers buy hardware-tuned speed, and precisely which optimizations get you there. Here is the answer.

LLM inference has two phases with opposite resource profiles, and co-locating them lets a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

These four get confused constantly, but they sit at different layers: data processing, distributed training, and general orchestration. The signal is fitting the tool to the workload instead of reaching for the one you know. Here is the answer.

Batch size is a training knob whose effects on speed, memory, and generalization are non-obvious. The signal is the large-batch tradeoffs and the learning-rate-scaling and gradient-accumulation tricks that make scaling actually work.

Almost nobody builds a serving stack by hand. What interviewers watch for is whether you know what frameworks give you (batching, multi-model, GPU scheduling) and why LLM-specific servers even exist when general ones batch already.

OOM is the wall you hit most often in deep learning, and 'buy a bigger GPU' is the weakest reply. What counts is naming the memory consumers, knowing which one dominates, and pairing the right lever with it.

nvidia-smi reading 100% can mask the fact that you are tapping only a fraction of the hardware's real compute. What matters is MFU (useful FLOPs vs peak) and the gap between 'the GPU is busy' and 'the GPU is efficient'.

Past FP16/BF16, FP8 sits on the next rung of the precision ladder, and the newest tensor cores compute on it natively. What counts is knowing the two formats, what FP8 really buys, and why dynamic range demands careful scaling.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. What counts is spotting it as the binding constraint and naming the levers that shrink it without wrecking quality.

A newly booted replica is sluggish on its opening requests while weights load and kernels compile, which fights with reactive autoscaling. The signal is identifying where cold start comes from plus the warmup and provisioning remedies.

Pruning strips redundant weights to make a model smaller and sometimes faster. The signal is the structured-versus-unstructured divide and why only structured pruning dependably cuts latency on commodity hardware.

Running one full fine-tuned model per customer grows GPU count linearly and drains the budget quickly. A method exists to pack hundreds of variants onto a single GPU without sacrificing batching. The interviewer wants to hear how.

CPU-based autoscaling that suits a web tier quietly breaks on GPU inference: the signal is wrong, and replicas need minutes to warm. The interviewer wants the signals you genuinely scale on and how you mask the cold start.

Paying for accelerators that sit idle half the time is the most common waste in ML training, and the reflex to add more GPUs only makes it worse. The interviewer wants the profiling discipline that pinpoints what is starving them.

The counterintuitive reality of LLM serving: token generation is capped by how quickly you can read weights out of memory, not by arithmetic. Once that clicks, the entire optimization menu follows from one number.

Linear scaling is the marketing figure; the actual curve bends early for reasons rooted in physics, not bugs. Here is where the speedup leaks and how to recover it.

A 256-GPU job that freezes with no error at 3am is a special kind of pain. The causes come from a short, recurring list. Here is the systematic way to identify which one hit you.

Spot GPUs frequently run 60-90% cheaper, and they disappear with two minutes' notice. The savings hold only if a preemption costs you minutes rather than the whole run. Here is how to make that the case.

Naive 4-bit quantization can wreck quality, and the reflex to give up and serve fp16 leaves a large speedup unclaimed. The accuracy is usually recoverable. Here is the ladder of fixes.

Users experience the p99, not the median, and the tail is where serving systems quietly break. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Pinning one GPU per model leaves most of a fleet stranded on idle silicon. Sharing safely is a genuine systems problem with four distinct mechanisms, each suited to a different traffic shape. Here is how to choose.

A serving box steady on short prompts falls over the instant a 30k-token request arrives, because KV-cache memory scales with sequence length times batch. Here is how to bound it without crashing.

At billion-parameter scale, training can be running smoothly and then the loss jumps and never returns, burning a fortune in compute. The causes and the playbook are familiar to the few who've done it. Here it is.

Launching an LLM service on a guessed GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

One long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill carves that prompt up so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt is re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks align exactly. Here is the hashing and eviction machinery underneath.

Round-robin routing throws away a feature that doubles throughput: cache locality. Designing the router means picking a replica by KV state, not just least-loaded. Here is the two-layer design.

Three batching strategies, three very different latency profiles. Choosing wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

GPU memory runs out well before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can turn into the new bottleneck. Here is the bandwidth math.

FP8 serving is fast and usually fine, until it quietly degrades on the one workload your benchmark did not cover. What counts is knowing what FP8 breaks and how to validate it. Here is the recipe.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to strip it out.

Most models do not fill a GPU, so one model per GPU wastes money. The three sharing mechanisms (MIG, MPS, swapping) differ widely in isolation and overhead. Here is how to pick.

Autoscaling reacts in minutes, but a burst lands in seconds, and the gap is where your tail latency dies. Absorbing bursts is about buffers and shedding, not just adding replicas. Here is the playbook.

No single parallelism dimension trains a 175B model by itself. The art is stacking three of them onto the right hardware so the expensive communication stays on the fast links. Here is how to size each axis.

ZeRO and FSDP both shard training state across data-parallel ranks, but the trick is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.

Full activation checkpointing saves memory but costs a flat 30% extra compute. Selective recomputation wins most of that back by recomputing only the cheap, memory-heavy operations. Here is how to pick what to recompute.

The collective communication in distributed training is pure overhead unless it runs while the GPU computes. Hiding it is the difference between 30% and 55% MFU. Here is how the overlap actually works and how you check it on a trace.

MoE gives you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle bring failure modes dense models never see. Here is the training playbook.

RLHF with PPO is not one model training, it is four models in the loop at once, three of them on the GPU during every step. The memory math and the generation bottleneck are what catch people out.

On a large training run a node will drop, and the interesting question is not whether but how many GPU-hours vanish when it does. Checkpoint cadence, sharded writes, and quick restart settle that.

When all-reduce is your bottleneck, the culprit is nearly always a misconfigured path: traffic on the wrong link, a dead NIC, or a topology NCCL failed to discover. Here is the methodical way to track down the missing bandwidth.

FP8 can nearly double training throughput over BF16, but with only a few mantissa bits the numerics leave no slack. Per-tensor scaling and a selective recipe are what get it to converge.

Data, tensor, and pipeline parallelism each leave a single sequence's activations on one device, so training past 200k tokens hits a wall none of them can clear. The fourth axis shards the sequence itself, and the interview turns on the communication math.

When each request thinks for 30,000 tokens, serving swings from compute-bound prefill to memory-bound decode, and the KV cache turns into the resource you genuinely schedule. The levers that governed chat traffic stop being the ones that count.

Most answers optimize the training run. For any model that gets used, inference dominates lifetime energy, and the biggest carbon lever is the serving stack. Measure first, then cut the thing that is actually large.

Anyone can list the engines. The signal is naming the one mechanism that distinguishes each (paged KV blocks, a radix prefix tree, ahead-of-time kernel compilation, quantized CPU inference) and picking from your traffic shape rather than a leaderboard.

Raw tokens per second is gameable: crank the batch size and the dashboard looks great while every request misses its latency target. Goodput is the throughput that actually meets your SLOs, and it is the number you size and autoscale on.

An image is not one input, it is a large prompt. Once you internalize that image tokens dominate prefill, the cost levers stop being model choices and start being resolution, tiling, and cache placement.

Sampling cost factors cleanly into steps, forward passes per step, and cost per step, and each factor has a different lever. Two of the biggest wins are free, and most teams take neither.