throughput

Batch scoring a billion items is a throughput-and-cost problem, the reverse of low-latency serving. Every choice that harms latency works in your favor here. Here is how to design for dollars-per-million-documents.

Knowing the algorithms is only half the job. The rest is the serving stack that actually delivers throughput within a latency budget. What they grade is whether you match the server to the workload and name the four knobs that move the needle.

Continuous batching is the biggest single throughput lever in modern LLM serving. The signal is explaining why static batching leaves the GPU idle on variable-length generation, and why PagedAttention is what makes the fix practical.

LLM inference has two phases with opposite resource profiles, and co-locating them lets a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

nvidia-smi reading 100% can mask the fact that you are tapping only a fraction of the hardware's real compute. What matters is MFU (useful FLOPs vs peak) and the gap between 'the GPU is busy' and 'the GPU is efficient'.

CPU-based autoscaling that suits a web tier quietly breaks on GPU inference: the signal is wrong, and replicas need minutes to warm. The interviewer wants the signals you genuinely scale on and how you mask the cold start.

Three batching strategies, three very different latency profiles. Choosing wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

Raw tokens per second is gameable: crank the batch size and the dashboard looks great while every request misses its latency target. Goodput is the throughput that actually meets your SLOs, and it is the number you size and autoscale on.