continuous batching

In naive LLM serving most GPU memory bleeds away to KV-cache fragmentation. PagedAttention adapts virtual memory paging to win it back. What matters is explaining the fragmentation problem and how blocks resolve it.

The whiteboard question that spans the entire stack, from BPE to the sampler. What separates a strong answer is knowing that prefill and decode stress opposite hardware resources, and that nearly every serving decision you will ever make falls out of that one fact.

Continuous batching is the biggest single throughput lever in modern LLM serving. The signal is explaining why static batching leaves the GPU idle on variable-length generation, and why PagedAttention is what makes the fix practical.

One long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill carves that prompt up so decode keeps flowing. Here is the mechanism and the knob that controls it.

Round-robin routing throws away a feature that doubles throughput: cache locality. Designing the router means picking a replica by KV state, not just least-loaded. Here is the two-layer design.

Three batching strategies, three very different latency profiles. Choosing wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

When each request thinks for 30,000 tokens, serving swings from compute-bound prefill to memory-bound decode, and the KV cache turns into the resource you genuinely schedule. The levers that governed chat traffic stop being the ones that count.