prefill

The whiteboard question that spans the entire stack, from BPE to the sampler. What separates a strong answer is knowing that prefill and decode stress opposite hardware resources, and that nearly every serving decision you will ever make falls out of that one fact.

LLM inference has two phases with opposite resource profiles, and co-locating them lets a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

One long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill carves that prompt up so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt is re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks align exactly. Here is the hashing and eviction machinery underneath.

An image is not one input, it is a large prompt. Once you internalize that image tokens dominate prefill, the cost levers stop being model choices and start being resolution, tiling, and cache placement.