← 🖥️ ML Infrastructure & ServingNEXT IN ML INFRASTRUCTURE & SERVINGSpeculative Decoding→
Advanced
Disaggregated Prefill/Decode and Prefix Caching
LLM inference has two phases with opposite hardware profiles: prefill is compute-bound (it works through the whole prompt in parallel) while decode is memory-bandwidth bound (one token at a time). Running both on the same GPU pool makes them compete, so long prefills stall ongoing decodes and you miss either the time-to-first-token or the time-per-output-token SLO. Disaggregation places them on separate GPU pools and moves the KV cache between them, and prefix caching reuses KV for shared prompt prefixes. AI, ML, and GenAI engineer interviews probe it because it is the current frontier of serving architecture and a real latency-SLO tradeoff.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
ML Infrastructure & GPUsHow does prefix caching work internally in an LLM server, and when does it actually help?→ML Infrastructure & GPUsHow do you offload the KV cache to CPU or NVMe, and when is it worth the bandwidth hit?→ML Infrastructure & GPUsServe a 70B-parameter model with high throughput. Do the memory math and name the optimizations.→ML Infrastructure & GPUsExplain the KV cache: prefill vs decode, why it grows, and how MQA/GQA and PagedAttention help.→ML Infrastructure & GPUsWhat is chunked prefill, and how does it stop long prompts from stalling decode?→ML Infrastructure & GPUsDesign the request router and scheduler for a multi-replica LLM serving cluster.→
COMPANIES THAT ASSUME THIS