prefix caching
AI, ML & GenAI interview questions tagged prefix caching, across every topic.
4 questions · 0 unlocked for you
Concepts behind "prefix caching"
The curriculum that explains the ideas these questions test.
Core
Context Compression and Prompt CompactionWhen a prompt is too big, compression is the last lever you should reach for, not the first. Restructuring for a stable cached prefix is bigger and cheaper, and compaction (summarizing old turns, dropping stale tool output, reranking so you send five good chunks instead of twenty mediocre ones) covers most of the rest. Hard compression trades a measurable accuracy tax for tokens, and it can raise your bill by destroying cache hits. AI, ML, and GenAI interviews probe this because candidates reach for the clever technique before the free one.🧠 Foundations of LLMs & GenAISign in
Advanced
Disaggregated Prefill/Decode and Prefix CachingLLM inference has two phases with opposite hardware profiles: prefill is compute-bound (it works through the whole prompt in parallel) while decode is memory-bandwidth bound (one token at a time). Running both on the same GPU pool makes them compete, so long prefills stall ongoing decodes and you miss either the time-to-first-token or the time-per-output-token SLO. Disaggregation places them on separate GPU pools and moves the KV cache between them, and prefix caching reuses KV for shared prompt prefixes. AI, ML, and GenAI engineer interviews probe it because it is the current frontier of serving architecture and a real latency-SLO tradeoff.🖥️ ML Infrastructure & Serving🔒 Premium