kv cache
AI, ML & GenAI interview questions tagged kv cache, across every topic.
18 questions · 2 unlocked for you
Concepts behind "kv cache"
The curriculum that explains the ideas these questions test.
Foundational
The Context WindowThe context window is the largest number of tokens a model can attend to at once, prompt plus generation. It is capped by attention's quadratic cost, the KV cache's linear memory growth, and the length the model trained on. A bigger window is neither free nor uniformly useful (models lose information in the middle), which is why retrieval often beats cramming everything into context. AI, ML, and GenAI engineer interviews probe it because it drives cost, latency, and the RAG-vs-long-context decision.🧠 Foundations of LLMs & GenAI
Foundational
GPU Memory and the Serving StackServing an LLM is largely a memory problem: the GPU has to hold the model weights along with a KV cache that scales with sequence length and batch size, and inference divides into a compute-bound prefill and a memory-bandwidth-bound decode. Understanding the memory math (weights plus KV cache), why decode is bandwidth-bound, and the levers (quantization, batching, paged attention) is the bedrock of LLM serving. AI, ML, and GenAI engineer interviews probe it because 'will this model fit and how fast will it run?' is a recurring production question.🖥️ ML Infrastructure & Serving
Core
Attention and Self-AttentionAttention casts each token as a query, key, and value, scores every query against every key, softmaxes those scores into weights, and returns the weighted sum of values, so each token draws information from the others. Self-attention does this within one sequence. The all-pairs scoring is why cost grows with the square of sequence length, which then explains context limits, long-prompt expense, and the KV cache. AI, ML, and GenAI engineer interviews probe it because it ties architecture to cost and latency in one mental model.🧠 Foundations of LLMs & GenAISign in
Core
Attention Variants: MHA, MQA, and GQAMulti-head attention gives every query head its own key and value heads, which is expressive but leaves the KV cache large and memory-bandwidth hungry at decode time. Multi-query attention shares one key-value head across all query heads to shrink the cache sharply, and grouped-query attention sits between them by sharing key-value heads across small groups. AI, ML, and GenAI engineer interviews probe this because it is the cleanest example of trading model quality against serving memory and throughput, and it explains why frontier models standardized on GQA.🧠 Foundations of LLMs & GenAISign in
Core
The KV CacheIn autoregressive decoding a model would recompute attention over the whole history at every step; the KV cache keeps each token's key and value vectors so a new token only attends and never recomputes. Compute is saved, but the cost shifts to memory: the cache grows with sequence length times batch size and usually turns into the binding constraint in serving. AI, ML, and GenAI interviews probe it because it explains why long contexts are costly to serve, why throughput (not model speed) is often the limit, and why MQA/GQA and PagedAttention exist.🧠 Foundations of LLMs & GenAISign in
Advanced
Disaggregated Prefill/Decode and Prefix CachingLLM inference has two phases with opposite hardware profiles: prefill is compute-bound (it works through the whole prompt in parallel) while decode is memory-bandwidth bound (one token at a time). Running both on the same GPU pool makes them compete, so long prefills stall ongoing decodes and you miss either the time-to-first-token or the time-per-output-token SLO. Disaggregation places them on separate GPU pools and moves the KV cache between them, and prefix caching reuses KV for shared prompt prefixes. AI, ML, and GenAI engineer interviews probe it because it is the current frontier of serving architecture and a real latency-SLO tradeoff.🖥️ ML Infrastructure & Serving🔒 Premium