GPU Memory and the Serving Stack

TL;DR: LLM serving is governed by memory. The GPU has to store the model weights (parameters times bytes-per-parameter) alongside the KV cache, which scales with sequence length times batch size and frequently outgrows the weights at scale. Inference runs in two phases: prefill (process the whole prompt, compute-bound) and decode (generate token by token, memory-bandwidth-bound). Grasping this memory math and which phase limits you, together with the levers (quantization, batching, paged attention), is the bedrock of serving.

The memory math

The opening question is "does it fit?" Two consumers draw it down:

• Weights: parameters times bytes per parameter. A 70B model is ~140GB in FP16, ~35GB in INT4. This sets the floor.
• KV cache: every token in every active request holds a key and value vector in every layer, so it scales with layers x heads x head-dim x sequence-length x batch. At long context and high concurrency it can exceed the weights and is what caps how many requests you can run at once.

Worked example: does a 70B fit on one A100 80GB?

Take Llama-2-70B (80 layers, hidden 8192, GQA with 8 KV heads, head-dim 128) on a single 80GB A100.

• Weights in FP16: 70e9 params x 2 bytes = 140GB. It does not fit, full stop. Quantize to INT4: 70e9 x 0.5 = 35GB of payload, plus scales and any tensors you held at higher precision. Call it 35GB and remember you rounded down.
• KV cache per token: 2 (K and V) x 80 layers x 8 KV heads x 128 head-dim x 2 bytes = 327,680 bytes ≈ 0.33MB/token. (GQA matters here: with 64 query heads instead of 8 KV heads the cache would be 8x larger.)
• Budget the headroom: 80GB minus 35GB leaves 45GB, but that is an upper bound, not a cache budget. Activations, workspace, allocator reservations and safety headroom come out first, so reserve a few GB and work with ~40GB: 40GB / 0.33MB ≈ 122,000 tokens of KV. That is a 4k-token request times ~29 concurrent users. The unqualified 45GB figure buys about 136,000 tokens and 33 users, and the gap between those two answers is exactly the overhead a candidate forgot.
• On long context: 128k tokens of KV is ~43GB on its own, so a single request at that length would eat the whole budget. Note that Llama-2-70B's own context is 4k, per Meta's model card. If you want to reason about 128k you have to name a model that supports it; memory capacity never grants context length by itself.

The takeaway an interviewer wants: weights set whether the model loads at all, but the KV cache sets how many users you can serve concurrently, and at long context the cache, not the weights, is what you run out of.

Prefill vs decode

Inference is two phases with opposite resource profiles:

• Prefill: process the entire prompt in one pass to produce the first token and fill the KV cache. Lots of parallel matrix math, so it is compute-bound, and it drives time-to-first-token.
• Decode: generate tokens one at a time, each reading the whole KV cache. Little compute per step, dominated by streaming weights and KV from memory, so it is memory-bandwidth-bound, and it drives inter-token latency.

This is why throughput is often limited by memory (bandwidth and capacity), not raw compute, and why GPU utilization can look "busy" while you are actually waiting on memory.

The serving levers

Because memory is the constraint, the optimizations target it:

• Quantization: fewer bits shrink weights and the KV cache and cut the bandwidth decode must read.
• Continuous batching: serve many requests together to raise throughput (decode is cheap per token; batching amortizes the weight reads).
• PagedAttention: manage the KV cache in pages to avoid fragmentation and fit more concurrent sequences.
• MQA/GQA shrink the KV cache structurally; FlashAttention removes attention's memory-IO overhead.

Specialized serving stacks (vLLM, TGI, TensorRT-LLM) bundle these.

Why interviewers probe this

"Will this model fit on our GPUs and how fast will it run?" is a constant production question, and the answer is memory math plus the prefill/decode distinction. A strong response computes the weight and KV-cache memory, identifies that decode is bandwidth-bound (so throughput is memory-limited), and names the levers (quantization, batching, paged attention). The reserved follow-up is usually "your p99 inter-token latency spikes under load, why?" The strong answer: decode is bandwidth-bound, so as the batch grows the per-step memory traffic grows with it, and once KV-cache capacity is exhausted the scheduler preempts or queues requests, which shows up as tail latency. That is the foundation every other serving optimization builds on.

Common misconceptions

• "Serving is compute-bound." Decode is memory-bandwidth-bound, and capacity (KV cache) often caps concurrency before compute does.
• "Weights are the only memory cost." The KV cache can exceed the weights at long context and high batch.
• "Prefill and decode are the same." Prefill is compute-bound (sets TTFT); decode is bandwidth-bound (sets inter-token latency).
• "High GPU utilization means it is efficient." It can be busy waiting on memory; measure throughput, not just utilization.

Key takeaways

• GPU serving is a memory problem: weights plus a KV cache that grows with sequence length times batch.
• Prefill is compute-bound (time-to-first-token); decode is memory-bandwidth-bound (inter-token latency).
• Throughput is usually limited by memory (bandwidth and KV-cache capacity), not raw compute.
• The levers are quantization, continuous batching, paged attention, and MQA/GQA/FlashAttention.