TL;DR: At FP16 a 70B model weighs ~140GB, too much for a single 80GB GPU; either split it with tensor parallelism over 2+ GPUs or quantize to INT8/INT4 (~70GB / ~35GB) so one card holds it. After that the binding limit becomes the KV cache, which scales with batch × sequence length and is what really caps concurrency. Serve it with continuous batching and PagedAttention (vLLM), MQA/GQA, and quantization.
How to approach it
Work the numbers aloud, keeping weights (constant) distinct from the KV cache (which climbs with load). Name the order you pull levers: get the weights to fit first (quantization or tensor parallelism), then push throughput (continuous batching, paged KV cache), then shave tail latency. Most candidates ignore the KV cache and cite only weight memory, yet that cache is the part that really bounds batch size.
A strong answer
Weights. Parameters × bytes-per-param. FP16/BF16 is 2 bytes, so 70B is about 140GB. That exceeds an 80GB H100/A100, so either tensor-parallel shard across at least 2 GPUs (split each layer's matrices column/row-wise, sync with all-reduce over NVLink) or quantize: INT8 lands near 70GB (fits one 80GB card with room for cache), INT4 near 35GB. Training is a different beast: for the standard mixed-precision Adam recipe (BF16 weights and gradients plus FP32 master weights, momentum and variance) that is 16 bytes/param, roughly 1.1TB before activations, which is why training shards with FSDP/ZeRO. Quote 16 bytes as that recipe's number rather than a law: a different optimizer or a sharded/8-bit state changes it. Inference carries the weights and the KV cache, plus activations, workspace and allocator reservations, which are small next to those two but not zero.
KV cache, the real limiter. Each generated token caches a key and value per layer per KV head, so the general formula is 2 × layers × kv_heads × head_dim × seq_len × batch × bytes. Use hidden in place of kv_heads × head_dim only for multi-head attention, where they coincide; on a GQA model with 64 query heads and 8 KV heads, hidden overstates the cache by 8x, which is the single most common arithmetic slip in this question. For Llama-2-70B (80 layers, 8 KV heads, head-dim 128, FP16) that is 0.33MB per token, so a few thousand tokens across a moderate batch is tens of GB, often dwarfing leftover weight headroom. KV cache, not weights, is what caps concurrency.
| Lever | What it buys | When it bites back |
|---|---|---|
| INT8/INT4 quantization | Fit on one GPU, faster decode | Small quality hit at INT4; validate on tasks |
| Tensor parallelism | Fit when one GPU is too small | An all-reduce per layer, so it wants NVLink; on PCIe the communication can dominate |
| Continuous batching | Biggest throughput win | Larger batch raises time-to-first-token |
| PagedAttention | Kills KV fragmentation, raises batch | Needs a paged-aware kernel (vLLM) |
| MQA/GQA | Shrinks KV cache several-fold | Must be in the model architecture |
Throughput versus latency is the central tradeoff: larger batches raise tokens/sec but raise per-request time-to-first-token, so set the batch policy to the SLO rather than maxing one number.
Key takeaways
- Weights are fixed (2 bytes/param at FP16); the KV cache grows with batch × sequence and is the true concurrency ceiling.
- Lever order: fit weights (quantize or TP), then throughput (continuous batching, PagedAttention), then latency.
- Quantize before reaching for tensor parallelism if a single quantized GPU fits; TP pays interconnect tax.
- Decode is memory-bandwidth-bound, so fewer bytes per weight and a smaller KV cache directly speed generation.
What interviewers probe next
- "Prefill vs decode?" Prefill processes the whole prompt in parallel (compute-bound); decode is one token at a time (memory-bandwidth-bound), which is why KV cache and bandwidth dominate steady-state cost.
- "INT4 quality risk?" Usually a small perplexity hit with AWQ/GPTQ; validate on your eval set, and keep sensitive layers higher precision if a regression shows up.
- "Speculative decoding?" A small draft model proposes tokens the big model verifies in parallel, cutting latency when acceptance is high.
Common mistakes
- Quoting only weight memory and forgetting the KV cache, which is what actually limits batch size.
- Using FP32 numbers for inference; inference runs in FP16/BF16 or lower.
- Reaching for tensor parallelism before quantization when a single quantized GPU would do, paying needless interconnect overhead.
- Conflating training memory (16 bytes/param with optimizer states) with inference memory (weights plus cache).