memory bandwidth

Picking GPUs is a genuine applied decision, and 'get the biggest one' misses the point. The signal is knowing which spec binds your workload (VRAM, bandwidth, interconnect) and the train-versus-serve difference that flips the answer.

The counterintuitive reality of LLM serving: token generation is capped by how quickly you can read weights out of memory, not by arithmetic. Once that clicks, the entire optimization menu follows from one number.