37You doubled the GPUs but training barely got faster. Why doesn't distributed training scale linearly?▼hardMetaNVIDIAGoogle1 replies◆ premiumLinear scaling is the marketing figure; the actual curve bends early for reasons rooted in physics, not bugs. Here is where the speedup leaks and how to recover it.Open full answer →
55How do you combine data, tensor, and pipeline parallelism (3D parallelism) to train a 175B model?▼expertNVIDIAMicrosoftMeta1 replies◆ premiumNo single parallelism dimension trains a 175B model by itself. The art is stacking three of them onto the right hardware so the expensive communication stays on the fast links. Here is how to size each axis.Open full answer →
