18What are the collective communication operations (all-reduce, all-gather, reduce-scatter) in distributed training?▼hardNVIDIAOpenAIGoogle2 replies○ sign inDistributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data travels. The signal is what each one does and which parallelism strategy relies on it. Here is the answer.Open full answer →
38Your distributed training job hangs or crashes intermittently. How do you debug it?▼hardMetaNVIDIAOpenAI2 replies◆ premiumA 256-GPU job that freezes with no error at 3am is a special kind of pain. The causes come from a short, recurring list. Here is the systematic way to identify which one hit you.Open full answer →
62Your training collectives are slow. How do you debug the NCCL/interconnect path and find where bandwidth is lost?▼hardNVIDIAMetaMicrosoft2 replies◆ premiumWhen all-reduce is your bottleneck, the culprit is nearly always a misconfigured path: traffic on the wrong link, a dead NIC, or a topology NCCL failed to discover. Here is the methodical way to track down the missing bandwidth.Open full answer →
