96How do you serve a Mixture-of-Experts model efficiently, and what makes expert parallelism hard?▼expertDeepSeekMistralNVIDIA2 replies◆ premiumMoE trims compute but is awkward to serve: experts have to be sharded, tokens routed across devices, and batches balanced. What matters is the all-to-all communication and the load-imbalance problem, not the training story.Open full answer →
59What makes Mixture-of-Experts models hard to train, and how do you handle routing, load balance, and all-to-all?▼expertGoogle DeepMindMistralMeta1 replies◆ premiumMoE gives you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle bring failure modes dense models never see. Here is the training playbook.Open full answer →
