mixture of experts

Both are conditional compute, but one scales parameters and the other scales depth. The signal is knowing which axis each routes along and why that shifts the FLOP story. Here is the sharp version.

MoE gives you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle bring failure modes dense models never see. Here is the training playbook.