moe

MoE explains why some frontier models carry huge parameter counts yet serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly conceal.

Attention gets all the attention, but the feed-forward layers hold most of a transformer's weights and handle much of the per-token 'knowledge' work. What shows depth is knowing what the FFN computes and why it dominates the parameter count.

MoE trims compute but is awkward to serve: experts have to be sharded, tokens routed across devices, and batches balanced. What matters is the all-to-all communication and the load-imbalance problem, not the training story.

Temperature 0 is not the same as deterministic, and the reason lives in the GPU kernels, not the sampler. What gets scored is naming the batch-invariance problem and knowing which fixes are real versus placebo.

MoE gives you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle bring failure modes dense models never see. Here is the training playbook.