Normalization in Transformers: LayerNorm, RMSNorm, Pre-Norm and Post-Norm

Normalization keeps activations in a range where a deep stack can actually train. LayerNorm re-centers and re-scales each token vector; RMSNorm drops the mean subtraction entirely and only divides by the root mean square, which costs nothing in quality while removing about half the elementwise arithmetic and one learned parameter tensor. Where you put the norm matters more: pre-norm leaves the residual path clean and is why 60-plus-layer stacks train at all, while post-norm can end slightly better but fights you the whole way. AI, ML, and GenAI engineer interviews probe it because it is the difference between a model that converges and one that diverges at step 300.