cross entropy

Perplexity is the classic LM metric. What matters is knowing it is just exponentiated cross-entropy, what it intuitively captures, and why low perplexity does not mean a good product.

A deceptively easy ML-coding ask. Writing exp/sum is trivial; the signal is the max-subtraction trick and the log-sum-exp form that stop it overflowing. Here is the stable implementation and the reason the naive version breaks.

A from-scratch ML-coding staple that tests whether you really know the math you rely on. The signal is the clean gradient (it reduces to Xᵀ(ŷ - y)/n), numerical stability, and vectorization. Here is the implementation and the details interviewers push on.

The loss sets what the model optimizes, and the wrong choice silently sinks it. What counts is matching the loss to the task and data (outliers, imbalance), rather than reflexively reaching for MSE or cross-entropy.

These four quantities sit under cross-entropy loss, decision-tree splits, distillation, and the KL penalty in RLHF. What matters is deriving them from one another and pointing to precisely where each turns up in a real training loop.

Cross-entropy is where you start, not where you finish. The signal is fitting the loss to the task structure: focal for detection's background flood, the IoU family for box overlap, Dice for imbalanced masks. Here is how to reason about it.