training

Distillation gives you most of a frontier model's quality for a fraction of the serving cost. The signal is naming the three LLM-specific variants and knowing which one holds up against a closed API. Here is the answer.

Distillation is not a single method. The signal is knowing when to match logits versus train on generated text, why on-policy distillation beats off-policy, and how reasoning models are distilled.

The proportions of web, code, books, and math in pretraining quietly set downstream skills. The signal is knowing how mixtures get chosen, why upsampling helps, and how methods like DoReMi automate it.

Continued pretraining adds domain knowledge that fine-tuning cannot, yet done carelessly it wrecks general ability. The signal is the replay ratio, learning-rate rewarming, and how you measure forgetting.

FP8 underpins modern H100-class training and serving. What matters is knowing the two FP8 variants, why a floating format beats INT8 on dynamic range, and where scaling still counts.

Many state-of-the-art training runs hold a shadow copy of the weights that averages the trajectory, then serve that rather than the final step. It's a few lines and a genuine accuracy gain. Here it is.

Nearly every modern training run relies on warmup-then-cosine, and getting it wrong destabilizes early training or squanders the tail. It's a closed-form function of the step. Here is the implementation and the why.

Optimizer questions probe whether you grasp what each one adapts and the subtle AdamW fix that the whole field now relies on. The signal is the per-parameter adaptivity story plus why tying weight decay to Adam was a bug. Here is that answer.

Backprop is the algorithm that makes deep learning trainable, and the interviewer wants the chain-rule mechanics, not just 'it computes gradients.' What interviewers reward is the forward-then-backward flow and why it is efficient. Here is the answer.

BatchNorm is one of the most-asked deep-learning questions, and the trap is the train/inference difference. What interviewers reward is what it normalizes, why it stabilizes and speeds training, and why it switches to running statistics at inference. Here is that answer.

The learning rate is the single hyperparameter that can NaN your run at step 50 or stall it indefinitely. Modern training schedules it rather than pinning it to one value. Interviewers want you to know precisely what warmup and decay each give you. Here is the answer.

'The model won't train' comes down to a short list of usual suspects, each with a distinctive loss-curve signature. The shape of the curve names the bug before you touch a single hyperparameter. Here is how to read it.

The notion that models, like students, learn better starting from easy examples is intuitive, and it sometimes works and sometimes does nothing. Knowing when it pays off is the real signal. Here is the honest answer.

Mixed precision is routine at scale, and the interviewer wants the numerics: why FP16 needs loss scaling, why BF16 mostly does not, and what remains in FP32. What they grade is whether you understand dynamic range vs precision. This is that answer.

Activations, not just weights, can dominate training memory, and gradient checkpointing is the standard remedy. The signal is the precise trade: recompute activations in the backward pass rather than storing them. Here is the answer.

Batch size is a training knob whose effects on speed, memory, and generalization are non-obvious. The signal is the large-batch tradeoffs and the learning-rate-scaling and gradient-accumulation tricks that make scaling actually work.

Past FP16/BF16, FP8 sits on the next rung of the precision ladder, and the newest tensor cores compute on it natively. What counts is knowing the two formats, what FP8 really buys, and why dynamic range demands careful scaling.

Paying for accelerators that sit idle half the time is the most common waste in ML training, and the reflex to add more GPUs only makes it worse. The interviewer wants the profiling discipline that pinpoints what is starving them.

At billion-parameter scale, training can be running smoothly and then the loss jumps and never returns, burning a fortune in compute. The causes and the playbook are familiar to the few who've done it. Here it is.

Full activation checkpointing saves memory but costs a flat 30% extra compute. Selective recomputation wins most of that back by recomputing only the cheap, memory-heavy operations. Here is how to pick what to recompute.

FP8 can nearly double training throughput over BF16, but with only a few mantissa bits the numerics leave no slack. Per-tensor scaling and a selective recipe are what get it to converge.