distillation

Synthetic data is how teams manufacture training examples when real data runs short, and it comes with sharp failure modes. What shows depth is pinning down when it helps, how to keep quality up, and why training on model output over successive generations narrows the distribution.

Distillation gives you most of a frontier model's quality for a fraction of the serving cost. The signal is naming the three LLM-specific variants and knowing which one holds up against a closed API. Here is the answer.

Distillation gives you a small fast model that retains most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

Pruning strips redundant weights to make a model smaller and sometimes faster. The signal is the structured-versus-unstructured divide and why only structured pruning dependably cuts latency on commodity hardware.

Sampling cost factors cleanly into steps, forward passes per step, and cost per step, and each factor has a different lever. Two of the biggest wins are free, and most teams take neither.

The technique works and everybody knows it works. The interview is not testing whether you can generate the training set; it is testing whether you know what you are agreeing to when you do.