profiling

LLM serving has its own metrics, and a single latency number hides the real bottleneck. The signal is separating prefill from decode and treating GPU utilization as a clue, not a verdict. Here is the diagnostic toolkit.

nvidia-smi reading 100% can mask the fact that you are tapping only a fraction of the hardware's real compute. What matters is MFU (useful FLOPs vs peak) and the gap between 'the GPU is busy' and 'the GPU is efficient'.

Paying for accelerators that sit idle half the time is the most common waste in ML training, and the reflex to add more GPUs only makes it worse. The interviewer wants the profiling discipline that pinpoints what is starving them.

The collective communication in distributed training is pure overhead unless it runs while the GPU computes. Hiding it is the difference between 30% and 55% MFU. Here is how the overlap actually works and how you check it on a trace.

When all-reduce is your bottleneck, the culprit is nearly always a misconfigured path: traffic on the wrong link, a dead NIC, or a topology NCCL failed to discover. Here is the methodical way to track down the missing bandwidth.