fsdp

The interviewer is checking that you understand how a model too large for any one GPU still gets trained, and can run the optimizer-state memory math that justifies sharding. This covers the parallelism taxonomy and the 16-bytes-per-parameter figure.

FSDP and ZeRO let you train models too large for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

ZeRO and FSDP both shard training state across data-parallel ranks, but the trick is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.