fault tolerance

At thousand-GPU scale, hardware failure is routine rather than rare, and a run that cannot resume burns weeks. The signal is checkpointing strategy, deterministic resume, and holding lost work to a minimum. Here is the fault-tolerant design.

On a large training run a node will drop, and the interesting question is not whether but how many GPU-hours vanish when it does. Checkpoint cadence, sharded writes, and quick restart settle that.