checkpointing

An hour-long agent run is a distributed workflow wearing an LLM hat. The signal is designing serializable state with step-level checkpoints, and knowing why resume is easy while replaying a side effect is not.

At thousand-GPU scale, hardware failure is routine rather than rare, and a run that cannot resume burns weeks. The signal is checkpointing strategy, deterministic resume, and holding lost work to a minimum. Here is the fault-tolerant design.

Spot GPUs frequently run 60-90% cheaper, and they disappear with two minutes' notice. The savings hold only if a preemption costs you minutes rather than the whole run. Here is how to make that the case.

On a large training run a node will drop, and the interesting question is not whether but how many GPU-hours vanish when it does. Checkpoint cadence, sharded writes, and quick restart settle that.