GPU Spot Training: Test Checkpoint Recovery Before Scaling

Interruptible or spot GPU capacity can reduce the price of a training hour, but the job may stop before it finishes. Savings depend on how much useful work survives interruption and how reliably the replacement resumes. A cheap instance that repeatedly loses progress can be more expensive per completed experiment than a stable one.

Checkpointing is therefore part of the training design. Test it on a small run before increasing model size or duration. Provider interruption notices are useful signals, but periodic durable checkpoints should not depend on receiving a last-minute warning.

Identify complete training state

A checkpoint may need model parameters, optimizer state, scheduler state, training step, random-state information, data-loader position, gradient-scaler state, and configuration. The exact requirement depends on the framework and algorithm. Saving only weights can support inference while failing to resume training correctly.

PyTorch's saving and loading guidance distinguishes common model and general checkpoint patterns. Use the framework's documented APIs and verify your particular distributed or mixed-precision workflow. A tutorial example is not proof that every stateful component is covered.

Store progress outside the instance

Use durable storage that survives instance termination. Record run ID, checkpoint generation, step, timestamp, code revision, dataset version, model configuration, and integrity evidence. Protect access and avoid storing secrets inside the checkpoint metadata.

Write to a temporary object or generation, finish the upload, then mark the checkpoint complete through a supported atomic or transactional mechanism. Readers should ignore partial checkpoints. Keep at least one previous known-good generation in case the latest write is corrupt or incompatible.

Choose a checkpoint interval

Balance checkpoint overhead against expected lost work. Short intervals reduce recovery loss but consume storage and I/O. Long intervals improve throughput during a healthy run but can waste substantial computation after interruption. Use measured save and restore times with observed workload characteristics.

Do not claim a universal ideal interval. A small fine-tuning job and a large distributed run have different costs. Include the storage path's bandwidth, serialization time, and coordination overhead. Review the interval when model size or data changes.

Test abrupt termination

Start a small reproducible run, save progress, and terminate the worker through an authorized test method. Launch a replacement, restore the checkpoint, and verify that it continues from the expected state. Test interruption during a save as well as between saves.

Compare resumed behavior with an uninterrupted control using an appropriate tolerance. Exact bit-for-bit equality may not be available for every accelerator and operation, but the chosen correctness criteria should be explicit. Check loss progression, optimizer state, data position, and duplicate sample handling.

A practical fine-tuning example

A team trains an adapter on interruptible capacity. Its first recovery test loads weights but resets the optimizer and scheduler, changing the learning-rate path. The team adds the missing state, pins code and data versions, and stores completed checkpoint generations externally.

A second test interrupts during upload. The replacement rejects the incomplete generation and uses the prior one. The job loses a bounded interval of work and preserves the intended training schedule. Only after this evidence does the team scale the experiment.

Handle provider notices carefully

Amazon EC2 documents spot interruption behavior. Check the relevant provider and instance type because notice, termination, stop, and hibernation behavior can differ. Treat a notice handler as an additional opportunity to preserve state, not the only safeguard.

A handler should stop accepting new work, attempt a safe checkpoint if time permits, mark status, and avoid starting a large upload it cannot complete. Keep the ordinary periodic process authoritative. Test the notice path independently from the abrupt-loss path.

Account for completed-work cost

Track paid compute, interrupted time, lost progress, checkpoint overhead, storage, transfer, restart delay, and failed resumes. Compare cost per completed qualified run, not only advertised hourly price. Availability shortages may delay experiments even when the nominal rate is attractive.

Set a maximum retry or spend budget. An experiment should not restart indefinitely because an automated scheduler treats every interruption as temporary. Preserve failure evidence and require review when the budget or recovery threshold is crossed.

Protect reproducibility

Pin runtime, dependencies, code, model artifact, and dataset. A replacement worker using a newer library may load the checkpoint but change behavior. Keep a manifest and test migration separately when versions change. For distributed training, verify all ranks agree on the completed generation and recovery state.

Verify checkpoint access from the replacement identity before stopping the original worker. A durable object is useless if the new instance lacks permission or cannot reach its region. Test read access, decryption, and integrity without exposing the storage credential in the job log.

Conclusion

Spot training works when interrupted progress is durable and resumption is proven. Save complete state, write checkpoints safely, test abrupt loss, and measure cost per completed experiment. The discount becomes useful only after recovery stops being a guess.