Spot and Preemptible GPUs for Training

Cloud providers offer spare or interruptible compute under names such as Spot or preemptible instances. Availability, interruption notice, replacement behavior, and billing differ by provider and resource type. A GPU allocated at a lower rate can be reclaimed when the provider needs capacity, so the job must tolerate losing a worker or whole instance. Training jobs can use this capacity when they save consistent checkpoints to durable storage. A checkpoint may include model weights, optimizer state, learning-rate scheduler state, random number generators, and progress counters. Saving only weights can resume inference but may not resume the same training trajectory. Checkpoint frequency trades storage and pause overhead against work lost after interruption. Interruption handling should be tested. If the platform emits a notice, the process can stop safely, finish or cancel in-flight work, write a checkpoint, and exit. The notice window and signal format are provider-specific and may not always be available. A scheduler then requeues the job on any compatible capacity. Distributed training requires coordinating ranks and writing a consistent checkpoint, or a single worker failure may leave other processes waiting. Spot capacity can be a poor fit for a short job whose startup dominates, a latency-critical online endpoint, or a training job that cannot checkpoint. Mixed fleets and fallback capacity can improve availability, but require compatibility across GPU memory, drivers, and frameworks. Keep dependencies and model artifacts accessible after an instance disappears. Compare effective cost per successful run, not just the hourly rate. Include interruption frequency, checkpoint writes, restart time, idle time, and data egress. Use provider-specific pricing and interruption documentation, set budgets, and retain a recovery path. Lower-cost capacity is useful when its volatility matches the workload's tolerance.