iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A resilient GPU backup plan starts with two numbers for each training workload: the maximum work you can afford to lose (recovery point objective, or RPO) and the maximum time you can wait before training resumes (recovery time objective, or RTO). Keep rapidly written checkpoints near the accelerators, copy validated checkpoints asynchronously to durable object storage in another region, and add a second cloud only when portability or provider-outage risk justifies the extra complexity.
Protecting a file called checkpoint.pt is not enough. A restart also needs optimizer and scheduler state, random seeds, the exact data position, tokenizer and configuration files, container images, source code, package locks, infrastructure definitions, dataset manifests, and references to secrets. The plan below turns those requirements into an operational design you can test and measure.
Start with workload-specific RPO and RTO
Do not choose a backup product before defining recovery objectives. A model that trains for six hours between checkpoints has a very different risk profile from a fine-tuning job that can be restarted in ten minutes. Set objectives per workload, not per company-wide slogan.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Recovery point objective (RPO)
RPO is the maximum amount of completed training you are willing to lose after a failure. If the RPO is 15 minutes, your durable checkpoint process must normally leave no more than 15 minutes of progress outside the protected copy. The interval includes checkpoint creation, validation, upload, and replication lag—not just the time between calls to a framework’s checkpoint function.
#1 Best Overall
- Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
- Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
- Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
- Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
- Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C
Recovery time objective (RTO)
RTO is the elapsed time from an incident to useful training computation on replacement GPUs. Include time to obtain capacity, create or select the correct GPU shape, attach storage, pull images, restore files, rebuild the environment, re-establish credentials, launch distributed workers, and verify that metrics continue from the expected step.
Choose the least complex strategy that meets both targets
AWS describes four broad disaster-recovery strategies: backup and restore, pilot light, warm standby, and multi-site active-active. Backup and restore is usually sufficient when an outage can tolerate a longer RTO. Pilot light keeps the essential data and minimal infrastructure ready. Warm standby maintains a scaled-down environment that can expand quickly. Active-active runs in multiple sites and carries the highest cost and operational burden. Select the simplest option that satisfies the measured RPO and RTO; adding a second cloud is not automatically better if your team cannot operate it during an incident.
| Workload situation | Typical design direction | What must be measured |
|---|---|---|
| Experiment jobs that can restart overnight | Frequent local checkpoints plus cross-region object-storage backup | Checkpoint loss and restore duration during a scheduled drill |
| Production training with a same-day deadline | Pilot light or warm standby in a second region | GPU allocation time, image pull time, and distributed-start time |
| Provider-outage or portability requirement | Cross-region copies plus a second cloud with independent credentials and automation | Cross-cloud transfer lag, egress cost, and successful launch without the primary cloud |
Separate what you protect into explicit classes
Use different policies for data that changes at different rates and has different restore requirements. A single bucket with one retention rule makes it difficult to control cost and prove that a recovery is complete.
| Protection class | Examples | Recommended treatment |
|---|---|---|
| Training state | Model parameters, optimizer state, learning-rate scheduler, global step, random seeds, data-shard position, tokenizer and model configuration, and metrics | Checkpoint frequently near the GPUs; replicate only validated, restartable checkpoints to durable storage |
| Datasets | Raw data, transformed shards, labels, and sampling indexes | Version immutable datasets or manifests; replicate according to legal, cost, and rebuild time requirements |
| Software supply chain | Container images, source commits, package-lock files, compiler and driver compatibility information | Copy to a registry or object store available in the recovery cloud; retain digests, not just mutable tags |
| Infrastructure and orchestration | Infrastructure-as-code, scheduler definitions, node-pool settings, launch scripts, and configuration | Store in version control and replicate; keep a tested recovery branch or release artifact |
| Secrets metadata | Secret names, vault paths, key identifiers, certificate locations, and rotation ownership | Back up references and recovery procedures, never plaintext secrets in checkpoints; maintain separately controlled replicas of required keys |
| Experiment records | Run IDs, hyperparameters, metrics, logs, lineage, and evaluation outputs | Export to durable storage so operators can identify the newest valid checkpoint and compare post-recovery metrics |
A checkpoint that lacks the exact software and data provenance may load successfully yet produce different results. Record a manifest beside every durable checkpoint with the source commit, image digest, framework and driver versions, dataset-manifest hash, parallelism layout, and training configuration.
Build a tiered checkpoint write path
Tier 1: a fast staging location beside the GPUs
Write the working checkpoint to local NVMe or a parallel file system when checkpoint latency is the immediate bottleneck. Azure identifies local NVMe or SSD as appropriate when I/O performance is the top priority for intensive AI training. AWS positions FSx for Lustre for high-performance random I/O and sharded checkpoint access. These tiers reduce pause time, but they are not disaster-proof: a terminated VM, failed host, or lost region can destroy them.
Tier 2: durable object storage in the production region
After a checkpoint is complete, flush it to durable object storage in the same region. In SageMaker designs, Amazon S3 is the durable checkpoint target. Treat the upload as a state transition: write to a temporary prefix, validate size and checksums, write the manifest, and then publish a success marker that recovery automation will recognize. Never let a partially uploaded file look like the newest usable checkpoint.
Tier 3: another region and, when warranted, another cloud
Asynchronously replicate the validated object and its manifest to a second region. Google recommends keeping a routine copy in the production region and a disaster-recovery copy in another region, or using dual- or multi-region storage. A second provider is justified when a provider-wide outage, contractual portability requirement, or concentration risk exceeds the cost and complexity of maintaining another environment. Use an independent account or project, separate keys, and recovery automation that does not depend on the primary control plane.
Recommended Free Tools
Rank #2
- NEARLY 2X FASTER THAN OUR PREVIOUS GENERATION(8) – move 1,000 high-res photos in under 60 seconds(6) with up to 2000MB/s transfer speeds(2).
- IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.
- POCKET-SIZED – fits easily in pockets and small bags.
- SPACE TO OWN YOUR AI CONTENT – speed and capacity to download your high-res clips and photo edits.
- 256-BIT AES ENCRYPTION(4) – helps keep private files secure with password protection.
Keep compute out of the checkpoint critical path
Use a background persistence process or buffer so accelerators continue computing while the previous state is copied. Synchronous writes can make every worker wait for network and object-storage latency. Asynchronous writes improve utilization, but they create a window in which the newest local state is not durable. Measure that window against the workload’s RPO and alert when replication falls behind.
Capture a restartable training state
Save all values needed to recreate the next training step, not only neural-network weights:
- Model parameters, including sharded parameters and any exponential-moving-average or auxiliary weights.
- Optimizer accumulators, momentum, mixed-precision loss-scaler state, and learning-rate scheduler position.
- Global step, epoch, gradient-accumulation position, and the exact data-shard or sample offset.
- Random seeds for the framework, data-loader workers, augmentation code, and distributed samplers.
- Tokenizer, vocabulary, preprocessing and model-configuration files.
- Evaluation metrics, configuration, and a machine-readable manifest with checksums.
Distributed jobs often write sharded checkpoints. AWS notes that checkpoint systems must account for sharded writes and reads. Store the shard map and world-size assumptions with the checkpoint. If the recovery cluster has a different worker count, use a format and loader that can remap shards, or document the exact topology required.
Set checkpoint cadence from failure cost, not habit
Frequent checkpoints reduce lost GPU-hours but increase compute overhead, storage I/O, network traffic, and metadata operations. The right cadence depends on the failure rate of the job, checkpoint duration, and the value of the accelerators.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →A practical cadence calculation
- Estimate the hourly cost of the GPUs and the time required to recompute one hour of training.
- Measure the full checkpoint pause or background-copy load at several candidate intervals, such as 5, 15, and 30 minutes.
- Set a maximum acceptable loss window, then add the observed upload and cross-region replication lag to the interval.
- Choose the longest interval that stays within the RPO while keeping checkpoint overhead acceptable.
- Create an emergency checkpoint before planned maintenance, capacity release, or a known preemption window.
For preemptible or spot capacity, the scheduler should checkpoint before a planned interruption when the platform provides notice, then restart from the newest durable marker after replacement capacity is allocated. SageMaker and Azure Machine Learning document checkpoint-based restart after interruption or failure. Keep at least one previous known-good checkpoint so a corrupted newest copy does not become a single point of failure.
Handle distributed writes and reads deliberately
In tensor, pipeline, or data parallel training, having every worker upload every historical checkpoint multiplies storage and network traffic. AWS warns that all-worker downloads can create long startup times and high S3 traffic. Prefer one of these patterns:
- Coordinator write: workers write shards to a shared high-performance file system; a coordinator assembles or records the shard manifest and publishes the durable copy.
- Owner-per-shard write: each worker writes only its assigned shard, with a manifest listing all expected objects and checksums.
- Hierarchical distribution: restore once to a regional file system or cache, then distribute locally to workers instead of making every worker download from object storage.
Test restores with the same and a different worker count. A checkpoint that works only at the original world size is a hidden RTO dependency.
Rank #3
- NONSTOP SPEED: Race through projects with our fastest SSD for creators; Load, edit and transfer with sustained read and write speeds of up to 2,000MB/s1; The T9 Portable SSD performs at high speeds even during longer processes
- DYNAMIC THERMAL GUARD: When you’re elbow-deep in a passion project, the T9 Portable SSD stays cool; Its advanced thermal solution withstands and controls heat to keep the SSD at ideal temperatures—even in heavy use2
- ADAPTS TO EVERY NEED: Whether you’re using a desktop, camera or a gaming console3, count on the T9 Portable SSD for extensive compatibility; It’s a true team player when it comes to heavy-duty application usage or file-saving
- STRONG. SOLID. STABLE: From concept to completion – the T9 Portable SSD gives you the longevity to last through heavy use with one convenient drive that dares to go the distance
- WORKS LIKE MAGIC: Your T9 Portable SSD performs like new with the always up-to-date Magician Software; With firmware updates, extra encryption and continual monitoring of your drive health, it works like a charm
Map the design to AWS, Azure, and Google Cloud
| Cloud | Relevant services and role | Operational notes |
|---|---|---|
| AWS | Amazon S3 for durable SageMaker checkpoints; FSx for Lustre for high-performance random I/O and sharded access | AWS describes S3 as providing 99.999999999% (11 nines) data durability in its 2025 storage-architecture material. Durability does not remove the need for versioning, recovery-region copies, or restore tests. |
| Azure | Local NVMe or SSD for low-latency staging; Azure Backup for vault policies, geo-redundant storage, soft delete, multi-user authorization, lifecycle tiers, and snapshot-based Instant Restore for supported workloads | Confirm that the selected GPU VM, storage type, and backup feature are supported in the target region and subscription. Soft delete helps recover accidental deletion; it is not a substitute for an independent recovery copy. |
| Google Cloud | Managed Lustre for low-latency synchronous checkpointing; Cloud Storage Rapid Bucket for asynchronous, multi-tier checkpointing; Backup and DR for policy-based protection | Backup and DR supports immutable vaults, retention enforcement, cross-region recovery, and cross-project cloning. Keep the recovery project governed separately from the production project. |
Google reported a 6.59% Goodput increase in a 35K-chip TPU v5p workload using multi-tier checkpointing in 2025. Treat that figure as a reported result for that workload and design, not a guaranteed gain for every GPU cluster; benchmark your own checkpoint frequency, model, network, and storage layout.
Replicate across failure domains
Region-level protection
Place the routine durable copy in the production region and replicate to a second region with sufficient GPU capacity. Check capacity before declaring the region a viable recovery target: a bucket may be available while the required accelerator shape is sold out or subject to quota limits.
Provider-level protection
If a provider outage is in scope, maintain a second-cloud copy in a format the other platform can read. Object formats, container images, IAM policies, GPU drivers, networking, and orchestration APIs differ. Portability therefore includes a tested image rebuild or export path, infrastructure-as-code for the recovery cluster, and a scheduler workflow that can select the alternate provider.
Independent control planes
Use separate accounts, projects, administrators, encryption keys, and recovery credentials for the disaster copy. A compromised production identity should not be able to delete every replica. Document who can approve destructive actions and how emergency access is audited.
Apply retention, immutability, and deletion controls
Training data and checkpoints can be targets for ransomware, operator error, or an over-broad cleanup job. Combine several controls:
Free tools Windows power users keep installed
One-click scans. No signup required.
- Enable object versioning or equivalent recovery history.
- Use soft delete or deletion protection for accidental removal.
- Apply immutable or WORM retention to selected recovery checkpoints. Google Cloud Backup and DR documents vaults that enforce write-once, read-many immutability and indelibility.
- Use least-privilege roles: training workers may create new objects but should not shorten retention or delete recovery copies.
- Require multi-person authorization for destructive operations. Azure Backup documents multi-user authorization alongside soft delete.
- Encrypt in transit and at rest, and maintain a recovery procedure for the keys or key references.
Retention should reflect the experiment lifecycle. Keep dense short-term checkpoints for rapid rollback, then retain milestone checkpoints, final model artifacts, and manifests for the period required by governance or reproducibility.
Protect dependencies and provenance
Before a restore drill, verify that each checkpoint can be paired with its environment. Back up container images by immutable digest, source commits, package lock files, compiler and CUDA-compatible driver requirements, infrastructure-as-code, scheduler configuration, dataset manifests, and experiment-tracking exports. Record where secrets are retrieved rather than embedding secret values in a checkpoint.
Rank #4
- High-Speed Performance: Netac ZX20 2TB Portable SSD provides file transfer at speeds up to 2000MB/s read and 1700MB/s write, saving time when storing and transferring data with up to 2TB of storage space for a full library of content in a pocket-sized design
- Endurance that is second to none: Take a trip without any apprehension with Netac powerful NVMe solid-state drives that save time saving and transferring data, suitable for students, photographers, business professionals, content creators and gaming
- Pocket-sized Design: With its small, light plastic housing, the portable NVMe SSD provides compact, handy storage measuring 71.5 x 43.5 x 8 mm (L x W x T) and weighing only 23g, making it portable, slim, and sporty
- Robust Design: Netac ZX20 Portable SSD features shock-resistant aluminium housing with reinforced inner frame that makes it more shock resistant and durable for everyday use
- Complete Package Contents: The pocket-sized drive comes with both a USB Type-C to C cable and a USB Type-C to A cable, welcome guide, and friendly customer service included
When datasets are too large to duplicate across clouds, preserve a versioned manifest and a documented rebuild path. The RTO must include the time to regenerate missing derived data; if that exceeds the target, replicate the derived shards instead of relying on a rebuild.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Design recovery procedures for four failure types
GPU or host failure
- Let the scheduler mark the failed worker unavailable and stop writes from that worker.
- Identify the newest checkpoint whose manifest and checksum are valid.
- Provision a replacement worker or resubmit the job with the original topology when possible.
- Restore the checkpoint, rehydrate the environment by image digest, and resume from the recorded step and data position.
- Compare the first post-restart metrics with the saved metrics to detect an incorrect seed, shard, or configuration.
VM or cluster failure
Recreate the node group from infrastructure-as-code, attach the regional durable copy or restored file-system cache, then launch workers from the same checkpoint manifest. Do not assume a local NVMe volume survives instance replacement.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteRegional outage
Switch the recovery workflow to the second-region object store, request or pre-reserve the required GPU quota, and use the recovery-region image registry and key references. Restore through a regional cache or parallel file system to avoid every worker independently pulling all shards.
Cloud-wide outage
Invoke the alternate-cloud runbook. It should specify the account or project, network and identity setup, image import or rebuild, object locations, dataset availability, GPU shape mapping, scheduler submission, and DNS or endpoint changes. If any step still requires the unavailable provider’s console, the design is not independent enough for this scenario.
Run restore drills and measure the real outcome
A backup is only an assumption until a clean environment resumes training. Schedule drills at a frequency appropriate to the workload and after major changes to images, drivers, storage, or orchestration.
- Choose a recent checkpoint and record its RPO age, manifest, image digest, and expected metric values.
- Provision the target GPU shape in the recovery region or alternate cloud using the recovery automation.
- Restore the object set and rebuild the runtime without using hidden files from the failed environment.
- Launch the distributed job, resume from the recorded step, and run enough iterations to produce comparable metrics.
- Record restore duration, data-loss interval, time spent waiting for GPUs, network egress, storage throughput, GPU utilization during restart, and every manual operator action.
- Fix one failure at a time, update the runbook, and repeat until the measured RPO and RTO meet their targets.
AWS states that disaster-recovery strategies should be regularly assessed and tested. Include failure cases such as a missing shard, an expired key, a corrupt newest checkpoint, a changed worker count, and a deliberate deletion attempt against an immutable vault.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchQuick Recap
Compare designs on the trade-offs that matter
| Design choice | RPO and RTO effect | Cost and complexity effect | Best fit |
|---|---|---|---|
| Local NVMe only | Very low write latency, but catastrophic loss if the host fails; poor regional RTO | Low transfer cost, no durable protection | Temporary staging, never the sole backup |
| Regional object storage | Good durability and moderate restore time; no protection from a regional outage | Simple operations and predictable storage cost | Most experiments and baseline production protection |
| Cross-region replication | Protects against regional loss; RTO depends on GPU capacity and restore throughput | Additional storage, replication, and possible egress charges | Production training with a defined regional DR requirement |
| Second-cloud replica | Protects against provider-wide loss and improves portability, provided the alternate environment is tested | Highest engineering, egress, identity, and operational cost | Critical workloads or explicit portability requirements |
| Warm standby | Faster RTO than backup-and-restore because capacity and software are partly ready | Continuous GPU or infrastructure expense | Deadlines that cannot tolerate cluster provisioning delays |
A practical implementation sequence
- List every GPU workload, owner, training duration, accelerator shape, data source, and business deadline.
- Assign an RPO and RTO, then document the allowed loss in GPU-hours and the maximum restart delay.
- Define checkpoint contents, format, naming, checksum, manifest, and success marker.
- Implement fast local or parallel-file-system staging and an asynchronous publisher to regional object storage.
- Validate each checkpoint before replication and copy it to a second region.
- Add a second provider only after the regional workflow is reliable; create independent accounts, keys, images, and infrastructure-as-code.
- Apply versioning, soft delete, immutable retention, encryption, least privilege, and multi-person approval.
- Automate scheduler restart from the newest valid checkpoint, with a fallback to the previous checkpoint.
- Run recovery drills, capture measured RPO/RTO and costs, and revise cadence, topology, or capacity reservations based on evidence.
Common failure modes and fixes
- Only model weights are saved: add optimizer, scheduler, seed, data position, tokenizer, configuration, and metrics.
- The newest object is incomplete: use temporary names, checksums, manifests, and an atomic success marker.
- Every worker downloads every checkpoint: use sharded or hierarchical distribution through a regional cache.
- Replication silently lags: alert on checkpoint age and last successful cross-region copy, not merely on job health.
- The recovery region has no GPUs: verify quota and accelerator availability during drills and maintain a documented shape substitution.
- Recovery depends on production secrets: provision independent recovery identities and test key retrieval before an incident.
- Immutable storage is treated as a complete DR plan: immutability prevents deletion but does not provide compute, images, network, or a tested restore workflow.
- Cross-cloud copies cannot start: keep portable containers, pinned dependencies, infrastructure definitions, and a tested scheduler path for the alternate provider.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
