Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A CUDA out-of-memory (OOM) error means a training operation needed more GPU memory at its peak than the device could provide. The peak includes far more than model weights: gradients, optimizer state, saved activations, temporary operation tensors, framework allocations and memory held by other processes all compete for capacity. Diagnose that peak first, then reduce the workload or change memory behavior before buying a larger GPU.
What a GPU OOM error actually means
PyTorch describes the failure as an allocation that the CUDA device could not satisfy. The important detail is that the failing allocation may be short-lived. A run can appear to fit for many steps and still fail when one attention operation, backward pass or optimizer update briefly pushes the live peak over the limit.
The message is often shown as CUDA out of memory or cuda runtime error(2): out of memory. It does not, by itself, identify whether the cause is the model, the batch, a retained computation graph, a fragmented cache, another process or a combination of them. The PyTorch FAQ explains the distinction and common retention mistakes.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →The memory categories competing for capacity
| Component | What it contains | What usually increases it |
|---|---|---|
| Model weights | Parameters used by the forward pass | Larger parameter count or higher storage precision |
| Gradients | Per-parameter values produced during backpropagation | More parameters; training rather than inference |
| Optimizer state | Extra tensors maintained by the optimizer | Optimizer choice and parameter count |
| Saved activations | Intermediate results retained so backward can calculate gradients | Per-device batch size, sequence length, architecture and checkpointing policy |
| Temporary tensors | Workspaces and intermediate results created by individual operations | Attention, convolutions, fused kernels and the shape of the current operation |
| Framework and device allocations | Allocator-reserved blocks, CUDA libraries and allocations made outside PyTorch | Allocator behavior, kernels, other libraries and concurrent processes |
As a lower-bound illustration, the PyTorch FAQ notes that training needs at least twice the size of the weights because gradients must also be stored. That is not a complete training estimate: optimizer state, activations, temporary tensors and runtime allocations can add substantially more.
#1 Best Overall
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- 0dB technology lets you enjoy light gaming in relative silence
- Dual BIOS switch lets you toggle between Quiet and Performance BIOS profiles
- Dual ball fan bearings last up to twice as long as sleeve bearing designs
Why batch size and sequence length matter so much
Activations generally grow with the amount of data processed by each device step. Increasing the per-device batch therefore raises the amount saved for backward. Sequence length has an additional effect in attention implementations that materialize attention-score matrices: the matrix can grow with the square of sequence length. Doubling a long sequence can consequently require much more than twice the attention memory. The Hugging Face GPU memory guide describes these categories and scaling effects.
Peak behavior also explains why an error can appear during an otherwise ordinary step. An operation may need a large temporary buffer while earlier activations and optimizer state are still live. The average memory shown between operations can be below the card’s capacity even though that instantaneous request cannot fit.
Diagnose the failure before changing the model
Use this sequence on a reproducible run. Record the failing stage and dimensions before applying a fix; otherwise a change may hide the cause without making the workload reliably fit.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11-
Record the workload at the failing step
Write down the model and parameter count, per-device batch size, sequence length or image dimensions, precision mode, optimizer, gradient-accumulation setting and whether the error occurs in forward, backward or the optimizer step. These dimensions determine which memory categories are likely to dominate. Keep the failing input shape, because a single longer sample can create the peak.
-
Compare allocated, reserved and peak memory
PyTorch distinguishes memory occupied by live tensors from memory reserved by its caching allocator. Capture both during the run:
import torch torch.cuda.reset_peak_memory_stats() # run the smallest reproducible training segment here print('allocated:', torch.cuda.memory_allocated() / 1024**3, 'GiB') print('reserved: ', torch.cuda.memory_reserved() / 1024**3, 'GiB') print('peak: ', torch.cuda.max_memory_allocated() / 1024**3, 'GiB') print(torch.cuda.memory_summary())A large reserved value with a smaller allocated value indicates cached blocks are present, not necessarily that live tensors consume all of that space. A large allocated value means live tensors themselves are close to the limit. Capture the numbers immediately before and after forward, backward and the optimizer step to locate the peak.
-
Check device-wide usage
Run
nvidia-smiwhile the job is active and note every process using the GPU, not just the Python process you launched. Device-level usage can exceed PyTorch’s accounting because CUDA libraries or other software may allocate memory outside PyTorch. Conversely,nvidia-smican show allocator-reserved cache as used. Treat it as a device-wide view, not a live-tensor report.Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.Rank #2
GIGABYTE GeForce RTX 5070 Ti Gaming OC 16G Graphics Card, 16GB 256-bit GDDR7, PCIe 5.0, WINDFORCE Cooling System, GV-N507TGAMING OC-16GD Video Card- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
nvidia-smi -
Capture an allocation history when the numbers are ambiguous
PyTorch documents recording allocation history and viewing a memory snapshot in Understanding CUDA Memory Usage. A minimal diagnostic pattern is:
import torch torch.cuda.memory._record_memory_history(enabled='all') # run one reproducible segment that reaches the OOM # If execution reaches this line, write the snapshot: torch.cuda.memory._dump_snapshot('cuda_oom_snapshot.pickle')Open the snapshot with the viewer described in the PyTorch documentation. It can show which allocations were live around the failure and how the allocator’s blocks changed over time. Use the API that matches your installed PyTorch version; these memory-inspection interfaces are diagnostic tools rather than a training requirement.
-
Look for another process or a lingering subprocess
If memory remains allocated after the main Python program exits, inspect the process list again. A still-running Python worker, notebook kernel or child process can keep a CUDA context alive. Stop that process or restart the kernel before judging whether the workload itself fits. The PyTorch FAQ specifically calls out live Python subprocesses as a common explanation for memory that does not immediately disappear.
-
Audit long-lived bookkeeping tensors
Search metric lists, loss histories, logging dictionaries and custom callbacks for tensors that still require gradients. Appending a graph-connected loss or output to a list retains its autograd history, along with the tensors needed to compute it, on every iteration. For values used only for reporting, store a detached tensor or a Python number:
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteSpecial offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.loss_history.append(loss.detach().cpu()) scalar_history.append(loss.detach().item())This is different from a normal allocator cache: the graph-connected tensors are live references and cannot be released until those references are removed.
Fixes that reduce the peak without buying hardware
Reduce the per-device batch size
Lower the batch processed by each GPU step first. This directly reduces the number of activation sets held for backward and often removes the peak that triggers the error. Change one variable at a time and rerun the same failing shape so you can measure the new peak.
The tradeoff is less work per device step and, unless you compensate, a smaller statistical batch. Batch size reduction cannot make model weights, gradients or optimizer state disappear; it addresses the activation portion of the peak.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Shorten the sequence or input dimensions
For language models, reduce the maximum sequence length, bucket examples by similar lengths or remove unnecessary padding. For vision or other modalities, reduce the spatial or temporal dimensions only when that change is acceptable to the task. Shorter inputs reduce activation volume, and they can have a particularly large effect when attention-score matrices are materialized because those matrices scale quadratically with sequence length.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Preserve the original context only for examples that need it. A shorter maximum length is a workload decision, not an allocator trick, so verify that truncation does not remove information required by the task.
Use gradient accumulation to preserve an effective batch
When the desired batch is too large to place on the GPU at once, split it into smaller device batches and delay the optimizer update. Hugging Face documents this pattern in its gradient accumulation guide.
optimizer.zero_grad(set_to_none=True)
for step, batch in enumerate(loader):
with torch.autocast(device_type='cuda'):
loss = model(**batch).loss
(loss / accumulation_steps).backward()
if (step + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad(set_to_none=True)If each device batch has four examples and accumulation_steps is eight, the update represents an effective batch of 32 examples, subject to the usual effects of padding and data ordering. Accumulation lowers activation demand per device step, but it does not reduce the memory required for model weights, gradients that remain resident, optimizer state or an oversized model. It also increases the number of forward and backward passes per optimizer update and can reduce throughput.
Enable gradient checkpointing
Normally, the forward pass keeps many intermediate activations for backward. Gradient checkpointing stores selected boundaries, discards other intermediates and recomputes them during backpropagation. That lowers saved-activation memory at the cost of extra computation and a longer step time. In Transformers models that expose the method, the configuration commonly looks like:
model.gradient_checkpointing_enable()Check the model’s documentation and framework version because support and interactions with caching differ by architecture. Hugging Face’s gradient-checkpointing documentation also describes offloading checkpointed activations to pinned host memory. Offloading can make a run fit when GPU memory is the bottleneck, but host-device transfers add latency and require sufficient host RAM and bandwidth. It is a capacity tradeoff, not a speed optimization.
Evaluate mixed precision rather than assuming it saves memory
Mixed precision stores or computes selected tensors in a lower-precision format while retaining higher precision where needed. On supported hardware it can improve throughput and may reduce the storage of some activations. The actual result depends on whether the run uses fp16, bf16 or another mode, which tensors remain in fp32, the optimizer implementation and the framework’s safety rules.
Rank #4
- Powered by Radeon RX 9070 XT
- WINDFORCE Cooling System
- Hawk Fan
- Server-grade Thermal Conductive Gel
- RGB Lighting
The Hugging Face memory guide discusses fp16, bf16 and tf32 and cautions that fp16 is not automatically memory-optimized when fp32 copies are retained. Measure allocated and peak memory with the chosen configuration instead of promising a universal percentage reduction. A setting that speeds kernels may have little effect on the tensors dominating your peak.
Detach values that are used only for metrics
Fix graph retention in the training loop before trying allocator settings. Replace history entries that hold a loss or output tensor with detached CPU values when gradients are no longer needed. Also inspect custom logging and callback code, where a reference can survive for many iterations even though the model code is correct. This remedy targets retained tensors and can stop a gradual climb in memory; it does not reduce the legitimate memory of a single forward/backward pass.
Understand what empty_cache() can and cannot do
PyTorch’s caching allocator keeps freed blocks so later operations can reuse them. Consequently, nvidia-smi may continue to show substantial usage after tensors have been freed. Calling torch.cuda.empty_cache() releases unused cached blocks so another GPU application can use them and can make device-level usage easier to interpret:
torch.cuda.empty_cache()It does not free live tensors, detach autograd graphs or increase the memory available for tensors that are still required by your training step. Use it after genuinely freeing references or when cooperating with other GPU applications, not as the general fix for a live-tensor OOM. Frequent calls inside the training loop can also add synchronization and allocation overhead.
Which remedy fits which constraint?
| Remedy | Primary memory pressure reduced | Throughput or step-time effect | Preserves effective batch or sequence? | Additional requirement |
|---|---|---|---|---|
| Smaller per-device batch | Saved activations | Less work per device step; may require more steps | Not by itself | Workload or configuration change |
| Shorter sequence or input | Activations and attention intermediates | Usually less computation; less context | No; context is reduced | Task must tolerate truncation or resizing |
| Gradient accumulation | Activations per device step | More forward/backward passes per update | Yes, approximately, by combining mini-batches | Training-loop or trainer configuration change |
| Gradient checkpointing | Saved activations | Extra recomputation; slower steps | Usually yes | Model/framework support |
| Activation offloading | GPU activation residency | Host-device transfer overhead | Usually yes | Pinned host memory and adequate RAM/bandwidth |
| Mixed precision | Some tensor storage and compute pressure | Can improve speed, but depends on hardware and dtype | Usually, if numerically stable | Supported hardware and careful precision settings |
| Detach bookkeeping values | Unintentionally retained graphs | Normally negligible; can prevent runaway growth | Yes | Code audit |
empty_cache() |
Unused allocator cache visible to other users | Possible allocation overhead if overused | Yes | Does not solve live-tensor capacity |
| GPU with more VRAM | Total capacity pressure | May avoid workarounds; purchase and platform costs | Yes | Compatible hardware, power, drivers and budget |
Use the error’s timing to narrow the cause
OOM on the first forward pass
Check model weights, precision, input shape and another process first. A first-pass failure usually means the baseline live set plus the first operation already exceeds capacity; reducing batch or sequence length, changing supported precision or using a larger-memory device are the relevant branches.
OOM during backward
Backward often creates its largest temporary tensors while saved activations are still resident. Reduce the per-device batch or sequence, then try gradient checkpointing. If the effective batch must remain unchanged, accumulate smaller mini-batches.
Recommended Free Tools
OOM after several successful iterations
Inspect histories, metrics and callbacks for graph-connected tensors. A steadily rising allocated value points to retained references rather than a single shape that is too large. Detach or convert reporting values to CPU scalars and rerun the same workload.
Best Value
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
OOM only on long examples
Log the actual sequence or input dimensions of the failing batch. Padding and attention-score materialization can make one unusually long sample the peak. Bucketing, truncation or a lower maximum length addresses that input-driven pressure.
nvidia-smi shows memory after the script exits
Check for notebook kernels, worker processes and child Python processes. If PyTorch’s allocated value is low but reserved memory remains, the allocator is holding reusable blocks; if another process owns the memory, terminate or reconfigure that process. Do not infer live-tensor usage from the device total alone.
When a larger GPU is the correct answer
Consider a GPU with more VRAM only after measurements show that the correctly configured workload’s live peak still exceeds available capacity. First remove accidental graph retention, account for other processes, reduce per-device activation pressure, and evaluate checkpointing or accumulation. If the remaining model state and required workload dimensions still cannot fit, more capacity is a reasonable hardware path.
A concrete scale example from the Hugging Face v5.12.0 GPU memory guide estimates roughly 85 GB for training a 4-billion-parameter model in mixed precision at batch size 16. That is an example under the guide’s assumptions, not a universal GPU-sizing formula. Actual requirements vary with architecture, sequence length, optimizer, precision, implementation and runtime behavior; the documentation notes that version 5.17.0 is newer than v5.12.0.
No particular card, price or compatibility combination follows from that example. Before purchasing, measure the peak for your exact model and decide whether you need room for the current workload only or for larger batches, longer sequences and future checkpoints. Verify system power, cooling, physical fit, driver support and the memory available to every process that will share the device.
A practical runbook
- Reproduce the OOM with one fixed input shape and record model, batch, sequence length, precision and optimizer.
- Capture allocated, reserved and peak PyTorch memory around forward, backward and the optimizer step.
- Compare those values with
nvidia-smiand identify other processes or non-PyTorch allocations. - Inspect a memory snapshot if the allocation history or a reserved-versus-allocated gap is unclear.
- Audit metric and logging code for tensors that retain autograd history.
- Lower per-device batch or sequence length and rerun the same reproduction.
- If the effective batch must remain, add gradient accumulation; if activations still dominate, enable checkpointing or carefully evaluate offloading.
- Measure mixed precision in the actual dtype and hardware configuration rather than assuming a saving.
- Use
empty_cache()only after freeing unused tensors or when releasing cached blocks to other applications. - Choose a higher-VRAM GPU only when the measured live peak remains above capacity after these changes.
GPU OOM troubleshooting is therefore a peak-capacity investigation. Once you know which tensors are live at the failing operation, the fix is usually a targeted change to workload dimensions, activation retention, bookkeeping or available capacity—not a blind call to clear the cache.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
