Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A GPU out-of-memory (OOM) error means the next allocation could not be satisfied from the VRAM available to the workload. The remedy depends on what consumed that memory: model weights, gradients and optimizer state, saved activations, inference context and KV cache, temporary workspace, another process, or allocator fragmentation. Identify the failing phase first, then reduce the corresponding workload, place part of the model elsewhere, or add capacity if the required working set genuinely cannot fit.
What a GPU OOM error actually means
When a framework reports cuda runtime error(2): out of memory or torch.OutOfMemoryError: CUDA out of memory, an operation requested a block of device memory and the request failed. NVIDIA describes the underlying condition plainly: “GPU out-of-memory (OOM) errors occur when the model needs more VRAM than the GPU provides.” The failure can happen while loading weights, during a training forward or backward pass, during an optimizer step, while allocating an inference KV cache, during graph capture or warmup, or on a later request after the service has already started.
An apparent contradiction is common: the model file fits, yet the job does not. A loaded model is only one part of the memory budget. Training adds gradients, optimizer state, saved activations and temporary tensors. Inference adds the key-value (KV) cache for the active context, memory for concurrent requests and outputs, graph-capture or warmup buffers, and model-specific workspaces. The allocation that fails is often a temporary or per-request allocation rather than the weights themselves.
VRAM shown by a system utility is also not identical to live tensors. PyTorch’s caching allocator keeps released blocks for reuse. Consequently, nvidia-smi can show a large footprint even when the program has less live tensor data. PyTorch distinguishes memory currently occupied by tensors from memory held by its allocator for possible reuse; those are different diagnostic questions.
#1 Best Overall
- AI Performance: 767 AI TOPS
- OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis
Find the failing phase before changing settings
Record the exact operation and phase at which the exception occurs. The phase determines which remedy is plausible; a setting that helps a KV-cache allocation will not solve an oversized training activation.
- Model loading: the weights, conversion buffers or initial placement exceed available memory.
- Training forward pass: activations or temporary operators exceed the remaining budget after weights and other state are resident.
- Training backward or optimizer step: saved activations, gradients or optimizer state create the peak.
- Inference prefill or decode: the prompt, generated sequence, active batch or KV cache is too large.
- Graph capture or warmup: the serving runtime needs a temporary headroom allocation.
- Later request: concurrency, a longer context, a leak or unreleased reference changes the working set after startup.
NVIDIA’s troubleshooting guide separates these cases for its NIM deployments. Its flags and examples are specific to NIM; use the phase distinction generally, but do not copy a NIM option as though it were a universal inference-server setting. The guide was updated October 6, 2026: NVIDIA’s GPU memory OOM troubleshooting documentation.
Check other processes and the system-level view
First determine whether another process is consuming the card. Run nvidia-smi at the time of failure and note the process list, total used memory and free memory. A process left by a previous job, a second service, a desktop compositor or another user can remove the headroom your program assumed it had. Stop only processes you own or are authorized to stop; otherwise select another device or coordinate with the administrator.
For a changing workload, repeatedly sample the view with watch -n 1 nvidia-smi. Compare the value immediately before the failing operation with the value after a batch, request or iteration. A steady rise across iterations points toward retained objects or a growing cache; a sharp peak at one operation points toward that operation’s working set.
Compare PyTorch allocated and reserved memory
In PyTorch, print allocator metrics around the operation:
import torch
print('allocated:', torch.cuda.memory_allocated())
print('reserved: ', torch.cuda.memory_reserved())
print(torch.cuda.memory_stats())memory_allocated() is memory occupied by live tensors known to PyTorch. memory_reserved() includes blocks held by the caching allocator, including blocks that are currently unused but available for reuse. A large reserved value with a much smaller allocated value is a clue to investigate allocator behavior or fragmentation; it is not proof that fragmentation caused the failure. A large allocated value indicates that live tensors themselves consume the budget.
For deeper analysis, PyTorch provides allocator statistics and memory snapshots in Understanding CUDA Memory Usage. Those tools describe allocations tracked by PyTorch’s allocator. If a library allocates CUDA memory outside that allocator, a separate investigation may be necessary.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Interpret an allocation failure, not just a full-looking meter
Fragmentation is possible when enough total free memory exists in aggregate but no suitable contiguous block is available for the next request. Treat that as a hypothesis only after checking the failing allocation, allocated-versus-reserved values, statistics and snapshots. Do not diagnose fragmentation solely because nvidia-smi looks full.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Where VRAM goes during training
Training normally has a larger peak than inference because it must preserve information for backpropagation and update state. The Hugging Face Transformers v5.12.0 GPU-memory guide identifies the principal categories:
| Memory category | What it contains | What changes its size |
|---|---|---|
| Model weights | Parameters used by the forward pass, stored at the selected precision. | Parameter count, data type and sharding or quantization choices. |
| Gradients | Parameter gradients retained for the update. | Parameter count, precision and whether gradients are enabled. |
| Optimizer state | Statistics maintained by the optimizer, such as momentum or variance. | Optimizer algorithm, parameter count and state precision. |
| Saved activations | Intermediate values needed by backward propagation. | Batch size, sequence length, model depth and hidden size. |
| Temporary tensors and workspaces | Short-lived operator outputs, reductions, attention workspaces and framework overhead. | Kernel choice, shapes, implementation and the operation at the peak. |
Activation memory is especially sensitive to batch and sequence dimensions. In the implementation patterns discussed by the Transformers guide, attention-score memory can grow with the square of sequence length. Doubling a long sequence can therefore require much more than twice the attention-score storage, depending on the attention implementation.
The same guide gives a named example: training a 4-billion-parameter model in mixed precision at batch size 16 requires roughly 85 GB of GPU memory. That is the guide’s estimate for its example, not a universal requirement. Architecture, optimizer, sequence length, implementation and other settings can move the peak substantially; the page consulted is versioned v5.12.0 and notes that a newer v5.19.0 version exists, so verify version-sensitive details.
Recommended Free Tools
Where VRAM goes during inference
Inference avoids gradients and optimizer state, but it still needs more than the weight files. The runtime may reserve memory for the input and output tensors, temporary kernels, concurrent requests, graph capture or warmup, and the KV cache. The KV cache stores attention keys and values for tokens already processed so that each subsequent decode step does not recompute the entire context. Longer input-plus-output sequences and more simultaneous requests increase that cache.
For an individual request, context length, generated length and batch or concurrency limits are practical memory controls. A service that starts successfully can still fail when a later request has a longer prompt or when several requests arrive together. Measure the peak under the largest context and concurrency your product actually permits, rather than relying only on startup success.
NVIDIA’s NIM documentation contains two phase-specific cautions. Reducing its --gpu-memory-utilization setting can shrink the budget reserved for KV cache and therefore make a KV-cache-capacity problem worse. In a documented graph-capture case, lowering the setting may instead leave the headroom needed for capture. Those recommendations apply to the stated NIM scenarios, not as a general rule for every server.
Reduce memory use in training or fine-tuning
Lower the per-device batch size
Batch size is often the fastest lever because activations and many temporary tensors scale with it. Reduce the per-device or microbatch size until the peak fits, then measure throughput and convergence. A smaller batch can alter optimization behavior, so do not assume it is behaviorally identical in every training setup.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Use gradient accumulation for a larger effective batch
Gradient accumulation runs several smaller microbatches before an optimizer update. It can preserve a desired effective batch while lowering the memory peak of each forward and backward pass. The trade-off is additional forward and backward work per update, changed scheduling and potentially lower throughput; verify that your loss scaling and normalization match the intended effective batch.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
Shorten sequence length where the task allows
Reduce the maximum training sequence or crop inputs when long context is not required. This targets activation and attention memory directly. Because attention-score storage can have quadratic sequence dependence in some implementations, a modest reduction in a very long sequence may produce a disproportionately large saving. Do not truncate information that the task requires.
Enable mixed precision only with supported hardware and software
Mixed-precision training stores and computes eligible tensors at lower precision while retaining higher precision where needed. It can reduce weight, activation and temporary storage, but savings depend on which tensors remain in higher precision and on the model, optimizer and hardware. Confirm that the framework, model and accelerator support the selected precision and monitor numerical stability; lower precision is not an unconditional drop-in.
Checkpoint activations
Activation or gradient checkpointing stores fewer intermediate activations and recomputes them during the backward pass. PyTorch describes this explicitly as a memory-for-compute trade-off in its Frequently Asked Questions. It can make a model fit when activations are the peak, but it adds computation and can reduce training throughput. Apply it to the layers or blocks that dominate memory, then remeasure.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Do not retain autograd history accidentally
A common leak is appending a tensor that still requires gradients to a Python list or logging object on every iteration. The reference keeps its computation graph alive, so memory grows until an OOM occurs. Store detached values for statistics, or convert them to an appropriate host value when you no longer need the graph. The PyTorch FAQ discusses this retained-autograd-history pattern and its remedies.
Reduce optimizer-state storage
Optimizer state can be comparable to or larger than the weights. Choose an optimizer and state precision that meet the training objective while fitting memory. The Transformers memory guide describes quantized Adam as one possible way to reduce optimizer-state storage, with implementation-specific effects on speed, compatibility and numerical behavior. Treat it as a targeted trade-off, not a universal fix.
Reduce memory use for inference and serving
Reduce context, output length or concurrency
Lower the maximum context length when the application can accept a shorter input-plus-output window. Reduce maximum generated tokens or the number of simultaneous requests if the KV cache is the failing allocation. These changes constrain product behavior, so document the new limits and test the longest supported request rather than an average request.
Select a model and precision profile that fit
A smaller model or a supported lower-precision profile reduces weight memory and often leaves more room for KV cache and temporary workspaces. The appropriate precision depends on the model, hardware and quality target; validate output quality and numerical support instead of assuming every GPU can run every profile. NVIDIA’s NIM examples use model- and hardware-specific tensor or pipeline parallelism and precision profiles, which should not be generalized beyond the documented deployment.
Place weights with device maps and offload
Hugging Face Accelerate can construct device maps that place portions of a large model on available GPUs and offload remaining weights to CPU memory or disk. The Big Model Inference guide cautions that nominal GPU capacity is not all available for weights: kernels, activations and outputs need headroom. Reserve host RAM for CPU offload and fast, sufficient storage for disk offload. Transfers across the host-device boundary make execution slower, sometimes substantially, but offload can turn an otherwise impossible load into a workable one.
Rank #4
- Powered by the NVIDIA Blackwell architecture and DLSS 4 OC mode: 2640MHz/Default mode: 2610MHz (Boost Clock)
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.125-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
Offload is most useful when the total model fits across the selected GPU, host RAM and storage, and when the latency penalty is acceptable. It does not remove the need to fit the active layer, inputs, outputs and runtime workspace on the device at the moment they are used.
Use multiple GPUs when placement is supported
Sharding or tensor and pipeline parallelism distributes weights and working memory across GPUs. It adds capacity, but the framework or serving runtime must support the placement strategy, interconnect and communication pattern. A second card is not automatically additive for every model: unsupported placement, uneven partitioning or insufficient per-device workspace can still produce OOM errors. Confirm the server’s documented parallelism mode and test the full request shape.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Allocator controls and cache cleanup
Understand what empty_cache() can and cannot do
torch.cuda.empty_cache() releases unused cached blocks from PyTorch’s allocator so that other GPU applications can use them. It does not free live tensors, and it does not increase the amount of VRAM available to PyTorch for those tensors. Calling it cannot make a genuinely oversized model, batch or context fit. Excessive calls inside a hot loop can also undermine allocator reuse and hurt performance.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallTune the allocator only after diagnosis
Allocator settings address particular fragmentation patterns; they do not create physical VRAM. PyTorch documents max_split_size_mb as a last resort for the native allocator when many inactive split blocks are present. NVIDIA documents expandable_segments:True for a specific NIM fragmentation scenario. These controls are version-, backend- and workload-sensitive. Apply them only after allocator statistics or snapshots support that diagnosis, and record the exact framework, CUDA and server versions so the change can be reproduced.
A practical troubleshooting sequence
- Capture the error and phase. Save the full exception, operation, input shape, batch or concurrency, context length and whether the failure occurs at load, training, prefill, decode, capture or a later request.
- Check competing processes. Use
nvidia-smiand identify whether another authorized process consumes the missing headroom. - Measure framework memory. Compare
torch.cuda.memory_allocated(),torch.cuda.memory_reserved()and memory statistics immediately before the failing operation. - Identify the dominant category. Weights point toward model size, precision or placement; activations point toward batch, sequence or checkpointing; KV cache points toward context or concurrency; optimizer state points toward training configuration; a large reserved-versus-allocated gap points toward allocator investigation.
- Apply the smallest workload change. Reduce batch, context, output length or concurrency only as far as the task allows, then rerun the same peak case.
- Change execution strategy if necessary. Add checkpointing, mixed precision, gradient accumulation, supported offload or multi-GPU placement according to the phase that failed.
- Investigate fragmentation last. Use snapshots and statistics before changing allocator settings. Recheck that the new setting improves the same failing allocation without merely hiding a capacity shortfall.
- Re-test production limits. Validate the largest supported model, sequence, batch and concurrency together, including startup and later requests.
When more VRAM is the correct answer
If the weights and the required working set still exceed available capacity after reasonable batch, context, precision, checkpointing, placement and offload choices, the limitation is physical capacity rather than a cleanup command. Options are a smaller or lower-memory model, a supported lower-precision profile, CPU or disk offload, distributing the model across GPUs, or hardware with more VRAM.
There is no universal minimum VRAM number for an AI workload. The correct capacity depends on parameter count, architecture, precision, framework, sequence or context length, batch or concurrency, optimizer, runtime overhead and the speed target. Use measured peak allocation for the exact workload, not the model-file size alone, when sizing a replacement device.
Choosing among the remedies
| Remedy | Memory it targets | Main constraint or cost | Compatibility questions |
|---|---|---|---|
| Smaller batch or microbatch | Activations and per-request temporaries | Lower throughput or a smaller effective batch unless accumulation is used | Does the task tolerate changed batch behavior? |
| Shorter sequence or context | Activations, attention work and KV cache | Less input-plus-output capacity | What is the longest context users actually need? |
| Gradient accumulation | Training peak per microbatch | More work per optimizer update and scheduling changes | Are normalization and loss-scaling semantics correct? |
| Mixed precision | Eligible weights, activations and temporaries | Possible numerical or quality trade-offs | Does the accelerator and model support the chosen precision? |
| Activation checkpointing | Saved training activations | Recomputation and lower throughput | Which blocks dominate the activation peak? |
| CPU or disk offload | Resident model weights on the GPU | Host RAM or storage use and transfer latency | Is the device map supported and is host bandwidth sufficient? |
| Multi-GPU placement | Weights and working memory per device | Hardware cost, communication and placement complexity | Does the runtime support the required parallelism mode? |
| Allocator tuning | Specific fragmentation patterns | Version-sensitive behavior and no added capacity | Do statistics actually show inactive split blocks or a documented case? |
| Higher-capacity GPU | The entire physical VRAM budget | Hardware, power and platform cost | Does the new device meet the model’s precision, host and speed requirements? |
Version and scope notes
The guidance above combines official framework and vendor documentation. PyTorch’s memory-management documentation is maintained on its current main branch at CUDA semantics — Memory management; allocator behavior can vary by backend and release. The Transformers memory figures and terminology cited here come from v5.12.0, while that page notes v5.19.0 is newer. NVIDIA’s phase-specific examples come from its NIM troubleshooting page, updated October 6, 2026. Check the documentation for the exact versions deployed before relying on a flag, profile or allocator option.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
