iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Estimate LLM GPU memory in two stages: calculate the model state, then add memory created by the workload and its runtime. For inference, start with parameter count × bytes per parameter ÷ tensor-parallel degree for weight storage, then add KV cache, activations, input/output tensors, workspaces and other runtime allocations. For training, include weights, gradients, optimizer states, saved activations and temporary tensors; there is no reliable universal multiplier.
The result is a planning estimate, not a launch guarantee. Context length, generated tokens, concurrency, microbatch size, precision, architecture, sharding and the serving or training engine can move peak usage substantially. Validate any borderline calculation by profiling the exact model and configuration on the target GPU.
What a GPU-memory estimate must answer
A useful estimate answers two separate questions:
- Will the model state fit? This is mostly weights for inference, and weights plus gradients and optimizer state for training.
- Will the planned workload fit at peak? This includes KV cache, activations, temporary tensors, communication buffers, CUDA workspaces, graph allocations and allocator overhead.
Parameter count and precision provide a fast first approximation, but they do not describe the complete allocation on a GPU. NVIDIA summarizes the inference problem this way: “In effect, the two main contributors to the GPU LLM memory requirement are model weights and the KV cache.” KV cache is only one of several additional allocations a real engine can make.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Keep the arithmetic and the assumptions together. A statement such as “this is a 14 GB model” normally means only that the weights require about 14 GB in a particular format; it does not mean a 14 GB GPU can serve it.
#1 Best Overall
- 16,384 NVIDIA CUDA Cores
- Supports 4K 120Hz HDR, 8K 60Hz HDR and variable refresh rate as indicated in HDMI 2.1A
- New streaming multiprocessors: up to 2x power and power efficiency
- Fourth generation tensor cores: up to 2x AI power
- Third-generation RT cores: up to 2x ray tracing performance
Record these inputs before calculating
Write down the exact model revision and runtime configuration. If a value is omitted, the estimate is not reproducible.
- Model and architecture: parameter count, number of layers, hidden size, number of key/value heads and head dimension. Parameter count is commonly listed on a model card or in safetensors index metadata.
- Weight format: BF16, FP16, FP8, INT4, NVFP4 or another format actually supported by the selected engine.
- KV-cache format: It can differ from the weight format.
- Inference workload: maximum prompt tokens, maximum generated tokens, total context length, concurrent sequences and batch policy.
- Training workload: full fine-tuning or parameter-efficient tuning, optimizer, mixed-precision policy, microbatch size, global batch size, sequence length and activation checkpointing.
- Distribution: single GPU, tensor parallelism, pipeline parallelism, data parallelism, optimizer sharding and any CPU offload.
- Runtime features: quantization metadata, adapters, multimodal modules, CUDA graphs, paged KV cache, communication libraries and workspace limits.
- Capacity to compare: memory available to the process, not merely the card’s advertised total. Reserve room for the operating system, display services and other processes when applicable.
Inference: calculate weights first, then the workload
1. Convert precision to bytes per parameter
For planning arithmetic, NVIDIA’s current NIM guidance uses these nominal sizes:
| Format | Planning size | What the number represents |
|---|---|---|
| BF16 or FP16 | 2 bytes per parameter | Nominal weight storage |
| FP8 | 1 byte per parameter | Nominal weight storage |
| INT4 or NVFP4 | 0.5 byte per parameter | Nominal weight storage |
These are arithmetic values, not exact allocations. Quantized formats can require scales, lookup data and other metadata, while runtimes allocate temporary buffers around the weights.
2. Estimate weight storage per GPU
Use:
weight memory per GPU = parameter count × bytes per parameter ÷ tensor-parallel degree
With no tensor parallelism, the degree is 1. A degree of 4 means the weight storage is divided across four tensor-parallel ranks, subject to the engine’s sharding implementation. Do not assume every other allocation is divided in exactly the same way.
3. Add KV cache using the actual architecture
Every generated token normally needs key and value entries for every transformer layer. AWS gives the general planning expression:
KV bytes = 2 × KV-cache bytes per element × layers × KV heads × head dimension × context length × batch size
Free tools Windows power users keep installed
One-click scans. No signup required.
The factor of 2 accounts for keys and values. Context length must include the prompt and the tokens the model may generate. If requests have different lengths, size for the maximum or use the serving system’s documented allocation policy.
For a full-head architecture where the KV-head arrangement maps directly to the hidden size, NVIDIA also presents the commonly used form:
KV bytes = batch × sequence length × 2 × layers × hidden size × bytes per element
Do not apply that simplified equation to grouped-query attention (GQA) or multi-query attention without checking the number of KV heads and head dimension. Those architectures can use much less KV memory than a full-head model with the same hidden size.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute4. Add runtime and workload allocations
After weights and KV cache, add the categories that the selected engine reports or documents:
Rank #2
- TRI FROZR 3-Stay cool and quiet. MSI’s TRI FROZR 3 thermal design enhances heat dissipation all around the graphics card.
- TORX FAN 5.0-Fan blades linked by ring arcs and a fan cowl work together to stabilize and maintain high-pressure airflow.
- Copper Baseplate-Heat from the GPU and memory modules is captured by a copper baseplate and then rapidly transferred to Core Pipes.
- Core Pipe-Precision-machined heat pipes ensure max contact and spread heat along the full length of the heatsink.
- Airflow Control-Sections of different heatsink fins disrupt unwanted airflow harmonics and reduce noise.
- Internal activations and intermediate operation tensors.
- Input and output tensors, including logits when they are retained.
- CUDA workspaces used by attention, matrix multiplication and other kernels.
- Communication buffers for tensor or pipeline parallelism.
- CUDA graph static buffers, if graphs are enabled.
- Adapter weights, multimodal encoders or other modules loaded beside the language model.
- Allocator reserve and fragmentation.
TensorRT-LLM identifies weights, internal activations and I/O tensors as major categories and documents configurable KV-cache allocation. The exact split depends on the engine, version and enabled features.
5. Compare with usable capacity, not nameplate capacity
Subtract memory already occupied by the driver, display server and other processes. Then compare the estimate with the runtime’s allocation policy. A model that theoretically fits with only a few hundred megabytes left is operationally risky because a different request length, kernel choice or allocator state can trigger an out-of-memory error.
Inference examples and what they do—and do not—prove
The following are source-published illustrations. They are useful checks on arithmetic, not guarantees for every runtime or workload.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Example | Published calculation | Interpretation |
|---|---|---|
| 8B model, BF16 | About 16 GB of weights | NVIDIA uses this as an example that can leave room for KV cache and overhead on a 24 GB GPU such as an RTX 4090 or A10G; the cited example does not guarantee every context or concurrency level will fit. |
| 70B model, BF16, tensor parallelism 4 | About 35 GB of weights per GPU | Remaining capacity for KV cache and runtime allocations depends on each GPU and the serving configuration. |
| 7B model, FP16 or BF16 | About 14 GB of weights | Weights only; KV cache and runtime allocations are additional. |
| 7B Llama 2, FP16, 4,096 tokens, batch 1 | About 2 GB of KV cache | Architecture-specific illustration, not a universal 7B value. |
| Mistral-7B, BF16 KV cache, 16,000 tokens | 1.95 GB for one request; 7.81 GB for four concurrent requests | AWS example using GQA with 8 KV heads, 32 layers and head dimension 128; concurrency multiplies the cache requirement. |
| 70B model, FP8 | Approximately 70 GB of weights | AWS notes that one 48 GB L40S is insufficient for that weight component alone. |
For a quick scenario, suppose an 8B BF16 model is served with one tensor-parallel rank. The weight estimate is approximately 16 GB. You then calculate KV cache for the model’s actual layers, KV heads, head dimension, cache precision, context and concurrency, and reserve space for the engine. The 16 GB figure is not the final requirement.
Training: account for model state and activations separately
1. Define the training mode and sharding strategy
Full fine-tuning keeps substantially more state on the system than inference. Record the optimizer, precision policy, microbatch and global batch, sequence length, activation checkpointing or recomputation, and the distribution strategy before doing arithmetic. Parameter-efficient methods can change the balance, but the available material does not support one universal LoRA or QLoRA total; calculate the frozen-model, trainable-adapter and optimizer allocations for the specific implementation.
2. Count weights, master copies, gradients and optimizer states
One mixed-precision arrangement documented by Hugging Face breaks model state into:
| Component | Planning amount per parameter | Reason |
|---|---|---|
| FP16 parameters plus FP32 weight copies | 6 bytes | 2 bytes for the low-precision copy and 4 bytes for the FP32 copy |
| Two FP32 Adam states | 8 bytes | 4 bytes each for the first and second moments |
| FP32 gradients | 4 bytes | Gradient storage in this accounting model |
That arrangement totals 18 bytes per parameter before activations and temporary tensors. Hugging Face gives a roughly 85 GB total GPU-memory illustration for a 4B model in mixed precision with batch size 16. Treat it as a configuration-specific example, not a multiplier for every optimizer or batch.
Recommended Free Tools
Megatron Bridge documents a different accounting expression: 18 bytes per parameter for model state when its distributed optimizer is disabled, or 6 + 12/shard_size bytes per parameter when the distributed optimizer is enabled. The shard size and exactly which states are distributed matter. Do not combine this expression casually with the Hugging Face breakdown or present either as a universal law.
3. Estimate activation memory from the microbatch
Training activations are produced for each layer and can dominate peak memory. Their size changes with:
- Microbatch size, rather than only the global batch size.
- Sequence length, including padding strategy.
- Number of layers and hidden size.
- Attention and feed-forward intermediate dimensions.
- Whether activations are stored, recomputed through checkpointing, or released by the framework.
- Temporary tensors created by individual kernels.
A larger global batch distributed across more data-parallel workers does not automatically increase activation memory on each GPU; the local microbatch does. Conversely, gradient accumulation can keep the microbatch constant while increasing the number of steps contributing to a global batch.
4. Add temporary and distributed allocations
Include gradient buckets, parameter all-gather or reduce-scatter buffers, pipeline send and receive buffers, optimizer partitions, CUDA workspaces and any CPU-offload staging memory. Megatron Bridge’s theoretical estimate for its most-loaded GPU explicitly excludes allocator fragmentation, CUDA workspace, NCCL buffers, CUDA graph static buffers, token-routing imbalance and CPU offloading. Those exclusions are why a theoretical fit must be checked with a real job.
Worked training arithmetic
For a first full-fine-tuning estimate using the Hugging Face mixed-precision arrangement, multiply parameter count by 18 bytes to estimate model state, then add activation and runtime estimates:
Rank #3
- NVIDIA Ada Lovelace Streaming Multiprocessors: Up to 2x performance and energy efficiency
- Tensor Cores of the 4th Generation: up to 2x AI performance
- RT-cores of the 3rd Generation: up to 2x raytracing performance
- OC mode: Boost clock 2595 MHz (OC mode) / 2565 MHz (gaming mode)
- Axial Tech fans deliver up to 23% higher airflow
training peak ≈ (parameter count × 18 bytes ÷ applicable state-sharding factor) + activations + temporary and communication buffers
For example, a 4B model has approximately 72 GB of model state under an 18-byte accounting (4,000,000,000 × 18 bytes, expressed approximately). The documented 85 GB illustration with batch size 16 implies additional memory for activations and temporary allocations. It does not establish that every 4B model, optimizer or sequence length requires 85 GB.
If the selected distributed optimizer uses Megatron Bridge’s expression, substitute its documented sharding term and then add the same categories of activations and runtime memory. Keep the framework, optimizer and shard assumptions beside the result so another engineer can reproduce it.
How sharding changes the estimate
Tensor parallelism
Tensor parallelism splits portions of a layer across GPUs, reducing the weight storage per rank by the tensor-parallel degree in the basic estimate. It also introduces collective communication and buffers. KV-cache partitioning is engine-dependent, so verify whether the selected runtime shards it, replicates it or uses another layout.
Pipeline parallelism
Pipeline stages hold different layers. This can reduce the model-state footprint on an individual GPU, but each stage needs activation transfers and scheduling buffers. The stage with the largest layer or the heaviest temporary allocation can determine the required capacity.
Optimizer and state sharding
Distributed optimizers partition optimizer states and sometimes gradients or parameter copies. Use the framework’s documented per-rank accounting rather than dividing a single-GPU total by the number of devices. Uneven layer sizes, embeddings and non-sharded components can leave one rank more heavily loaded.
Communication and performance trade-offs
Making a model fit across more GPUs does not automatically make it fast. Interconnect bandwidth, collective latency, pipeline bubbles, request scheduling and batch size affect throughput and latency. AWS recommends evaluating latency and throughput after an instance meets the memory requirement.
A repeatable inference worksheet
- Identify the exact model. Record parameter count, layers, hidden size, KV heads and head dimension from the model’s configuration.
- Choose weight and KV formats. Convert each to bytes per element; do not assume they match.
- Set the workload envelope. Record maximum prompt tokens, maximum generated tokens, total context, concurrent requests and batch policy.
- Calculate weights. Use
parameters × weight bytes ÷ tensor-parallel degree. - Calculate KV cache. Use the general KV-head formula, multiplying by the planned context and batch or concurrency.
- Add non-model allocations. Include activations, I/O tensors, workspaces, communication buffers, adapters, multimodal components and allocator reserve.
- Compare with process-visible memory. Leave margin rather than targeting 100 percent utilization.
- Validate on the target engine. Load the model, exercise the maximum context and concurrency, and record peak allocated and reserved memory as well as latency and throughput.
A repeatable training worksheet
- Classify the job. Distinguish full fine-tuning, adapter tuning and pretraining; identify whether parameters are frozen.
- Record precision and optimizer state. Write down low-precision parameters, master copies, gradient format and every optimizer tensor.
- Select the framework’s accounting model. Use the documented bytes-per-parameter or per-rank formula for the optimizer and sharding strategy in use.
- Calculate local model state. Apply parameter and state sharding to the most-loaded rank, not just an average.
- Estimate activations. Use local microbatch, sequence length, layer count, hidden size and checkpointing policy.
- Add temporary and communication memory. Include gradient buckets, NCCL or equivalent buffers, workspaces, graph allocations and offload staging.
- Profile a representative step. Run the real sequence length and microbatch, include evaluation or checkpoint phases if they peak higher, and capture the maximum across ranks.
Validation: turn a calculation into an operational answer
Use a representative target configuration before committing a long training run or production deployment.
- Load the exact checkpoint and quantization artifacts.
- Use the intended driver, CUDA version, framework, engine and kernel settings.
- Test the maximum prompt-plus-generation length, not only a short sample.
- Drive the planned number of concurrent requests or the actual training microbatch.
- Measure peak allocated and reserved memory on every GPU; the fullest rank is the limit.
- Repeat after enabling adapters, CUDA graphs, paged KV cache, speculative decoding or other production features.
- Record throughput and latency along with memory. A configuration that fits but misses service-level targets may require a different parallelism or batching strategy.
Estimators can omit fragmentation, workspaces, communication buffers, graph allocations, routing imbalance and other runtime-specific costs. A measured peak on the target stack is therefore stronger evidence than a spreadsheet result.
When the model does not fit
Inference out of memory
- Reduce maximum context or generated-token limits; both consume KV cache.
- Lower concurrent requests or the effective batch size.
- Use a supported lower-precision weight or KV-cache format after checking quality and hardware support.
- Enable the engine’s documented KV-cache management or paging features.
- Use tensor or pipeline parallelism, then retest communication overhead and latency.
- Move to a GPU with more usable memory only after confirming that the workload, not a leaked allocation or another process, caused the failure.
Training out of memory
- Lower the local microbatch and recover global batch with gradient accumulation if training semantics allow.
- Shorten sequence length or enable activation checkpointing/recomputation.
- Use optimizer, gradient or parameter sharding supported by the framework.
- Reduce temporary bucket sizes or disable graph capture when those allocations are the peak.
- Use parameter-efficient tuning when its frozen-model and adapter states meet the quality requirement.
- Inspect the fullest rank for imbalance instead of relying on an average across GPUs.
Unexpected peak after a successful load
If loading succeeds but the first request or training step fails, the missing category is often KV cache, activations, a kernel workspace, communication buffers or graph capture. Capture a memory trace around the failing operation and compare the runtime’s allocation report with the worksheet categories.
Common estimation mistakes
- Calling parameter count the GPU requirement. It describes neither KV cache nor training state.
- Using one training multiplier for every job. Optimizer, precision, sharding and checkpointing change the accounting.
- Using hidden size in every KV formula. GQA and multi-query models require actual KV-head count and head dimension.
- Counting only the prompt. Generated tokens remain in the KV cache until the sequence is finished or evicted.
- Dividing every allocation by tensor-parallel degree. Some runtime buffers, replicated modules and caches follow different layouts.
- Comparing with advertised card memory. The process sees less than the nameplate total when the system or another process consumes memory.
- Treating a published example as a guarantee. The 8B, 7B, 70B, Mistral-7B and 4B examples are configuration-specific illustrations.
- Ignoring performance after sharding. A multi-GPU fit still needs latency and throughput testing.
Bottom line
For inference, calculate weight bytes, then calculate architecture-specific KV cache for the full prompt-plus-output workload, and finally add runtime allocations. For training, calculate the optimizer-specific model state and add activation, temporary and communication memory for the local microbatch. Record every assumption, leave headroom, and profile the exact stack before treating the result as safe capacity.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
