iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Estimate local-LLM GPU memory in three layers: model weights, KV cache, and runtime headroom. A useful first pass is parameter count × bytes per parameter ÷ tensor-parallel degree for the weights, then add memory for the context window, generated tokens, concurrent requests, CUDA workspaces and other applications. In practice, compare that total with free VRAM, not the capacity printed on the graphics card.
As rough anchors, an 8B model in BF16 needs about 16GB for weights, a 13B model about 26GB, and a 70B model about 140GB. Those figures are not complete system requirements: long context and batching can add many gigabytes, and a 24GB card may run an 8B model comfortably while failing on a 13B model unless you quantize or offload.
How do I calculate the weight memory for a local LLM?
Multiply the model’s parameter count by the number of bytes used for each parameter, then divide by the tensor-parallel (TP) degree when the model is split across GPUs:
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →weight memory per GPU = total parameters × bytes per parameter ÷ TP
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
This is the memory occupied by the stored weights only. It is the best starting point for sizing, but it does not include the KV cache, temporary activations, CUDA allocations or your desktop.
Use the model’s actual precision
| Format | Approximate bytes per parameter | Quick estimate |
|---|---|---|
| BF16 or FP16 | 2 bytes | About 2 × the parameter count in GB |
| FP8 | 1 byte | About 1 × the parameter count in GB |
| INT4 or NVFP4 | 0.5 byte | About 0.5 × the parameter count in GB |
NVIDIA’s current NIM documentation uses 2 bytes for BF16/FP16, 1 byte for FP8 and 0.5 byte for INT4/NVFP4. Hugging Face’s Transformers documentation expresses the BF16/FP16 rule as: loading a model with X billion parameters requires roughly 2 × X GB of VRAM. These are decimal, planning-level estimates; framework metadata and allocator rounding make an actual allocation different.
Account for tensor parallelism
With TP=1, one GPU must hold the complete weight set. TP=2 ideally divides the weights between two GPUs, and TP=4 divides them four ways. The division is not free: communication buffers and replicated layers consume additional memory, so treat the formula as a lower bound rather than a guarantee.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat memory does the KV cache add?
The KV cache stores attention keys and values for tokens already in the prompt or generated output. It grows with context length, the number of tokens generated, and the number of simultaneous sequences. A model can therefore load successfully and still run out of VRAM when a long prompt arrives or a second user starts a request.
Context and concurrency are multiplicative pressure
Maximum context is a reservation target, not merely a marketing number. A server configured for 128K tokens may reserve substantially more cache than a desktop chat configured for 8K, even when both use the same weights. Batch size and concurrent users multiply the cache requirement because each active sequence needs its own token history.
NVIDIA Developer reported that Llama 3 70B at a 128K context can use about 40GB of KV cache for one user. That figure is a single-user example under the stated model and context settings, not a universal constant; different attention implementations, precisions and backends produce different values.
Prompt tokens and output tokens both count
The cache normally retains the prompt as well as newly generated tokens. A request with a 60K-token document and a 4K-token answer can therefore require a cache sized for roughly 64K tokens, subject to the backend’s eviction and sliding-window behavior. If you only budget for the answer length, a large prompt can trigger an allocation failure.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
How much headroom should I leave?
Reserve memory for activations, the CUDA context, temporary workspaces, communication buffers, CUDA graphs, adapters such as LoRA, allocator fragmentation and unrelated applications. The exact amount depends on the inference engine and workload.
NVIDIA’s NIM example sets gpu-memory-utilization to 0.9 by default, reserving 90% of total GPU memory for model operations and leaving 10% unreserved. That is a backend heuristic, not a rule that every engine follows. A desktop session, browser with GPU acceleration or another process can consume part of the remaining capacity, so measure the free memory at the time you will actually launch the model.
What is the practical step-by-step sizing method?
Use the following sequence before downloading a large checkpoint or configuring a server.
- Identify the exact checkpoint. Record parameter count, architecture, weight format, intended maximum context and whether the published size is for a full-precision or quantized file. A 7B model in BF16 and a 7B model in 4-bit are different sizing problems.
- Compute the weight floor. Multiply parameters by bytes per parameter. For a single GPU, TP is 1. For multiple GPUs, divide by the planned TP degree, then add a margin for imperfect partitioning.
- Set a realistic context target. Use the largest prompt plus generated output you will permit in production, not only the model’s advertised maximum. If users usually work at 8K tokens, sizing for 128K can waste most of the card.
- Estimate concurrency or batch. One interactive user and eight simultaneous requests have very different KV-cache needs. Include continuous batching if the server will admit new sequences while others are generating.
- Add runtime headroom. Allow space for activations, CUDA graphs, temporary allocations, adapters and fragmentation. A configuration that leaves only a few hundred megabytes free is fragile even if the weights technically fit.
- Measure free VRAM. Subtract the peak use of the desktop, display server and other processes. Compare your estimate with this free value rather than with the card’s advertised total.
- Validate with the real backend. Start with the intended context and concurrency, inspect allocation logs, then test a long prompt and the expected generation length. Different engines reserve memory at load time, during KV-cache creation or during CUDA-graph capture.
How much VRAM do 7B, 13B and 70B models need?
The table below gives weight-only planning figures. Every row still needs KV-cache and runtime space.
| Model size and example | Precision | Weight memory | What that means |
|---|---|---|---|
| 7B-class, such as Mistral 7B | BF16 | 13.74GB | Hugging Face’s current Transformers figure; a 16GB card has little room for cache and overhead. |
| 7B-class, such as Mistral 7B | 8-bit | 6.87GB | Hugging Face’s current Transformers figure; remaining capacity still depends on context and backend allocations. |
| 8B-class, such as Llama 3.1 8B | BF16 | About 16GB | NVIDIA says it can fit on a 24GB RTX 4090 with room for KV cache and overhead under its example configuration. |
| 13B-class | BF16 | About 26GB | Normally exceeds a 24GB card before cache or overhead; use quantization, multiple GPUs or CPU offload. |
| 70B-class, such as Llama 2 70B | BF16 | About 140GB | Hugging Face’s current Transformers figure; a single consumer GPU is insufficient. |
| 70B-class | BF16 with TP=4 | About 35GB per GPU | The arithmetic split only; each GPU also needs cache, communication buffers and headroom. |
The Mistral figures are model-specific published estimates, while the 8B, 13B and 70B rounded values come from the parameter-count rule. Actual files can be slightly larger or smaller because of embedding layers, metadata and implementation details.
Will an 8B model fit on a 24GB graphics card?
Usually, an 8B BF16 model is a plausible starting point on a 24GB card: roughly 16GB goes to weights, leaving about 8GB for the KV cache and runtime. Whether it remains stable depends on the chosen context, concurrent requests, desktop usage and backend reservations.
NVIDIA’s Llama 3.1 8B example specifically describes a 24GB RTX 4090 running the BF16 weights with room for cache and overhead. Do not generalize that result to every 24GB card: memory capacity is only one axis, and bandwidth, compute throughput, driver support and inference-engine kernels affect speed and usable performance. A card used for display may also have substantially less than 24GB free.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
How does quantization change the requirement?
Quantization reduces weight memory by storing parameters with fewer bits. As a first approximation, multiply parameter count by the quantized bytes-per-parameter value, then add the same KV-cache and headroom budgets used for full precision.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches| Example | Weight-only estimate | Important qualification |
|---|---|---|
| 13B at BF16 | About 26GB | Too large for most 24GB cards before non-weight memory. |
| 13B at 4-bit | About 6.5GB | Leaves more room, but the actual file and runtime may use higher-precision tensors. |
| 70B at 4-bit | About 35GB | Still generally requires a larger or multi-GPU system once cache and overhead are included. |
GGUF and other quantized files are not always a pure 0.5-byte-per-parameter product. Some tensors may remain at higher precision, and quantization metadata adds space. Treat the file size as a better weight estimate than the label alone, then budget separately for the KV cache.
What can I do when the model does not fit?
Choose a smaller or lower-precision checkpoint
Moving from 13B BF16 to 13B 8-bit or 4-bit can bring the weights under a 24GB limit. Moving from a 70B model to a 7B or 8B model may preserve responsiveness when the hardware cannot support the larger model. Check quality requirements before trading precision or parameter count for capacity.
Reduce context and concurrency
Lower the maximum context length, cap output tokens and reduce simultaneous sequences. This directly limits KV-cache growth and is often safer than forcing a model to load with no free memory.
Use multiple GPUs with tensor parallelism
Increase TP so the weight set is distributed across cards. Four-way TP turns roughly 140GB of 70B BF16 weights into about 35GB per GPU before overhead. Inter-GPU links, PCIe topology and backend support determine whether the resulting throughput is acceptable.
Free tools Windows power users keep installed
One-click scans. No signup required.
Offload part of the model to system RAM
CPU offload can make an otherwise oversized model start, but every token must move data between host and GPU. Expect lower throughput and higher latency, and reserve enough system RAM for the offloaded layers, operating system and other processes.
Which GPU specifications matter besides VRAM?
Capacity decides whether a configuration can fit; it does not decide how fast it will run. Compare systems using the same workload assumptions:
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
- Usable free VRAM: total capacity minus display and other-process usage.
- Model and precision: parameter count, BF16/FP16, FP8, 8-bit or 4-bit weights.
- Context target: typical and maximum prompt-plus-output tokens.
- Concurrency or batch size: number of active sequences and continuous-batching policy.
- Parallelism or offload: TP degree, CPU layers and the interconnect between GPUs.
- Backend reservations: memory-utilization limits, CUDA-graph capture and workspace requirements.
- Throughput hardware: memory bandwidth, tensor-core performance, power limits and cooling.
When shopping for a 24GB VRAM graphics card, use the 24GB figure only as a capacity filter. Two cards with the same capacity can deliver very different tokens-per-second results, and a display workload can reduce the memory available to either one.
How do I diagnose an out-of-memory error?
It fails while loading weights
The weight floor is already above usable VRAM, or the backend has reserved additional buffers. Select a smaller model, lower precision, more GPUs or a higher TP degree. Do not solve a load-time failure by merely shortening context; the weights must fit first.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →It fails while allocating the KV cache or during profiling
The weights loaded, but the requested context or concurrency did not. Lower maximum context, reduce batch or concurrent users, shorten output limits, or enable a quantized KV cache if the backend supports one.
It fails during warmup or CUDA-graph capture
Temporary allocations and captured graphs need more room than the steady-state estimate. Lower the backend’s memory-utilization target, disable graph capture if appropriate, reduce batch size and inspect the engine’s allocation log.
It fails only when the desktop is active
Measure peak graphics usage with the display and browser running, then subtract that peak from total VRAM. A model that works on a headless server may fail on a workstation because the desktop consumed the missing margin.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Worked sizing decisions
8B BF16 on 24GB
Start with about 16GB for weights. The nominal remainder is about 8GB, but reserve part of it for CUDA and activations. An 8K context and one user may be comfortable; a 128K context or multiple users can exhaust the card despite the successful load.
13B BF16 on 24GB
About 26GB is required just for weights, so this configuration normally cannot fit. A 4-bit or 8-bit build, a larger card, TP across GPUs or CPU offload is required before context planning begins.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
70B BF16 across four GPUs
Divide the roughly 140GB weight set by four for about 35GB per GPU, then add KV cache, communication buffers and headroom on every card. A 128K single-user workload can add about 40GB of KV cache in the cited Llama 3 70B example, so the context target may require more GPUs or a shorter window even after the weights are distributed.
What should I record in a deployment plan?
- Exact model family, checkpoint revision and parameter count.
- Weight format and the measured file size.
- GPU model, total VRAM and free VRAM at launch.
- Target context, maximum output and concurrent sequences.
- TP degree, CPU-offload layers and interconnect.
- Backend memory-utilization setting and whether CUDA graphs or adapters are enabled.
- Observed peak VRAM during load, warmup, long-prompt processing and generation.
This record turns a rough estimate into a repeatable capacity limit. Recheck it whenever you change the model, quantization, context policy, driver, backend or number of users.
FAQ
Does system RAM replace GPU VRAM for a local LLM?
System RAM can hold offloaded layers, but it does not provide the same bandwidth as VRAM. Offloading may make a model launch when it otherwise would not, while substantially reducing tokens per second and increasing latency. Keep enough RAM for the operating system and the offloaded portion instead of treating it as a free extension of the card.
Recommended Free Tools
Should I size for the model’s maximum advertised context?
Only if your application will actually accept that many tokens. Set the limit from the largest real prompt plus output you need; otherwise a lower cap leaves more VRAM for users, batching and runtime safety. A model’s advertised maximum is an architectural limit, not a promise that every GPU can allocate it.
Can two different GPUs combine their VRAM automatically?
No. The inference backend must explicitly support tensor parallelism, pipeline parallelism or another sharding method. Mixed cards can be limited by the smaller card, the slowest link or uneven partitioning, so total capacity alone does not predict the result.
How can I check free NVIDIA GPU memory before launching?
Run nvidia-smi and inspect the memory-used and memory-free columns while your normal desktop workload is active. For a scriptable snapshot, use nvidia-smi --query-gpu=memory.free,memory.total --format=csv; repeat the check during a long prompt because cache growth can change the peak.
Frequently Asked Questions
Does system RAM replace GPU VRAM for a local LLM?
System RAM can hold offloaded layers, but it does not provide the same bandwidth as VRAM. Offloading may make a model launch when it otherwise would not, while substantially reducing tokens per second and increasing latency. Keep enough RAM for the operating system and the offloaded portion instead of treating it as a free extension of the card.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Should I size for the model’s maximum advertised context?
Only if your application will actually accept that many tokens. Set the limit from the largest real prompt plus output you need; otherwise a lower cap leaves more VRAM for users, batching and runtime safety. A model’s advertised maximum is an architectural limit, not a promise that every GPU can allocate it.
Can two different GPUs combine their VRAM automatically?
No. The inference backend must explicitly support tensor parallelism, pipeline parallelism or another sharding method. Mixed cards can be limited by the smaller card, the slowest link or uneven partitioning, so total capacity alone does not predict the result.
How can I check free NVIDIA GPU memory before launching?
Run nvidia-smi and inspect the memory-used and memory-free columns while your normal desktop workload is active. For a scriptable snapshot, use nvidia-smi --query-gpu=memory.free,memory.total --format=csv; repeat the check during a long prompt because cache growth can change the peak.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
