iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Measure the workload before buying hardware. Run a controlled test with the exact model revision, quantization, prompt and output limits, context lengths, concurrency, batching policy, and service-level objectives you intend to operate. Record GPU memory, model-weight and KV-cache usage, preemptions, queue depth, time to first token (TTFT), tail latency, throughput, and active requests as time series. A persistent vllm:kv_cache_usage_perc value above 0.9, any nonzero vllm:num_preemptions, or a growing waiting queue while latency misses its objective is evidence of a memory-related capacity limit. Then repeat the same test with the proposed larger GPU, additional GPU, memory-sharing fabric, host-memory tier, fractioning, or swap design. Choose the least complex option that meets latency, throughput, concurrency, and burst-headroom targets.
What you are actually measuring
“Running out of memory” is not a single number. An LLM server uses memory for model weights, runtime allocations, activations, communication buffers, and the key-value (KV) cache that stores attention state for active sequences. The amount required changes with model revision, quantization, context length, batching, prompt mix, output length, and concurrent requests. A server can therefore fit a model at idle and still fail when several long requests arrive together.
Separate three questions:
- Can one replica load? If model weights plus required runtime reservations do not fit on one device, loading fails before serving begins.
- Can the replica sustain the target workload? A model may load but exhaust KV-cache space as concurrency or context length rises.
- Is memory the bottleneck? High compute utilization with spare memory points to a compute, batching, or scheduler limit instead of a memory-expansion requirement.
Use a sustained test, not a single peak reading. A brief spike may be harmless if requests complete, queues drain, and the service-level objective (SLO) remains satisfied. Persistent pressure, preemption, or queue growth is the actionable signal.
Recommended Free Tools
Metrics that prove a capacity limit
| Metric | What it tells you | Interpretation |
|---|---|---|
vllm:kv_cache_usage_perc |
Fraction of the KV-cache capacity in use. | NVIDIA AIPerf identifies a value above 0.9 as KV cache near full capacity. Treat a value that stays above 0.9 during the target load as a capacity limit, not merely a transient peak. |
vllm:num_preemptions |
Requests or sequences preempted because memory could not support them concurrently. | NVIDIA AIPerf identifies any value above zero as memory pressure causing preemptions. Correlate the event with latency and queue data. |
| Waiting-request or queue depth | Demand that cannot start immediately. | A queue that grows under a steady offered load indicates saturation. If memory is near its limit at the same time, expansion or pooling may be required. |
| GPU memory used and free | Total device allocation, including weights, cache, and runtime buffers. | Use a time series. Free memory disappearing as concurrency or context grows supports a capacity diagnosis; a flat memory footprint with high compute load does not. |
| Model-weight footprint | Memory consumed before serving traffic. | Compare the footprint with device capacity and the headroom needed for KV cache and runtime allocations. A failed weight load is a hard fit limit. |
| TTFT and p99 latency | Startup delay and worst-case response behavior for the chosen percentile. | Memory pressure often appears as rising TTFT, long tails, or preemption-related stalls. Always evaluate against your stated SLO, not an arbitrary latency number. |
| Token throughput | Completed output (and, where tracked, input) tokens per unit time. | Throughput that falls while queues grow can indicate memory-driven batching limits; throughput that falls with spare memory may indicate compute saturation. |
| Active requests and concurrency | How many sequences are resident at each moment. | Use this to relate KV-cache consumption to the workload shape and to identify the concurrency at which the service stops meeting its SLO. |
The 0.9 and greater threshold and greater than zero preemption signal come from NVIDIA’s current AIPerf metric reference. They are diagnostic indicators, not a universal promise that every deployment fails at exactly the same point; validate them against your own SLO and engine configuration.
#1 Best Overall
- Valued Carpenter Pencil Set: You will get 2 pcs solid carpenter pencils with 26 piece 2.8 mm refills, 1 replaceable sharpener, 1 plastic storage box.The complete carpenter pencils combination allows you to finish your work faster and more easily
- Deep Hole Marker Pencil: The deep-hole construction pencils adopts 45mm elongated tip design, which is more convenient to mark in the small hole or in other tight areas that other carpenter markers cannot reach
- Carpenter Pencils with Sharpener: The sharpener is screwed into the top of the work pencil, which won't get lost either. Built-in pencil sharpener that keep the lead with pointed and smooth to Improves line of sight in fine work
- Stronger Solid Lead: This work pencil is matched with a 2.8 mm thick lead , which is much thicker and stronger during the drawing process of construction work, it will not break or damage easily
- Marks on Various Surfaces: 3 colors solid construction pencil can marks on various surfaces,such as metal, plastic, wood, paper etc. Ideals for woodworkers, contractors, craftsmen, builders, merchants and masons
Build a representative test matrix
Freeze the variables that change memory demand before comparing hardware. Record them with every run so a later result is comparable.
- Model: exact checkpoint or model revision, tokenizer, serving-engine version, and quantization.
- Prompt distribution: representative input lengths rather than one convenient prompt.
- Output limits: maximum and typical generated-token limits.
- Context lengths: include the longest context you promise to support; larger contexts consume more KV cache.
- Concurrency and arrival pattern: target concurrency, ramp rate, burst behavior, and whether traffic is open-loop or closed-loop.
- Batch policy: continuous batching, maximum batch size, scheduling policy, and any admission limits.
- SLOs: TTFT, p99 or another agreed latency percentile, throughput, error rate, and acceptable queueing.
- Platform: GPU model, host RAM, chassis, driver, operating system, topology, and serving configuration.
Do not compare a short-context, low-concurrency run on one design with a long-context burst on another. Keep prompts, load shape, and measurement windows identical when testing a proposed change.
Seven-step measurement procedure
- Freeze the matrix. Write down the model revision, quantization, prompt and output distributions, context limits, concurrency steps, batching policy, and SLOs. Keep the test client and server configuration fixed.
- Establish a low-load baseline. Run enough traffic to observe normal behavior, then record GPU memory used/free, model-weight footprint, system RAM, KV-cache ratio, TTFT, p99 latency, throughput, queue depth, and active requests. Capture time series rather than only the maximum value.
- Increase demand in controlled steps. Raise concurrency and context length separately where possible, then test their combined worst case. Hold each step long enough to see whether queues drain and whether cache usage settles or keeps climbing.
- Mark the first failing point. Note the first step at which
vllm:kv_cache_usage_percremains above 0.9,vllm:num_preemptionsbecomes nonzero, waiting requests grow, or latency breaches the objective. Record the corresponding active-request count and context distribution. - Separate capacity from compute saturation. Compare memory readings with GPU utilization, token throughput, and queueing. High compute load with substantial memory headroom suggests a compute or scheduling bottleneck. Near-full KV cache, preemptions, or queue growth with available compute capacity supports a memory diagnosis.
- Re-run with the candidate design. Use the same prompts, arrival pattern, ramp, measurement window, and SLO. Test a larger-memory GPU, an added GPU, a different usable-memory fraction, CPU-memory swap, NVLink/NVSwitch, or CXL configuration only after documenting the baseline.
- Choose on headroom and failure behavior. Select the least complex design that meets p99 latency, throughput, target concurrency, and burst headroom. Document what happens when the limit is reached: queued requests, rejected requests, preemptions, degraded latency, or replica failure.
How vLLM settings affect the result
vLLM allocates model weights before sizing the KV cache. Its gpu_memory_utilization setting controls the usable fraction of device memory after profiling; it does not create additional physical memory. Raising the fraction can leave less safety margin for runtime allocations and other processes, so validate stability rather than maximizing the setting blindly.
The max-model-len setting affects the reservation available for KV cache. A larger maximum context can reserve more cache memory even when most requests are shorter. Test the maximum context you genuinely need, and do not infer production headroom from an artificially low value. Compare cache usage, preemptions, and tail latency after every change.
Rank #2
- Ergonomically Designed: Work in tight areas with a compact design that gets into tough spots
- Both tools are designed to fit into difficult to reach spaces
- Complete a wide range of applications with a high-performance motor that delivers up to 300 UWO, and an impact driver delivering up to 1,400 in-lbs of torque
- Both the DEWALT impact driver and drill driver feature integrated LED work lights with a convenient 20-second delay, ensuring enhanced visibility in dimly lit or challenging work areas
- One-Handed Loading - Keep one hand free with a 1/4 in. hex chuck that accepts 1 in. bit tips
When physical memory expansion is the right answer
Add physical GPU or system memory when a single replica cannot load, KV-cache allocation fails, or measured pressure is sustained on one device and degrades the SLO. A larger-memory accelerator is the clearest remedy for a hard single-device fit limit because it avoids relying on remote memory for every token operation.
Larger-memory GPU
NVIDIA’s current HGX AI Factory reference lists these per-GPU capacities:
| GPU | Memory listed in the reference | What the figure means |
|---|---|---|
| H100 | 80 GB HBM3 | Per GPU in the cited enterprise reference architecture; usable serving capacity is lower after weights, runtime allocations, and configured headroom. |
| H200 | 141 GB HBM3e | Per GPU in the same reference; verify the exact board, server, firmware, power, and cooling support. |
| B200 | 180 GB HBM3e | Per GPU in the same reference; availability and platform compatibility depend on the deployment date and system design. |
These are capacity figures, not a throughput guarantee. Before replacing a GPU, check chassis clearance, power delivery, cooling, PCIe or NVLink topology, firmware, driver, operating-system support, and the serving engine’s support for the specific model.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Additional GPUs
Adding GPUs can increase aggregate capacity when the model and serving stack support tensor, pipeline, or other forms of parallelism. It also introduces communication and scheduling overhead. Measure the complete replica, not the sum of isolated device capacities: synchronization, replicated weights, communication buffers, and topology determine the memory available to the model and cache.
Rank #3
- 【Great Compatibility】This Katerk 1/4 inch hex shank bit holder is specifically designed for 1/4 inch hex shank drill bits. It's compatible with most 1/4 fast hex handles, hex sockets, various electric screwdrivers, and handheld screwdrivers. The bit holder makes it a valuable addition for any handyman.
- 【Secure and Safe】Built with a secure backup nut design, each drill bit holder securely locks onto your bits, ensuring they stay firmly in place. Additionally, our bit holder incorporates a high-quality steel ball rolling design that holds up to several kilograms of weight, ensuring your various drill bits don't fall off.
- 【Easy One-Handed Operation】The bit holder for impact driver allows you to change bits single-handedly, simplifying your workflow. Its multi-color design further allows for quick identification of the drill bit you need.
- 【Compact and Convenient】Thanks to its compact size, this 1/4 inch bit holder is easy to carry around. The bit holder allows for easy attachment to various tools, making this a convenient addition to your construction accessories. The Katerk bit holder is cast from high-quality alloy material, promising a long product lifespan. Despite its rugged strength, the bit holder remains lightweight, making it portable.
- 【Cool Christmas Gift For Men Stocking Stuffers】 This screwdriver bit holder, driver bit holder, impact bit holder, can be given as a gift to your loved one, especially for anyone involved in construction or electrical work. It's a must-have for stocking stuffers for men and women, tools gifts for dad, tech gadgets for men, gifts for dad, gifts for him, gifts for husband, gifts for boyfriend, cool gadgets for men, and cool gifts for dad.
More host RAM
Additional system RAM helps with operating-system, preprocessing, staging, and some offload designs. It does not increase local GPU HBM capacity by itself. Treat host memory as an expansion tier only when the engine explicitly supports the required offload or swap path, and measure its effect on TTFT, p99 latency, throughput, and isolation.
When pooling or sharing is the better design
Pooling is a topology and software decision, not simply “adding up” memory labels. The path between the GPU and the remote memory, the serving engine’s memory-management support, and failure behavior determine whether pooled capacity helps your workload.
NVLink and NVSwitch GPU-memory sharing
NVIDIA documents NVSwitch as a high-bandwidth, low-latency fabric for peer-to-peer GPU communication and pooled GPU memory on supported systems. NVIDIA’s IMEX guide describes a multi-node NVLink architecture in which GPUs can access memory from other nodes within an NVLink domain; it extends the multi-GPU NVLink peer-to-peer programming model to multiple compute nodes.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →This approach is suited to model parallelism or shared state that needs high-bandwidth access. Support is constrained by the GPU platform, switch and cabling topology, hypervisor, operating system, required fabric services, and software versions. A nominally connected PCIe system is not equivalent to an NVLink or NVSwitch domain. Verify the topology presented to the serving process and test remote-memory traffic under the target load.
Rank #4
- Long Nib and Deep Hole Marker: Our mechanical carpenter pencil with 45mm nib is designed for easy marking of deep holes or narrow areas. These construction pencils are the great choice for woodworking tools, construction tools, carpenter tools, contractor tools, wood carpentry tools and architect tools
- Extra Refills in 2 Colors for Versatile Marking: The construction mechanical pencil comes with 12 extra 2.8mm refills, including 6 red and 6 black refills. The black refill is suitable for light surfaces, while the red wax is perfect for dark surfaces. Our carpenter mechanical pencil makes sure that you'll have an ample supply for extended use
- Built-in Sharpener: Our construction pencil comes with a built-in sharpener to ensure the mechanical pencil tip is always sharp and ready for use. Never buy an extra pencil sharpener again. A great tool for any woodworker pencil, contractor pencils. The refill can easily be extended or retracted with a simple click of the pencils mechanical, allowing you to work more efficiently and accurately
- Portable Clip Design: Our deep hole construction pencil features a portable clip design, easy to carry and attach to your pocket or tool box, so that you can keep the carpenter pencils mechanical close at hand, making it a convenient tool to have on the go. Great gifts choice for carpenters
- Stronger Pencil Lead: The black refills are made of lead, sturdy and smooth. The red refills are made of wax, clear and light. These marking pencils are much thicker and stronger than normal pencils during the marking process of construction work, suitable for various surfaces, such as glasses, metal, boards, floors, walls, furniture, etc. The written marks can be easily wiped with a wet paper towel when needed
CXL host-memory expansion and multi-server pooling
Compute Express Link (CXL) is an open interconnect standard described in a 2023 arXiv introduction as supporting accelerator attachment, memory-capacity expansion, multi-server resource pooling, and peer-to-peer communication. CXL can increase effective host capacity or provide a shared memory tier, but it should not be assumed to deliver GPU-HBM latency or bandwidth.
Benchmark the actual model path: which tensors move to the CXL tier, how often they move, what paging or eviction occurs, and how p99 latency behaves during bursts. A larger apparent address space is useful only if the resulting transfer and tail-latency costs remain inside the SLO.
Memory fractions and CPU-memory swap
NVIDIA Run:ai documents GPU-memory fractions for dividing a GPU among heterogeneous workloads. Fractioning can raise utilization, broaden access, and shorten startup waits when workloads do not all need the full device, but it also reduces the memory available to each allocation. Test isolation and interference at peak concurrency.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Run:ai’s GPU memory-swap documentation describes extending effective GPU memory into CPU memory so more workloads can run on one physical GPU. Swap can postpone a hardware purchase or absorb occasional bursts, but every transfer can affect TTFT, throughput, and tail latency. Measure transfer overhead, eviction behavior, and what happens when CPU memory or the interconnect is saturated.
Best Value
- Milwaukee Ink all Fine Point Marker, Black, 4 Per Pack
- 4 per pack Features Clog Resistant Marker Tip Writes through Dusty, Wet and Oily Surfaces Durable Marker Tip for Writing on Concrete, OSB and Rough Surfaces
- Clog resistant tip writes on dusty, wet and oily surfaces and is optimized for rough surfaces such as OSB, cinderblock and concrete
- Hard hat clip- attaches for easy access
- Quick dry time with reduced smearing and marking
Expansion versus pooling: decision comparison
| Decision axis | Physical expansion | Pooling or sharing |
|---|---|---|
| Model-fit capacity | A larger local GPU directly removes a single-device fit limit; additional GPUs require supported parallelism. | Can expose aggregate or remote capacity, but only for software and topologies that support the memory path. |
| KV-cache headroom | More local HBM generally gives predictable cache space and lower remote-access dependence. | May increase effective capacity, but cache placement, transfers, and eviction must be measured. |
| p99 latency | Usually easier to reason about when the working set remains local; validate new hardware and clocks. | Can preserve SLOs on high-bandwidth fabrics, while CXL or CPU swap may add tail latency. |
| Throughput | Depends on GPU compute, memory bandwidth, and parallelism; do not infer it from capacity alone. | Depends on interconnect bandwidth, contention, synchronization, and paging behavior. |
| Interconnect topology | Requires compatible PCIe, NVLink, or multi-GPU design for multi-device serving. | Topology is central: NVLink/NVSwitch or CXL capabilities must match the platform and software. |
| Software support | Requires driver, firmware, operating-system, and engine support for the new device. | Also requires memory-sharing, fabric, scheduler, hypervisor, and placement support. |
| Failure isolation | A failed device can take down a replica, but the fault domain is comparatively direct. | A fabric, switch, pooled-memory device, or remote node can become a shared fault domain. |
| Power and cooling | Higher-memory accelerators may require more power, airflow, or chassis changes. | Switches, links, memory devices, and additional nodes add their own power and cooling demands. |
| Operational complexity | Hardware replacement and validation, followed by a simpler memory path. | More topology, placement, fabric-service, monitoring, and failure-mode dependencies. |
| Cost per served request | Compare purchase and operating cost with the extra stable throughput and headroom. | Include fabric, nodes, memory devices, software, power, and latency-related capacity loss. |
Compatibility checklist before purchasing
- GPU and board: exact SKU, memory type and capacity, board form factor, and supported clocks.
- Chassis and power: slot spacing, power connectors, PSU capacity, airflow, thermal limits, and rack constraints.
- Bus and fabric: PCIe generation, lane allocation, NVLink/NVSwitch topology, switch firmware, and cable or backplane requirements.
- Host platform: CPU, system RAM, NUMA layout, BIOS, firmware, and virtualization or hypervisor configuration.
- Software: driver, CUDA or equivalent runtime, operating-system release, serving engine, scheduler, and memory-management features.
- Failure behavior: what occurs if a peer GPU, link, switch, CXL device, or host-memory tier disappears.
- Observability: verify that GPU memory, KV-cache ratio, preemptions, queue depth, TTFT, p99 latency, throughput, and active requests remain visible after the change.
Common measurement mistakes and fixes
Using idle memory as proof of capacity
An idle model shows weight fit, not production capacity. Repeat the test with representative concurrency and context lengths so KV-cache demand is exercised.
Looking only at total GPU utilization
A high utilization percentage can mean compute saturation, while a lower percentage can coexist with a full KV cache and stalled requests. Correlate utilization with memory, cache, queues, latency, and throughput.
Testing one prompt length
Short prompts understate cache demand for long-context traffic. Include the distribution you promise and a deliberate worst-case context test.
Changing several variables at once
If you alter model quantization, context limit, concurrency, and memory fraction together, you cannot identify the cause of improvement or failure. Change one design variable per comparison where practical.
Relying on a single peak sample
Use time-series data and sustained windows. Capacity pressure is demonstrated by persistent near-full cache, recurring preemptions, or queues that do not drain, not by one instantaneous reading.
Assuming pooled capacity is local capacity
Remote HBM, CXL memory, and CPU swap have different bandwidth and latency. Test the model’s real access pattern and tail behavior, especially during bursts and concurrent transfers.
How to turn measurements into a purchase decision
- Define the required operating point. State the model, context promise, concurrency, throughput, TTFT, p99 latency, and burst duration that the service must support.
- Locate the first failing point. Use the metric thresholds and queue/SLO evidence to identify whether the limit is weight fit, KV cache, compute, interconnect, or scheduling.
- Estimate required headroom. Base it on observed burst behavior and growth plans, not on the device’s advertised total memory. Reserve room for runtime allocations and operational processes.
- Shortlist designs that address the diagnosed limit. Choose a larger local GPU for a hard fit or persistent local pressure; consider NVLink/NVSwitch when high-bandwidth model parallelism or shared state is supported; consider CXL or swap for a deliberately measured slower tier; use fractioning for controlled multi-tenant partitioning.
- Re-test unchanged traffic. Keep the original matrix and compare p99 latency, throughput, queue depth, preemptions, KV-cache ratio, and burst headroom. A design that fits the model but misses the SLO is not a successful expansion.
- Record limits and rollback. Document supported topologies, software versions, maximum context, admission behavior, and the recovery path if a pooled device or fabric fails.
The practical rule is simple: expand local memory when the working set must remain local and the current device cannot provide stable headroom; pool memory when the platform and serving stack explicitly support the required topology and your measurements show that remote access still meets the SLO. The test, not the memory label, determines which choice is justified.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
