iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
There is no reliable “one server” specification for AI inference. Size the system from the workload outward: identify the model and precision, quantify prompt and output lengths, concurrency, request rate, and latency targets, then benchmark a candidate configuration under the same conditions your service will see. GPU memory usually sets the first limit, but CPU scheduling, host memory, storage, interconnects, and the serving engine can determine whether that GPU is actually usable.
Start with a workload specification, not a hardware list
Model parameter count is only one input. Two services using the same model can need very different infrastructure if one handles short interactive prompts at low concurrency and the other runs long contexts in large batches. Write down the following before comparing servers:
| Input | What to record | Why it changes sizing |
|---|---|---|
| Model identity | Exact model name, revision, architecture, parameter size, and serving format | Weights, layer structure, attention implementation, and runtime support affect memory and throughput. |
| Numerical format | FP16, BF16, INT8, or another quantization, including calibration details where applicable | Precision changes weight memory, arithmetic behavior, and sometimes quality or kernel availability. |
| Prompt and output shape | Distribution of input and generated tokens, not just an average | Long-tail requests can consume substantially more KV-cache memory and increase latency. |
| Context or sequence length | Maximum and typical total sequence length | Attention state and cache demand grow with active sequence length. |
| Traffic | Requests per second, peak rate, expected concurrency, and burst pattern | More active requests require more scheduling capacity and cache space. |
| Service objective | Time-to-first-token (TTFT), inter-token latency (ITL), end-to-end latency, and availability targets | A latency-bound interactive service cannot be sized like a throughput-oriented batch job. |
| Workload mode | Interactive, batch, or mixed; synchronous or asynchronous | Batching and queueing can improve utilization while changing response time. |
| Runtime policy | Serving engine, backend version, batching policy, cache policy, tensor or pipeline parallelism, and autoscaling trigger | Software settings determine process count, memory allocation, and scaling behavior. |
AWS guidance identifies model architecture and size, concurrent requests, acceptable latency, and sequence length as compute drivers. NVIDIA’s reference architecture likewise calls out model size, prompt and output shape, target concurrency, cache policy, GPU count, storage and network needs, and the autoscaling trigger. Treat these as required fields in a capacity plan rather than optional metadata.
Size GPU memory around weights, KV cache, and runtime headroom
Account for every memory consumer
For a language-model server, GPU memory must hold the model weights and the execution state needed to generate tokens. The active key-value (KV) cache stores attention data for in-flight requests. Runtime workspaces, framework allocations, communication buffers, and temporary tensors also need room. Leave operational headroom for allocator fragmentation and traffic variation; a configuration that barely loads the model is not a production configuration.
#1 Best Overall
- This product has been replaced by our latest generation. Please search for the SANDISK Optimus GX 7100 NVMe SSD
- HIGH-OCTANE GAMING. Experience speeds up to 7,250MB/s read and 6,900MB/s write (1-2TB models), with up to 35% faster performance than previous generation.
- PURPOSE-BUILT. Designed for serious on-the-go gamers, with a PCIe Gen4 interface and SANDISK’s next generation TLC 3D NAND.
- MORE TIME TO CLEAR THAT CHECKPOINT. Built with laptops and handheld gaming devices in mind, with up to 100% more power efficiency over the previous generation.
- DO MORE WITH DASHBOARD. Ensure your drive is optimized for prime performance with the downloadable WD_BLACK Dashboard (Windows only).
KV-cache demand is workload-dependent. It rises with the number of active sequences and their lengths, and it is affected by the model architecture, precision, cache format, and serving-engine settings. A short benchmark at one request does not establish capacity for a multi-user service.
Use published figures only as starting points
| Published figure | What it actually describes | How to use it |
|---|---|---|
| Approximately 2 GB of KV cache | NVIDIA’s 2023 inference-optimization example for Llama 2 7B in 16-bit precision at batch size 1, under that article’s stated sequence assumptions | Use as an illustration of why cache matters, not as a typical or universal cache requirement. |
| Around 24 GB of GPU memory | NVIDIA’s 2026 Technical Blog example for a smaller 7–8B model under particular prompt-length and multi-user assumptions | An initial candidate for testing that scenario; verify the actual model, precision, context, concurrency, and runtime. |
| 48 GB of GPU memory | The same 2026 discussion’s example for a 13B model with room for KV cache at its discussed prompt lengths | A conditional recommendation, not a minimum for every 13B deployment. |
These figures are not population averages or cross-vendor guarantees. A different revision, quantization, context limit, batching policy, or concurrency target can move the requirement substantially.
Estimate, then prove the estimate with a memory test
- Load the exact production artifact and precision in the intended serving engine.
- Reserve the runtime’s normal GPU-memory fraction for cache and workspaces rather than allocating every byte to weights.
- Drive requests that match the measured prompt-length, output-length, and concurrency distributions.
- Increase concurrency and sequence length until the service reaches the latency objective, an out-of-memory condition, or an unacceptable error rate.
- Record peak allocated and reserved memory, KV-cache utilization, queueing, and the point at which latency begins to violate the objective.
If the model fits but the cache does not, adding capacity to the cache, reducing maximum sequence length, using a supported quantization, or distributing the model with tensor parallelism may help. Each option changes latency, quality, or operational complexity and therefore requires another benchmark.
Do not equate a GPU count with capacity
Two GPUs can provide more aggregate memory yet perform worse than one GPU if inter-GPU communication, parallelism overhead, or CPU scheduling becomes the bottleneck. For every multi-GPU design, document the topology and interconnect, how layers or tensors are partitioned, and the largest request that must fit on each device. Measure single-request latency as well as sustained throughput; a high aggregate token rate can hide an unacceptable tail latency.
Rank #2
- Ideal for high speed, low power storage
- Gen 4x4 NVMe PCle performance
- Up to 6,000MB/s read, 4,000MB/s write
- Includes Acronis cloning software
- 5-year limited warranty
Size CPU capacity for the serving path
Understand what the CPU still does
The GPU performs model arithmetic, but the host commonly handles network connections, authentication or request validation, tokenization and detokenization, input preparation, queueing, batching decisions, monitoring, and process coordination. A CPU-starved host can leave an expensive GPU idle or introduce queueing before work reaches it.
Use the serving engine’s process model
vLLM documents at least one API-server process, one engine-core process, and one GPU-worker process per GPU: at least 2 + N processes for a deployment using N GPUs. Its documentation warns that having fewer physical CPU cores than these processes can create contention and degrade throughput and latency. This is a vLLM-specific floor, not a universal “cores per GPU” rule for every engine.
Plan additional CPU capacity for tokenization, request bursts, data pre- and post-processing, telemetry, and any co-located services. Pinning or reserving cores may be useful when predictable latency matters, but the correct allocation depends on the runtime and traffic pattern.
Benchmark CPU behavior under GPU load
- Measure physical-core utilization, run queue length, context switches, and scheduler time while the GPU is busy.
- Separate API, tokenizer, scheduler, and worker processes in your measurements.
- Check whether one core is saturated while aggregate utilization looks moderate; a single-threaded stage can throttle the pipeline.
- Compare CPU starvation with GPU utilization. Low GPU utilization plus a growing request queue often points to host-side work, synchronization, or input preparation.
- Repeat the test at peak request rate, not only at a convenient average.
When CPU contention is visible, add or reserve physical cores, move ancillary services, reduce per-request preprocessing, or change the batching and scheduling policy. Validate the change with the same request distribution and latency limits.
Rank #3
- PCIe 4.0 Performance: Delivers up to 7,100 MB/s read and 6,000 MB/s write speeds for quicker game load times, bootups, and smooth multitasking
- Spacious 1TB SSD: Provides space for AAA games, apps, and media with standard Gen4 NVMe performance for casual gamers and home users
- Broad Compatibility: Works seamlessly with laptops, desktops, and select gaming consoles including ROG Ally X, Lenovo Legion Go, and AYANEO Kun. Also backward compatible with PCIe Gen3 systems for flexible upgrades
- Better Productivity: Up to 2x faster than previous Gen3 generation. Improve performance for real world tasks like booting Windows, starting applications like Adobe Photoshop and Illustrator, and working in applications like Microsoft Excel and PowerPoint
- Trusted Micron Quality: Built with advanced G8 NAND and thermal control for reliable Gen4 performance trusted by gamers and home users
Size host memory for the platform and co-located work
NVIDIA’s Certified Systems Configuration Guide specifies system memory of at least twice the total GPU memory for the inference-server configurations it covers, distributed across CPU sockets and memory channels. For example, a host with 2 GPUs totaling 96 GB would start from that guide’s 192 GB system-memory rule for the covered configuration. That figure is a platform recommendation, not a law for every model server.
Your actual requirement can be higher or lower depending on model loading, CPU-side caches, pinned transfer buffers, framework behavior, offload, container overhead, telemetry, and other services on the host. Account for the operating system and orchestration agents before assigning memory to the inference process.
Check topology, not just capacity
- Populate memory channels according to the server manufacturer’s balanced configuration.
- On multi-socket systems, verify that the CPU and memory serving a GPU are on the intended NUMA node.
- Measure host-memory bandwidth and cross-socket traffic when preprocessing or offload is heavy.
- Watch swap activity; swapping during inference is a capacity failure, not a performance feature.
- Reserve enough free memory for model reloads and rolling updates if the deployment must keep serving during a replacement.
Size storage for artifacts, caches, and recovery time
Separate capacity from performance
Storage holds model artifacts, tokenizer files, multiple revisions, container layers, compilation caches, logs, crash dumps, and rollback copies. NVIDIA’s covered server configurations specify a 1 TB minimum local storage capacity and recommend one local NVMe drive per CPU socket. Those values apply to the configurations in that guide; they are not a universal requirement for every deployment.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Capacity planning should add the actual retained artifacts and operational copies. A single model may exist as an original download, a converted or quantized version, a compiled engine, and a previous version for rollback. Log retention and cache eviction policy can dominate disk usage in a long-running service.
Rank #4
- HUGE SPEED BOOST: Get random read/write speeds that are 40%/55% faster than 980 PRO; Experience up to 1400K/1550K IOPS, while sequential read/write speeds up to 7,450/6,900 MB/s reach near the max performance of PCIe 4.0*
- BREAKTHROUGH POWER EFFICIENCY: Use less power and get more performance; Enjoy up to 50% improved performance per watt over 980 PRO, plus optimal power efficiency with max PCIe 4.0 performance**
- SMART THERMAL CONTROL: Samsung's own nickel-coated controller delivers effective thermal control; With its slim size, 990 PRO is a perfect fit for desktops and laptops that meet the PCI-SIG D8 standard***
- THE CHAMPION MAKER: Up to 65% improvement in random performance enables faster loads for an ultimate gaming experience on PS5 and DirectStorage PC games****
- SAMSUNG MAGICIAN SOFTWARE: Get the most out of your SSD with Samsung Magician's advanced yet intuitive optimization tools; Monitor drive health, protect valuable data, and receive important updates for your 990 PRO
Measure the storage path that affects readiness
- Measure download or image-pull time for a cold host.
- Measure model load time from local storage and from the shared storage path used in production.
- Measure cache-hit and cache-miss times for compilation and tokenizer artifacts.
- Measure disk-to-GPU transfer time during startup and reload.
- Record time to first ready state during a normal deployment and during a node replacement.
Storage can shorten startup and improve recovery, but it does not replace GPU memory for active weights or KV cache. If the execution memory tiers cannot hold the model and working state, a larger SSD alone will not make the service fit.
Build a benchmark that answers the buying question
Hold comparison conditions constant
When comparing candidate systems, keep the model artifact revision, serving backend and version, hardware software stack, network mode, cache state, request distribution, concurrency, and comparison criteria identical. NVIDIA notes that omitting details such as backend version, model revision, network mode, or cache state is insufficient for an architectural conclusion.
| Dimension | Record for every run |
|---|---|
| Software | Driver, runtime, serving engine, backend, tokenizer, container, and model revision |
| Hardware | GPU model and count, memory capacity, CPU model and allocated cores, host memory, NUMA placement, storage, and interconnect topology |
| Traffic | Prompt and output distributions, sequence limits, request-rate pattern, concurrency, batching settings, and cache state |
| Results | TTFT, ITL, end-to-end latency percentiles, output tokens per second, completed requests, errors, queue time, and resource utilization |
Use a test matrix instead of one headline number
- Warm the service until model loading and compilation are complete; run a separate cold-start test for readiness planning.
- Test low, target, and peak concurrency using representative request lengths.
- Include the longest supported context and the high-percentile prompt and output sizes.
- Test the batching or continuous-batching settings intended for production.
- Repeat each point long enough to observe steady-state behavior and tail latency, not just a short burst.
- Record the maximum concurrency that meets the stated latency and error objectives.
- Repeat after changing GPU count, precision, cache allocation, CPU allocation, or parallelism; change one major variable at a time.
Report TTFT separately from ITL. A system can begin generating quickly but produce tokens slowly, or wait in a queue before producing at a high rate. End-to-end percentiles show the user experience, while output throughput and concurrency show capacity. Error rate and out-of-memory events determine whether a seemingly fast point is deployable.
Recommended Free Tools
Tune for the service objective
Interactive inference
Interactive services usually prioritize TTFT, ITL, and tail latency. Queueing too many requests or using an oversized batch may improve aggregate utilization while making individual responses unacceptable. Set an explicit concurrency limit, protect the KV cache from runaway contexts, and autoscale before the latency objective is breached.
Best Value
- SPEED UP PROJECTS. Launch creator applications fast with uncompromising PCIe 4.0 read speeds up to 7,100MB/s,[2] (1TB and 2TB[1] models) and write speeds up to 6,700MB/s[2] (1TB[1]-4TB[1] models).
- CREATE AND STORE MORE. Make more room for your 4K videos and high-resolution images with capacities from 500GB[1] up to 4TB[1] on M.2 2280 built with our trusted 8th generation SANDISK BiCS QLC 3D CBA NAND.
- IT GOES WHERE YOU GO. With an all-new power efficient design, your drive delivers high performance with low power, giving you more time to be productive while on the go.
- UNCOMPROMISED RELIABILITY. With up to 1,200 TBW[3] (4TB[1] model) endurance rating, your drive is designed for creators.
- KEEP YOUR DRIVE UPDATED. Monitor your SSD’s performance and check for updates with the downloadable SANDISK Dashboard application.[5]
Batch inference
Batch jobs can prioritize completed tokens per second and total job time. Larger batches often improve GPU utilization, but they consume more memory and can increase per-request latency. Find the largest batch that fits the memory budget and does not cause unacceptable failures or retry storms.
Mixed traffic
Interactive and batch requests competing for one cache and scheduler can interfere with each other. Consider separate pools or admission controls, then benchmark the mixture rather than adding the isolated results. Autoscaling triggers should reflect queue depth, latency, cache utilization, or another signal tied to the service objective—not GPU utilization alone.
Address memory pressure deliberately
vLLM identifies increasing GPU memory available to the KV cache and using tensor parallelism as ways to address memory pressure, while also warning that CPU starvation affects scheduling. These are tuning choices, not automatic fixes. Increasing cache allocation may reduce the number of model replicas that fit; tensor parallelism can add communication overhead. Measure the resulting latency, throughput, and failure behavior.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Compare candidate systems with a decision table
Use a table such as the following for each tested configuration. Do not substitute vendor specifications for measured values.
| Criterion | Candidate A | Candidate B | Acceptance rule |
|---|---|---|---|
| GPU memory and measured headroom | Record capacity and peak use | Record capacity and peak use | Fits weights, cache, runtime overhead, and traffic headroom. |
| Maximum compliant concurrency | Measured value | Measured value | Highest concurrency meeting the latency and error targets. |
| TTFT, ITL, and request latency | Percentiles from the same traffic | Percentiles from the same traffic | Meets the service objective, including tail latency. |
| Output throughput | Measured tokens per second | Measured tokens per second | Useful only when paired with compliant latency and error rate. |
| CPU behavior | Utilization, queueing, process contention | Utilization, queueing, process contention | No persistent CPU starvation at target load. |
| Host memory and topology | Capacity, bandwidth, NUMA placement | Capacity, bandwidth, NUMA placement | Balanced configuration with reload and co-service headroom. |
| Startup and recovery | Load, cache-hit, and ready times | Load, cache-hit, and ready times | Meets deployment and recovery objectives. |
| Interconnect and network | Topology and measured transfer behavior | Topology and measured transfer behavior | No communication bottleneck at the chosen parallelism. |
| Cost per useful output | Calculate only from reliable price and workload data | Calculate only from reliable price and workload data | Compare at the same compliant service point. |
The available guidance does not establish a universal cross-vendor benchmark or current cost comparison. If pricing, power, or hosted capacity is part of the decision, use current quotations and your measured workload rather than extrapolating from a single throughput figure.
Common sizing mistakes and recovery steps
| Symptom | Likely cause | Recovery |
|---|---|---|
| Model loads, then fails as concurrency rises | KV cache or runtime headroom was omitted | Measure peak cache use, reduce context or concurrency, change precision where acceptable, or add memory and retest. |
| GPU utilization is low while requests queue | CPU tokenization, API handling, scheduling, or synchronization is limiting the path | Profile host processes, reserve physical cores, reduce preprocessing overhead, and retest at target rate. |
| Average latency passes but tail latency fails | Bursts, long-tail contexts, CPU contention, or excessive batching | Test percentiles with the real distribution, cap admission, tune batching, and isolate traffic classes. |
| Multi-GPU version is slower than a single GPU | Parallelism or interconnect overhead dominates | Check topology and per-device memory, compare alternative partitioning, and use measured end-to-end latency. |
| Deployments take too long | Cold downloads, compilation, cache misses, or slow disk-to-GPU transfer | Measure each startup stage, retain local artifacts where appropriate, and size NVMe capacity and endurance for the retention policy. |
| Performance changes between benchmark runs | Different model revision, backend, cache state, network mode, or traffic mix | Record and lock the full environment before drawing an architectural conclusion. |
| Swap or host-memory pressure appears | Insufficient system memory, unbalanced NUMA placement, offload, or co-located services | Remove swap dependence, rebalance channels and sockets, reserve memory, and repeat the workload test. |
A repeatable sizing checklist
- Exact model revision, architecture, precision, and quantization are recorded.
- Prompt, output, context, concurrency, request-rate, and burst distributions are measured.
- TTFT, ITL, end-to-end latency percentiles, throughput, and error objectives are explicit.
- GPU memory includes weights, KV cache, runtime overhead, and operational headroom.
- CPU allocation covers the serving engine’s processes plus tokenization, scheduling, and co-located work.
- Host memory is balanced across sockets and channels, with room for loading, reloads, caches, and the operating system.
- Storage capacity includes artifacts, versions, caches, logs, and rollback copies; startup and transfer times are measured.
- GPU, CPU, memory, storage, NUMA, network, and interconnect topology are documented.
- Benchmark conditions are identical across candidates and include cold-start and steady-state tests.
- The selected configuration is the one that meets the service objective at the required concurrency, not the one with the highest isolated throughput.
Re-run the test when the model revision, precision, context limit, serving engine, batching policy, traffic mix, or hardware changes. Inference capacity is a property of that complete combination, not of a GPU model or parameter count in isolation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches
