iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Sometimes—but not automatically. A faster SSD or NVMe drive can shorten model loading, improve startup readiness, and help workloads that actively move data through storage, such as some KV-cache offload designs. It usually will not increase token-generation speed when the model and cache already fit in accelerator memory and the bottleneck is compute, accelerator memory bandwidth, or another serving component.
The reliable way to decide is to benchmark a production-like workload. Separate model-readiness timings from steady-state generation, test cold and warm caches, and keep the model, backend, hardware, network, software, and request mix unchanged between storage configurations.
What “inference performance” includes
Inference is not one timing. A user may experience several phases, and storage can affect some phases while leaving others unchanged.
Startup and model readiness
Before the first request can be served, an inference system may download model files, read them from a local cache, transfer weights into accelerator memory, initialize the runtime, and report that the model is ready. NVIDIA’s inference architecture guidance recommends recording separate timestamps for model download, cache-hit time, weight transfer, first-ready time, and first-token behavior. A faster storage path matters when one of those reads or transfers is on the critical path.
#1 Best Overall
- MEET THE NEXT GEN: Consider this a cheat code; Our Samsung 990 PRO Gen4 SSD helps you reach near max performance with lightning-fast speeds; Whether you’re a hardcore gamer or a tech guru, you’ll get power efficiency built for the final boss
- REACH THE NEXT LEVEL: Gen4 steps up with faster transfer speeds and high-performance bandwidth; With a more than 55% improvement in random performance compared to 980 PRO, it’s here for heavy computing and faster loading
- THE FASTEST SSD FROM THE WORLD'S FLASH MEMORY BRAND: The speed you need for any occasion; With read and write speeds up to 7450/6900 MB/s you’ll reach near max performance of PCIe 4.0 powering through for any use
- PLAY WITHOUT LIMITS: Give yourself some space with storage capacities from 1TB to 4TB; Sync all your saves and reign supreme in gaming, video editing, data analysis and more
- IT’S A POWER MOVE: Save the power for your performance; Get power efficiency all while experiencing up to 50% improved performance per watt over the 980 PRO; It makes every move more effective with less consumption
Request-time generation
Once weights are resident and requests are running, the important measures are time to first token (TTFT), inter-token latency, end-to-end request latency, and throughput. Many deployments generate tokens from accelerator memory without reading model weights from an SSD for each token. In that case, changing the drive can leave generation unchanged even though startup becomes faster.
Cache-related work
Some serving systems use storage as one tier in a larger memory hierarchy. Prefix caches, model caches, and KV caches can be placed in GPU memory, CPU RAM, local disk, or remote storage. If requests repeatedly fetch data from a lower tier, storage latency and bandwidth become part of the request path. If the cache is rarely reused, the extra tier may add complexity without improving the observed result.
When a faster SSD can help
Large model loads and reloads
Model files can be too large for a fast local cache or may be loaded frequently because workers restart, scale out, or switch models. A faster local drive can reduce the disk-read portion of startup. The improvement is visible in first-ready time, not necessarily in the latency of a warm request after the model is already resident.
Cloud systems add another dependency: the path from object storage or a container image to the local disk. Google Cloud’s Cloud Run GPU guidance describes SSD caching as a way to reduce model-loading latency and notes that network throughput can bottleneck the build or load path. That is a cloud-service-specific behavior; the same result should not be assumed on another platform with different image, network, or cache mechanics.
Rank #2
- Ideal for high speed, low power storage
- Gen 4x4 NVMe PCle performance
- Up to 6,000MB/s read, 4,000MB/s write
- Includes Acronis cloning software
- 5-year limited warranty
Disk-backed or tiered KV-cache designs
During long-context or multi-user serving, the KV cache can consume substantial GPU VRAM. Some engines offload cache entries to CPU RAM or NVMe storage and bring them back when needed. MLCommons documents a KV-cache workload that simulates this offloading behavior. In such an architecture, a lower-latency, higher-throughput NVMe device can reduce time spent moving cache blocks compared with a slower disk.
The result depends on the engine’s implementation, block size, transfer scheduling, cache-hit rate, concurrent users, and whether transfers overlap with accelerator work. The existence of an NVMe tier is therefore a reason to measure, not proof of a fixed gain.
Local cache misses during scale-out
When a new worker starts, it may fetch a model or tokenizer that another worker has already cached elsewhere. Faster local storage can reduce the time between worker launch and usable capacity, particularly when network delivery is fast enough that the disk is the limiting stage. If the network is slower, upgrading the drive may simply expose the network as the next bottleneck.
Free tools Windows power users keep installed
One-click scans. No signup required.
When storage speed will not change token generation
An SSD upgrade is unlikely to alter steady-state output when all of the following are true:
Rank #3
- SPEED UP PROJECTS. Launch creator applications fast with uncompromising PCIe 4.0 read speeds up to 7,100MB/s,[2] (1TB and 2TB[1] models) and write speeds up to 6,700MB/s[2] (1TB[1]-4TB[1] models).
- CREATE AND STORE MORE. Make more room for your 4K videos and high-resolution images with capacities from 500GB[1] up to 4TB[1] on M.2 2280 built with our trusted 8th generation SANDISK BiCS QLC 3D CBA NAND.
- IT GOES WHERE YOU GO. With an all-new power efficient design, your drive delivers high performance with low power, giving you more time to be productive while on the go.
- UNCOMPROMISED RELIABILITY. With up to 1,200 TBW[3] (4TB[1] model) endurance rating, your drive is designed for creators.
- KEEP YOUR DRIVE UPDATED. Monitor your SSD’s performance and check for updates with the downloadable SANDISK Dashboard application.[5]
- Model weights and frequently used cache data remain in GPU memory or CPU RAM.
- The accelerator is limited by arithmetic throughput or its own memory bandwidth.
- No request-time path reads model data or KV-cache blocks from disk.
- Requests are short enough that cache capacity and eviction are not problems.
In that situation, the storage device is outside the critical path after startup. A faster drive may still improve restart time or operational recovery, but it should not be credited with a token-rate increase that the serving path does not use.
Storage tiers in an inference system
| Tier | Typical role | What to measure | Main limitation |
|---|---|---|---|
| GPU VRAM | Active weights, attention state, and hot KV-cache blocks | Available capacity, memory bandwidth, evictions | Limited capacity and high cost per gigabyte |
| CPU RAM | Expanded cache and staging area for transfers | Copy time, saturation, cache-hit rate | Slower access than VRAM and shared with other processes |
| Local NVMe SSD | Model cache, spill tier, or supported KV-cache offload | Read latency, sustained throughput, queue depth, transfer time | Still much slower than RAM; requires engine support |
| Remote or network storage | Shared model repository or remote cache | Network latency, throughput, retry behavior, locality | Variable performance and dependence on the network path |
The table describes roles rather than guaranteed performance. A system can use several tiers simultaneously, and the fastest device may not be the tier that serves the most requests.
Why capacity can trade against latency
Adding CPU, disk, or remote storage can extend cache capacity beyond GPU memory. That can allow more prefixes or KV blocks to remain available, potentially helping throughput when reuse is high. However, every miss that must be fetched from a slower tier incurs transfer latency. AWS guidance cautions that when prefix reuse is low, the latency cost of a larger but slower cache can outweigh throughput gains.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Evaluate the reuse pattern of your own traffic rather than assuming that a larger cache is beneficial. A chatbot with repeated system prompts may have high prefix reuse; a workload composed mostly of unrelated prompts may not. Record cache hits, misses, evictions, and the tier that supplied each block.
Rank #4
- HUGE SPEED BOOST: Get random read/write speeds that are 40%/55% faster than 980 PRO; Experience up to 1400K/1550K IOPS, while sequential read/write speeds up to 7,450/6,900 MB/s reach near the max performance of PCIe 4.0*
- BREAKTHROUGH POWER EFFICIENCY: Use less power and get more performance; Enjoy up to 50% improved performance per watt over 980 PRO, plus optimal power efficiency with max PCIe 4.0 performance**
- SMART THERMAL CONTROL: Samsung's own nickel-coated controller delivers effective thermal control; With its slim size, 990 PRO is a perfect fit for desktops and laptops that meet the PCI-SIG D8 standard***
- THE CHAMPION MAKER: Up to 65% improvement in random performance enables faster loads for an ultimate gaming experience on PS5 and DirectStorage PC games****
- SAMSUNG MAGICIAN SOFTWARE: Get the most out of your SSD with Samsung Magician's advanced yet intuitive optimization tools; Monitor drive health, protect valuable data, and receive important updates for your 990 PRO
How to benchmark storage impact correctly
Use a controlled comparison rather than a synthetic drive-speed test. The following procedure isolates storage effects while preserving the behavior of the serving system.
- Choose a representative workload. Use the model, tokenizer, backend, quantization, prompt lengths, output lengths, request mix, and concurrency that reflect production. AWS identifies model size and architecture, concurrent requests, acceptable latency, and sequence length as key inference-sizing factors.
- Document the system. Record accelerator model and count, CPU and RAM, storage device and filesystem, storage connection, network mode, operating-system and driver versions, inference-server version, container image, and model revision.
- Define cache states. Run at least one cold-cache trial and one warm-cache trial. A cold run should start without the model in the local cache; a warm run should use the normal cache-hit path. Do not mix the two when calculating averages.
- Measure startup separately. Capture model-download start and end, local read time, weight-transfer time, first-ready time, and the time of the first token. This shows whether the drive affects readiness or only a later phase.
- Measure request behavior. Record TTFT, inter-token latency, complete request latency, output-token rate, aggregate throughput, queue time, and errors at each target concurrency.
- Test cache-offload paths explicitly. If the engine supports KV-cache offload, identify whether blocks use CPU RAM, local NVMe, or remote storage. Record cache-hit and reuse rates, block transfers, evictions, and concurrent load. A benchmark that never triggers offload cannot demonstrate an offload benefit.
- Repeat enough trials. Use the same prompts and scheduling policy for each storage configuration, discard startup outliers only with a stated rule, and report distributions such as median and high-percentile latency rather than a single best run.
- Change only storage. Keep software, model, accelerator clocks, network path, cache policy, and concurrency fixed. If another variable changes, attribute the result to the complete configuration rather than to the drive alone.
Metrics that reveal the bottleneck
| Metric | Storage-sensitive symptom | Interpretation |
|---|---|---|
| Model download and local-load time | Falls on the faster device | Storage or its delivery path is limiting startup |
| First-ready time | Falls while generation metrics stay flat | Operational readiness improved, not token generation |
| TTFT | Improves only on cold runs or cache misses | Request startup or cache fetch is involved |
| Inter-token latency | Improves during active offload | Cache transfers may be on the generation path |
| Throughput at concurrency | Rises while latency remains acceptable | Additional cache capacity or fewer reloads may be helping |
| Error rate and tail latency | Worsen as storage queue depth grows | The device or path is saturated under load |
A drive’s advertised sequential bandwidth is not a substitute for these measurements. Small random reads, queueing, filesystem behavior, PCIe sharing, CPU copies, and network delays can dominate an inference workload.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to interpret benchmark results
Startup-only improvement
If first-ready time falls but TTFT, inter-token latency, and throughput are unchanged on warm requests, the purchase is a deployment-speed improvement. It may be worthwhile for autoscaling, frequent restarts, disaster recovery, or developer iteration, but it should be described as a readiness benefit.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Improvement only under offload
If gains appear when the KV cache spills to NVMe and disappear when the cache fits in RAM, storage is helping a specific memory-pressure scenario. Report the model, context length, concurrency, offload tier, and reuse rate with the result; those conditions determine whether another deployment will see the same effect.
Best Value
- This product has been replaced by our latest generation. Please search for the SANDISK Optimus GX 7100 NVMe SSD
- HIGH-OCTANE GAMING. Experience speeds up to 7,250MB/s read and 6,900MB/s write (1-2TB models), with up to 35% faster performance than previous generation.
- PURPOSE-BUILT. Designed for serious on-the-go gamers, with a PCIe Gen4 interface and SANDISK’s next generation TLC 3D NAND.
- MORE TIME TO CLEAR THAT CHECKPOINT. Built with laptops and handheld gaming devices in mind, with up to 100% more power efficiency over the previous generation.
- DO MORE WITH DASHBOARD. Ensure your drive is optimized for prime performance with the downloadable WD_BLACK Dashboard (Windows only).
No measurable difference
If all timings remain within normal run-to-run variation, investigate accelerator utilization, memory bandwidth, CPU preprocessing, scheduler queues, network delivery, and kernel configuration before buying faster storage. The drive may already be faster than the workload requires.
Practical guidance for local and cloud deployments
Local model storage
Use an NVMe SSD when model files are large, reloads are frequent, or the inference engine documents a local-disk cache or offload tier. Place the active model cache on the fastest available path, verify filesystem and mount options, and leave enough free capacity to avoid eviction churn. Do not expect a particular consumer drive to raise token throughput without a benchmark showing that disk reads occur during requests.
Cloud instances
Check whether the instance’s local SSD is ephemeral, whether it survives restart, and whether the model is pulled over a network on every replacement. Measure object-storage or image download, network throughput, local write, and local read separately. A faster disk cannot compensate for a throttled network path.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsShared infrastructure
For multi-tenant systems, monitor queue depth, per-worker cache locality, and noisy-neighbor effects. A shared NVMe device can deliver lower performance under concurrent model loads than an isolated benchmark suggests. Capacity planning should include the number of models, replicas, cache policies, and expected restart rate.
Common mistakes and fixes
- Comparing drive specifications instead of serving metrics: run the complete inference benchmark and include tail latency.
- Mixing cold and warm runs: label cache state and analyze each state independently.
- Changing several variables at once: hold model, backend, hardware, network, and software versions constant.
- Testing a cache tier the engine never uses: verify that KV-cache or model-cache offload is enabled and triggered.
- Ignoring concurrency: test the request levels that create realistic queueing and cache pressure.
- Claiming a universal percentage: publish results only for the measured model, engine, hardware, and workload. No general controlled comparison establishes one uplift figure for all inference systems.
Decision checklist
- Is model loading or cache population a significant part of your user-visible latency?
- Do workers restart, autoscale, or switch models often enough for readiness time to matter?
- Does the serving engine support and actually trigger SSD-backed cache offload?
- Are cache reuse and concurrency high enough to offset slower-tier transfer latency?
- Have you measured cold and warm behavior, TTFT, inter-token latency, full latency, throughput, and errors?
- Have you ruled out accelerator, CPU, memory-bandwidth, scheduler, and network bottlenecks?
If the answers point to a storage-bound phase, compare the current path with an NVMe SSD under identical conditions. If the workload is accelerator-bound with resident data, prioritize the component that the measurements identify instead.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
