Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A powerful model or accelerator can still run slowly when data cannot reach it quickly enough. Every training batch, checkpoint, inference request, model load and cache update travels through links with finite bandwidth and latency: storage, CPU memory, PCIe or another interconnect, GPU memory and the network. If any stage is slower than the compute stage, the accelerator waits and end-to-end performance falls.
The remedy is not automatically a faster GPU or SSD. First identify whether storage, host-to-GPU copies, preprocessing, memory capacity, communication, synchronization or network topology is limiting the workload. Then change only the path that measurements show is constraining it.
The direct answer: compute is only one stage of an AI system
AI performance is the speed of a pipeline, not the theoretical speed of its model or accelerator. A GPU may be capable of processing enormous amounts of arithmetic, yet remain partly idle while another component fetches the next batch, converts data, writes a checkpoint or exchanges tensors with another device.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsA typical training input can travel from a remote or local storage system into host memory, through a CPU-side data loader, across an interconnect and into GPU memory. Inference adds other paths: loading model artifacts, serving network requests, reading or writing key-value (KV) cache data and moving responses between devices or machines. These paths share buses, switches, memory channels, CPUs and network links. A transfer can therefore be limited by either the link itself or by contention with unrelated work.
#1 Best Overall
- Ideal for high speed, low power storage
- Gen 4x4 NVMe PCle performance
- Up to 6,000MB/s read, 4,000MB/s write
- Includes Acronis cloning software
- 5-year limited warranty
Two properties matter most:
- Bandwidth is how much data a path can carry per unit of time. Large datasets, model weights and checkpoints are bandwidth-heavy.
- Latency is the delay before a transfer or operation completes. Small batches, short requests, cache misses and synchronization are latency-sensitive.
High bandwidth does not eliminate latency, and low latency does not create unlimited throughput. A workload that issues many small reads can be slow on a device with impressive sequential bandwidth. Conversely, a large, well-buffered transfer may tolerate latency but saturate a shared link.
Where the data goes, and where it can wait
Storage to host memory
Training examples, tokenized records, image files and checkpoints may reside on local NVMe storage, a shared file system or a remote object store. Remote storage adds network hops and service-side variability. Shared storage can be fast in isolation but slow when many jobs compete for it. Local storage avoids those network hops, yet its controller, thermal state, queue depth and filesystem still determine sustained behavior.
Host memory to GPU memory
In the conventional loading path described in NVIDIA interoperability documentation, data is read into CPU host memory and then copied again into GPU memory. The CPU-side buffer is often called a bounce buffer. The extra copy consumes time, memory bandwidth and CPU resources; it can also require synchronization before the GPU can use the data.
On supported systems, a direct storage-to-GPU path can let a storage-side DMA engine place data in GPU memory without that CPU bounce buffer. NVIDIA’s GPUDirect Storage design material describes this as a way to reduce copies and potentially lower latency and CPU load. Availability and benefit depend on the GPU, storage device, PCIe topology, drivers, filesystem and software stack. A direct path is not a universal replacement for host memory.
Rank #2
- MEET THE NEXT GEN: Consider this a cheat code; Our Samsung 990 PRO Gen4 SSD helps you reach near max performance with lightning-fast speeds; Whether you’re a hardcore gamer or a tech guru, you’ll get power efficiency built for the final boss
- REACH THE NEXT LEVEL: Gen4 steps up with faster transfer speeds and high-performance bandwidth; With a more than 55% improvement in random performance compared to 980 PRO, it’s here for heavy computing and faster loading
- THE FASTEST SSD FROM THE WORLD'S FLASH MEMORY BRAND: The speed you need for any occasion; With read and write speeds up to 7450/6900 MB/s you’ll reach near max performance of PCIe 4.0 powering through for any use
- PLAY WITHOUT LIMITS: Give yourself some space with storage capacities from 1TB to 4TB; Sync all your saves and reign supreme in gaming, video editing, data analysis and more
- IT’S A POWER MOVE: Save the power for your performance; Get power efficiency all while experiencing up to 50% improved performance per watt over the 980 PRO; It makes every move more effective with less consumption
Interconnects and shared resources
PCIe links, switches, CPU sockets, accelerator interconnects and network adapters all have finite capacity. A GPU and a storage device may share a switch uplink, or a device may sit behind a different CPU socket from the GPU that consumes its data. Crossing sockets can introduce extra hops and contention. In a multi-GPU job, peer transfers and collective communication can compete with input and output traffic.
GPU memory and caches
GPU memory capacity affects movement even when its bandwidth is high. If a model, activations, batches or KV cache do not fit, the system must evict, reload, offload or recompute data. Those operations can turn a memory-capacity problem into a PCIe or network problem. Cache hits may keep data close to the accelerator; cache misses expose the slower backing store.
Why training stalls
Input fetching and preprocessing
Training alternates between preparing a batch and running the model step. Fetching files, decoding images, tokenizing text, decompressing archives, augmenting examples and collating variable-length records can all take longer than the GPU computation. The University of Texas at Austin dissertation Accelerating Deep Learning Training: A Storage Perspective (December 2021) treats fetching, preprocessing and checkpoint writing as data-pipeline bottlenecks rather than secondary housekeeping.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallIf the next batch is not ready when the current step finishes, the accelerator has nothing to execute. Increasing the number of data-loader workers may help a CPU-bound pipeline, but it can also increase contention for cores, memory and storage. More workers do not fix a saturated disk or a serialized preprocessing operation.
Rank #3
- BREAKTHROUGH PCIe 5.0 PERFORMANCE: Supercharge your workflow and gaming with PCIe 5.0, boasting up to 14,700/13,400 MB/s* sequential read/write speeds. Tackle massive files and power up your gaming with Gen5—twice as fast as the 990 PRO SSD.
- EVERY TASK, TURBOCHARGED: Speed past productivity limits. With random read/write speeds up to 1,850K/2,600K IOPS*, enjoy fast game loads, seamless AI apps, and efficient multitasking. Virtually no lag, no limits—just nonstop performance.
- THINK FAST, CREATE FASTER: With random read/write speeds of up to 1,850K/2,600K IOPS*, the 9100 PRO SSD fuels seamless AI content creation, swift loads, and smooth gameplay. Work, play, and create at lightning speed.
- SPEED, WHENEVER YOU NEED: From laptops to desktop PCs, experience blazing PCIe 5.0 speeds and up to 8TB of storage. Perfect for video editing, gaming, and creative tasks, with the compatibility to match your device.
- STAY COOL, RUN FAST: Push limits, not temperatures. A 5nm controller boosts power efficiency up to 49% over the 990 PRO SSD*, while advanced thermal control keeps performance smooth and reliable.
Checkpoint writes
Checkpointing can pause or slow training when model state, optimizer state and metadata must be written to shared storage. Large checkpoints create sustained output traffic; remote destinations add network latency and service contention. Checkpoint frequency is therefore a performance trade-off: more frequent recovery points reduce potential lost work but consume more I/O time.
Distributed communication
Data movement between GPUs or machines can dominate a step when models, batches or gradients are partitioned. Synchronization forces fast devices to wait for the slowest participant. A job may appear compute-bound on one GPU while the group is actually waiting on a congested link, an imbalanced shard or a straggling host.
Pipeline overlap and imbalance
Efficient systems overlap data transfer with computation, but overlap is limited by buffer space, ordering dependencies and resource contention. If preprocessing, transfer and compute stages have unequal rates, queues either empty (starving the GPU) or grow (consuming memory and increasing latency). A faster individual stage does not improve throughput when another stage remains slower.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Inference has different movement patterns
Cold start and model loading
Before serving the first request, a system may read model artifacts from storage, construct runtime state and copy weights into GPU memory. Large models make this startup path visible during deployment, recovery and autoscaling. A fast steady-state GPU does not shorten a cold start if storage or host-to-GPU transfer is the limiting stage.
Rank #4
- HUGE SPEED BOOST: Get random read/write speeds that are 40%/55% faster than 980 PRO; Experience up to 1400K/1550K IOPS, while sequential read/write speeds up to 7,450/6,900 MB/s reach near the max performance of PCIe 4.0*
- BREAKTHROUGH POWER EFFICIENCY: Use less power and get more performance; Enjoy up to 50% improved performance per watt over 980 PRO, plus optimal power efficiency with max PCIe 4.0 performance**
- SMART THERMAL CONTROL: Samsung's own nickel-coated controller delivers effective thermal control; With its slim size, 990 PRO is a perfect fit for desktops and laptops that meet the PCI-SIG D8 standard***
- THE CHAMPION MAKER: Up to 65% improvement in random performance enables faster loads for an ultimate gaming experience on PS5 and DirectStorage PC games****
- SAMSUNG MAGICIAN SOFTWARE: Get the most out of your SSD with Samsung Magician's advanced yet intuitive optimization tools; Monitor drive health, protect valuable data, and receive important updates for your 990 PRO
Per-request data and KV cache
During autoregressive generation, prompts, intermediate states and KV cache entries move through GPU memory and, in some designs, between memory tiers or machines. Longer sequences and larger batches increase cache traffic and capacity pressure. Evicting cache entries can trigger reloads; cache misses increase latency even when the model’s arithmetic rate is unchanged.
Network and topology
Inference requests and responses traverse network interfaces, load balancers and service-to-service links. A topology that places a client, model shard, cache and accelerator on distant or congested paths adds latency. NVIDIA’s inference reference architecture diagrams treat storage, cache, network and placement as connected parts of the data flow, not independent components.
Throughput and latency can also pull in opposite directions. Batching improves device utilization and amortizes transfer overhead, but waiting to form a batch increases individual-request latency. The best setting changes with request size, sequence length, cache hit rate and traffic pattern.
How to find the actual bottleneck
Measure the pipeline while the workload runs. A single GPU-utilization number is not enough: utilization can be low because of input starvation, synchronization, memory pressure or short kernels.
Best Value
- SPEED UP PROJECTS. Launch creator applications fast with uncompromising PCIe 4.0 read speeds up to 7,100MB/s,[2] (1TB and 2TB[1] models) and write speeds up to 6,700MB/s[2] (1TB[1]-4TB[1] models).
- CREATE AND STORE MORE. Make more room for your 4K videos and high-resolution images with capacities from 500GB[1] up to 4TB[1] on M.2 2280 built with our trusted 8th generation SANDISK BiCS QLC 3D CBA NAND.
- IT GOES WHERE YOU GO. With an all-new power efficient design, your drive delivers high performance with low power, giving you more time to be productive while on the go.
- UNCOMPROMISED RELIABILITY. With up to 1,200 TBW[3] (4TB[1] model) endurance rating, your drive is designed for creators.
- KEEP YOUR DRIVE UPDATED. Monitor your SSD’s performance and check for updates with the downloadable SANDISK Dashboard application.[5]
- Divide the workload into phases. Record separate timings for data reads, preprocessing, host-to-device copies, GPU kernels, synchronization, network transfers and checkpoint I/O. For inference, measure cold start, prompt processing, token generation, cache operations and response transfer separately.
- Check queue behavior. An empty input queue before a GPU step indicates that preparation or fetching is too slow. A persistently full queue with low GPU utilization points elsewhere, such as synchronization or memory contention.
- Measure sustained throughput and tail latency. Compare actual read and write rates with the storage and link capabilities under the same concurrency. Include variability and contention, not just a short peak result.
- Count copies and synchronization points. Trace whether data goes storage-to-host-to-GPU, whether it is copied more than once, and where the consumer must wait for completion.
- Inspect locality and topology. Identify the GPU, storage device, network adapter, PCIe switch and CPU socket handling each transfer. A nominally fast device can underperform when it shares a path or sits behind an unfavorable hop.
- Change one variable at a time. Test cache warm versus cold, different batch sizes, sequence lengths, worker counts or storage locations. A bottleneck that moves when the workload changes is a system property, not a fixed specification.
- Compare end-to-end time. A faster copy or disk benchmark matters only if total step time, request latency or useful throughput improves without creating a new bottleneck.
Interventions matched to measured constraints
| Measured constraint | Potential intervention | What must be verified |
|---|---|---|
| Repeated host-memory bounce copies | Supported direct storage-to-GPU I/O | GPU, storage, filesystem, driver and PCIe compatibility; actual copy and CPU time |
| Cold or repeated reads from slow storage | Local staging or a cache | Capacity, invalidation policy, warm-up cost, locality and concurrent jobs |
| CPU-bound decoding or tokenization | More efficient preprocessing, parallel workers or preprocessed records | CPU availability, memory bandwidth and whether storage then becomes limiting |
| Input queue starvation | Asynchronous loading and transfer overlap | Buffer memory, ordering requirements and whether overlap is actually occurring |
| PCIe, switch or network contention | Topology-aware placement or dedicated paths | Link ownership, socket locality, traffic from other jobs and failure behavior |
| Checkpoint pauses | Staggered, asynchronous or less frequent writes | Recovery requirements, durability and impact on shared storage |
| Distributed synchronization | Balanced sharding, communication tuning or placement changes | Stragglers, collective traffic, model partitioning and network tail latency |
When direct I/O is appropriate
Direct I/O is most compelling when profiling shows that host-mediated copies and CPU overhead are material parts of the critical path. It cannot compensate for a slow source, an undersized link, poor locality or a workload dominated by preprocessing. Unsupported combinations may fall back to a conventional path, so confirm the path actually used rather than assuming a feature is active.
When caching helps
Caching helps when the same data is reused and the cache is faster or closer to the consumer. It helps less when access is effectively one-pass, the cache is too small, or warming it costs as much as the original reads. Account for eviction, coherence, restart behavior and the storage consumed by cached model or dataset artifacts.
When topology changes matter
Moving a workload so that its storage, network adapter and GPU share a favorable PCIe and CPU path can reduce hops and contention. Topology changes add operational complexity and may reduce scheduling flexibility, so they should follow measurements that show locality is limiting performance.
What the published numbers do and do not prove
A technical blog from NVIDIA reports a local-NVMe DMA example at 13.3 GB/s, an improvement of about 10% relative to the CPU-to-GPU transfer rate in its stated setup, and 3.8× lower end-to-end latency for explicit direct-transfer examples. These are observations from the configurations described in that blog, not guarantees for every SSD, GPU, driver, filesystem or workload. They should be used to understand the possible scale of a benefit, not as a purchasing promise.
No representative cross-workload statistic establishes a universal percentage of AI performance lost to data movement. Results vary with model size, batch size, sequence length, cache behavior, storage location, parallelism and contention. Vendor documentation and research abstracts explain the mechanism, but they do not substitute for measurements on the system being tuned.
A practical decision sequence
- State the symptom precisely: low accelerator utilization, long step time, high first-request latency, poor tokens per second, checkpoint pauses or unstable tail latency.
- Map the data path: list every source, buffer, link, device and destination for that phase.
- Profile and establish a baseline: capture timings, throughput, queue depth, CPU use, memory pressure and network behavior under representative load.
- Apply the smallest targeted change: optimize preprocessing, stage data locally, enable a compatible direct path, adjust overlap or change placement according to the measured constraint.
- Re-profile end to end: retain the change only if useful throughput, latency or recovery time improves without unacceptable cost or reliability trade-offs.
Bottom line
Model capability sets an upper bound on computation, not on completed work. AI systems must continually move data through storage, host memory, interconnects, GPU memory and networks, and the slowest or most contended stage controls the result. Treat data movement as part of the model’s execution path, measure it by workload phase, and choose direct I/O, caching, overlap or topology changes only when they remove the measured wait.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
