iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Bandwidth (or achieved throughput), IOPS and latency describe different parts of storage performance. Bandwidth measures how much data a path can move per second; IOPS measures how many individual operations it completes per second; latency measures how long one operation takes. For AI storage, large sequential training transfers usually make sustained throughput the leading concern, while small random lookups, retrieval, metadata activity and concurrent agent sessions often depend more on IOPS and latency.
No single headline SSD number predicts application performance. The useful comparison is the one that matches your workload’s read/write mix, request size, concurrency, queue depth and measurement point, then confirms the result with an end-to-end application metric such as time per training epoch or inference throughput.
The three metrics in plain language
Storage specifications often place bandwidth, IOPS and latency next to one another, but they answer different questions. The distinction matters because an AI pipeline can contain several very different I/O patterns at the same time: loading a multi-gigabyte model, fetching many small metadata records, writing a checkpoint and serving concurrent requests are not equivalent tests.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Bandwidth: the data-rate ceiling
Bandwidth is a data rate, normally expressed in bytes per second. NVIDIA describes it as the theoretical maximum attainable data rate. A drive or storage fabric can therefore advertise a bandwidth ceiling that an application never reaches. The number may describe the device, the PCIe link or another part of the path rather than the complete client-to-storage transaction.
#1 Best Overall
- MEET THE NEXT GEN: Consider this a cheat code; Our Samsung 990 PRO Gen4 SSD helps you reach near max performance with lightning-fast speeds; Whether you’re a hardcore gamer or a tech guru, you’ll get power efficiency built for the final boss
- REACH THE NEXT LEVEL: Gen4 steps up with faster transfer speeds and high-performance bandwidth; With a more than 55% improvement in random performance compared to 980 PRO, it’s here for heavy computing and faster loading
- THE FASTEST SSD FROM THE WORLD'S FLASH MEMORY BRAND: The speed you need for any occasion; With read and write speeds up to 7450/6900 MB/s you’ll reach near max performance of PCIe 4.0 powering through for any use
- PLAY WITHOUT LIMITS: Give yourself some space with storage capacities from 1TB to 4TB; Sync all your saves and reign supreme in gaming, video editing, data analysis and more
- IT’S A POWER MOVE: Save the power for your performance; Get power efficiency all while experiencing up to 50% improved performance per watt over the 980 PRO; It makes every move more effective with less consumption
Use the word throughput when you mean an achieved rate, such as the sequential read rate measured during a test. In casual storage discussions, bandwidth and throughput are sometimes used interchangeably, so a useful report says whether the figure is theoretical or measured and identifies where it was measured. NVIDIA’s GPUDirect Storage Benchmarking and Configuration Guide provides this distinction and discusses the host and PCIe factors that can keep an NVMe result below its specification.
IOPS: operation count per second
IOPS means input/output operations per second: the number of read or write operations storage completes in one second. IOPS does not include the size of each operation. “One million IOPS” is therefore incomplete without the request size, read/write mix, access pattern, concurrency and queue depth used to produce it.
A test issuing 4 KiB random reads and a test issuing 1 MiB reads can both report IOPS, but they move very different amounts of data. Micron’s IOPS glossary explains why operation size and workload conditions affect achievable IOPS.
Latency: time for one operation
Latency is the time required to complete an operation. Always state the measurement point: a NAND device, an appliance, a filesystem client or the complete application path can each have a different latency. AWS defines EBS latency as the interval from sending a request until receiving acknowledgement that the operation is complete. That service-level definition is not automatically the same as a local drive’s controller latency or an application’s end-to-end response time.
Average latency can hide pauses that users notice. For interactive inference, retrieval and agentic systems, report a distribution such as median and high percentiles when the test provides them, rather than relying on a single average. The important question is how quickly the required operation returns under the concurrency the application actually creates.
How the metrics relate—and why they cannot substitute for one another
A useful approximation is:
data rate = operations per second × bytes per operation
For example, 100,000 operations per second at 16 KiB per operation corresponds to roughly 1.6 GB/s before accounting for protocol overhead, device limits, queueing and other bottlenecks. The multiplication is a way to sanity-check a result, not a promise that a system will sustain that rate.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Small requests can produce a high IOPS number while moving modest data. Conversely, a large sequential request can deliver high throughput with a much lower operation count. A high bandwidth ceiling also says nothing by itself about the response time of an individual request. The same storage system may show excellent sequential throughput and disappointing latency when many small random requests arrive concurrently.
Rank #2
- Ideal for high speed, low power storage
- Gen 4x4 NVMe PCle performance
- Up to 6,000MB/s read, 4,000MB/s write
- Includes Acronis cloning software
- 5-year limited warranty
Request formation further changes the relationship. A storage layer may merge adjacent small requests, split a large request, or issue several operations in parallel. Queue depth can increase device utilization, but it can also increase waiting time. Consequently, a comparison is meaningful only when the request size, transformations, concurrency and measurement location are visible.
Which metric matters for each AI workload?
AI is not one storage workload. NVIDIA’s AI-storage guidance identifies the patterns and object-size ranges below. They are workload descriptions, not universal limits or performance guarantees; a particular framework, dataset layout and serving architecture can behave differently.
| Workload | I/O pattern described by NVIDIA | Typical object size listed by NVIDIA | Metrics to investigate |
|---|---|---|---|
| Training | Large sequential model and dataset reads, periodic large sequential checkpoint writes, parallel streams | 256 KiB–4 MiB | Sustained achieved throughput or bandwidth, read/write mix, concurrency and checkpoint behavior |
| Fine-tuning | Frequent base-weight reads; iterative small or medium writes of adapter deltas and checkpoints | 64 KiB–1 MiB | Read throughput plus write behavior and latency under the actual mix |
| Inference | Hot model-weight reads and metadata-heavy lookups in the serving graph | 4 KiB–256 KiB | Throughput for weight loading, plus IOPS and latency for lookup activity |
| KV cache | Random reads and writes, temporal locality, and possible persistence or eviction | 1–5000 KiB | Mixed-operation performance, latency, IOPS, locality and cache behavior |
| Query and retrieval | Random reads for approximate-nearest-neighbor or vector lookup; small ingestion and update writes | 1–120 KiB | Random-read IOPS and latency, including write and update behavior |
| Agentic workflows | Many small state, checkpoint, memory, trace, tool-output and artifact operations across concurrent sessions | 1 KiB–4 MiB | Latency and IOPS under concurrency, aggregate throughput and predictable behavior |
These ranges explain why an SSD selected only on its maximum sequential-read figure can disappoint in a retrieval or agent workload. The storage may have ample data-rate capacity, yet spend most of its time completing small requests and waiting on metadata operations.
Training: sustained movement first
Training commonly streams large model and dataset reads and writes large checkpoints at intervals. Start with sustained achieved read throughput, then examine write throughput during checkpoint windows, the read/write mix and the number of parallel streams. A drive that looks fast in a short burst can behave differently when a checkpoint competes with reads or when the dataset is distributed across workers.
Measure the effect at the application level as well: time per epoch, data-loader wait time and checkpoint duration connect a storage result to training progress. A subsystem benchmark is useful for finding a bottleneck, but it does not replace a full training run under representative concurrency.
Fine-tuning: mixed reads and writes
Fine-tuning repeatedly reads base weights while writing adapter deltas and checkpoints. Read throughput remains important, but write behavior and latency during the mixed phase can determine whether workers stall. Test the actual proportions and request sizes instead of applying a pure sequential-read result.
Inference: separate loading from serving
Model loading can be a large, mostly sequential transfer, so throughput affects startup and reload time. Once the model is resident, metadata-heavy lookups and serving-graph activity may use much smaller requests. For that steady state, IOPS and latency under the expected number of concurrent requests are at least as important as the initial load rate.
KV cache: locality and mixed operations
KV-cache systems can issue random reads and writes, exploit temporal locality and evict or persist entries. A single sequential bandwidth test cannot represent that behavior. Investigate mixed-operation IOPS, latency as the cache fills and evicts, and whether the benchmark reproduces the application’s locality. Include the write component; a read-only test can conceal contention created by updates or persistence.
Rank #3
- SPEED UP PROJECTS. Launch creator applications fast with uncompromising PCIe 4.0 read speeds up to 7,100MB/s,[2] (1TB and 2TB[1] models) and write speeds up to 6,700MB/s[2] (1TB[1]-4TB[1] models).
- CREATE AND STORE MORE. Make more room for your 4K videos and high-resolution images with capacities from 500GB[1] up to 4TB[1] on M.2 2280 built with our trusted 8th generation SANDISK BiCS QLC 3D CBA NAND.
- IT GOES WHERE YOU GO. With an all-new power efficient design, your drive delivers high performance with low power, giving you more time to be productive while on the go.
- UNCOMPROMISED RELIABILITY. With up to 1,200 TBW[3] (4TB[1] model) endurance rating, your drive is designed for creators.
- KEEP YOUR DRIVE UPDATED. Monitor your SSD’s performance and check for updates with the downloadable SANDISK Dashboard application.[5]
Query and retrieval: small random requests
Approximate-nearest-neighbor and vector lookup paths commonly perform random reads, while ingestion and updates add small writes. For these systems, random-read IOPS and latency usually deserve priority over a sequential bandwidth headline. Test the index in its deployed layout and include update activity if production performs inserts, deletes or compaction.
Agentic workflows: concurrency and tail behavior
Agents generate many state, memory, trace, tool-output, checkpoint and artifact operations, often from several sessions at once. The object range can extend from 1 KiB to 4 MiB, so aggregate throughput still matters, but latency and IOPS under concurrency are central. Look for predictable response times as sessions increase, not just the peak result from one worker.
When small I/O makes IOPS and latency more important
NVIDIA uses requests under 500 kB as an example of small I/O for which IOPS and latency are more informative than headline throughput: the requests may not be large enough to use the available data rate. This is a rule of thumb, not a universal cutoff. The boundary depends on the device, queueing, interface, filesystem and workload.
Free tools Windows power users keep installed
One-click scans. No signup required.
Large sequential movement makes achieved throughput more visible because each operation carries more data and adjacent requests can keep the path busy. Small or random requests spend a greater share of time initiating, locating and completing individual operations. If your application mixes both, benchmark both phases and report the mix rather than choosing one metric for the entire system.
How to compare storage specifications fairly
Before comparing two NAND-based NVMe SSDs, a storage appliance and a network volume, normalize the test conditions. Record the following for every result:
- Access pattern: sequential or random, and read-only, write-only or a stated read/write mix.
- I/O size: the bytes in each request, including whether the storage layer merged or split requests.
- Concurrency and queue depth: the number of outstanding operations and workers. State whether the test used one queue or many.
- Measurement point: device, host, appliance, filesystem, client or end-to-end application.
- Duration: whether the number is a short peak, a steady-state result or a sustained run after caches and thermal conditions have settled.
- Latency reporting: average and, where available, a distribution or high-percentile value so outliers are visible.
- Application outcome: training time per epoch, checkpoint time, model-load time, inference throughput or query response time alongside component metrics.
NVIDIA cautions that narrow, single-dimension tests may miss workload diversity, concurrency, stress and full-stack interactions in AI infrastructure. Its configuration guide describes microbenchmarks such as fio as useful for measuring a subsystem, while emphasizing that those results are building blocks rather than substitutes for the complete application.
A practical benchmark sequence
- Describe the workload first. Capture the operation sizes, read/write ratio, sequential or random behavior, expected workers and target latency. Use production traces when available; otherwise document the assumptions.
- Measure the isolated storage path. Run a controlled test such as
fioto establish throughput, IOPS and latency under the chosen conditions. Keep the test settings identical when comparing systems. - Increase concurrency deliberately. Repeat at the queue depths and worker counts your application uses. Observe both utilization and latency; a higher queue can raise throughput while worsening response time.
- Test sustained behavior. Include the duration of a training stream, a checkpoint interval, a cache-fill cycle or a retrieval update period. Record changes over time instead of keeping only the best moment.
- Run the real application. Measure the outcome that matters to users: epoch duration, model-load time, tokens or inferences per second, query latency or completed agent sessions.
- Identify the limiting layer. Compare device results with host configuration, PCIe link capacity, network path, filesystem and shared-storage behavior. Improve the layer that actually limits the workload.
Why queue depth and request supply change the result
A storage device cannot complete requests that have not been issued. Too few outstanding operations can leave a high-IOPS or high-bandwidth volume underfed, so a benchmark may report less than the provisioned capability even though the volume itself is healthy. Conversely, adding requests can raise utilization and throughput while increasing queueing latency.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsThere is no universally correct queue length. Choose it from the application’s concurrency and latency target, then show how throughput and latency change as the queue grows. A result at queue depth 32 should not be presented as the expected latency for a single-request client.
Rank #4
- HUGE SPEED BOOST: Get random read/write speeds that are 40%/55% faster than 980 PRO; Experience up to 1400K/1550K IOPS, while sequential read/write speeds up to 7,450/6,900 MB/s reach near the max performance of PCIe 4.0*
- BREAKTHROUGH POWER EFFICIENCY: Use less power and get more performance; Enjoy up to 50% improved performance per watt over 980 PRO, plus optimal power efficiency with max PCIe 4.0 performance**
- SMART THERMAL CONTROL: Samsung's own nickel-coated controller delivers effective thermal control; With its slim size, 990 PRO is a perfect fit for desktops and laptops that meet the PCI-SIG D8 standard***
- THE CHAMPION MAKER: Up to 65% improvement in random performance enables faster loads for an ultimate gaming experience on PS5 and DirectStorage PC games****
- SAMSUNG MAGICIAN SOFTWARE: Get the most out of your SSD with Samsung Magician's advanced yet intuitive optimization tools; Monitor drive health, protect valuable data, and receive important updates for your 990 PRO
AWS EBS illustrates why implementation rules matter
AWS EBS provides a concrete, service-specific example of request accounting. Its SSD volumes cap an I/O at 256 KiB; larger requests may be split, while sequential smaller requests may be merged. Random smaller requests are counted separately. The service also notes that the appropriate queue length varies by workload and that an insufficient request supply can prevent a volume from reaching provisioned IOPS or throughput. See the AWS explanation at EBS I/O characteristics and monitoring.
Those rules describe EBS, not NAND or SSD technology in general. They demonstrate why two tests with the same nominal I/O size can produce different operation counts and why a cloud-volume result cannot be transferred directly to a local drive. When publishing or reviewing a benchmark, state the provider’s request-size and accounting rules with the number they affect.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Interpreting local NVMe SSD specifications
For a local, direct-attached NVMe SSD, the specification sheet may list maximum throughput, IOPS and latency. NVIDIA notes that theoretical performance also depends on PCIe bandwidth and the configured lane count. The drive’s advertised maximum is therefore not an application-level guarantee.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchCheck that the host exposes the intended PCIe generation and lane width, that the workload can supply enough parallel requests, and that the measurement reaches the same endpoint your application uses. A drive-level figure can be useful for comparing devices under identical conditions, but a GPU-to-storage or client-to-storage result includes additional software, bus and scheduling effects.
When shopping or planning a local storage tier, compare NVMe SSD specifications using the workload table above rather than searching for a universal “best for AI” model. A product category can be appropriate while the right endurance, capacity, interface and performance profile still depend on whether the workload is streaming training data, serving random retrievals or handling many agent sessions.
A decision framework for choosing the metric
Use this sequence to turn a specification sheet into a workload decision:
- Classify the dominant operation. Is the path moving large sequential files, or completing many small random requests?
- Write down the request size. Without bytes per operation, an IOPS figure is not comparable.
- Set the concurrency target. Include workers, users, serving threads and queue depth rather than testing one request in isolation.
- Choose the primary metric. Start with sustained throughput for large streams; start with IOPS and latency for small, random or interactive activity.
- Keep the secondary metrics. Training still needs checkpoint latency; retrieval still needs update throughput; inference still needs model-load throughput.
- Verify the bottleneck end to end. Confirm that the storage result changes the application outcome and that PCIe, network, host or shared-storage limits are not dominant.
- Prefer predictable behavior. A slightly lower peak with stable latency and sustained throughput can be more useful than a short burst that collapses under the production workload.
Common mistakes and how to correct them
Choosing the highest advertised bandwidth
Problem: A sequential maximum is used to predict random retrieval or metadata performance.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Correction: Test the random request sizes, IOPS and latency that the application issues, then validate query or inference response time.
Best Value
- This product has been replaced by our latest generation. Please search for the SANDISK Optimus GX 7100 NVMe SSD
- HIGH-OCTANE GAMING. Experience speeds up to 7,250MB/s read and 6,900MB/s write (1-2TB models), with up to 35% faster performance than previous generation.
- PURPOSE-BUILT. Designed for serious on-the-go gamers, with a PCIe Gen4 interface and SANDISK’s next generation TLC 3D NAND.
- MORE TIME TO CLEAR THAT CHECKPOINT. Built with laptops and handheld gaming devices in mind, with up to 100% more power efficiency over the previous generation.
- DO MORE WITH DASHBOARD. Ensure your drive is optimized for prime performance with the downloadable WD_BLACK Dashboard (Windows only).
Comparing IOPS without request size
Problem: Two “million IOPS” claims are treated as equivalent.
Correction: Record bytes per operation, read/write mix, queue depth and whether the result is a peak or sustained value.
Reporting average latency alone
Problem: A low mean hides occasional long pauses that affect users or synchronization.
Recommended Free Tools
Correction: Capture a latency distribution or high percentile under realistic concurrency, and state where latency was measured.
Underfeeding the device
Problem: A benchmark issues too few outstanding requests and concludes that provisioned performance is unavailable.
Correction: Increase queue depth and worker count to the application’s level while tracking the latency cost.
Treating a microbenchmark as the application result
Problem: A subsystem number is presented as training, inference or retrieval performance.
Correction: Use the microbenchmark to isolate storage, then run the full workload and report the end-to-end outcome.
Ignoring the rest of the path
Problem: A drive specification is blamed when PCIe lanes, host settings, network bandwidth or shared storage is the actual limit.
Correction: Measure at multiple points and confirm the configured interface and lane count before changing hardware.
The bottom line for AI storage planning
Bandwidth or throughput tells you how quickly a system can move data; IOPS tells you how many operations it can complete; latency tells you how long an operation takes. Large sequential training streams emphasize sustained throughput, while small random retrieval, metadata-heavy inference, KV-cache activity and concurrent agent workflows often make IOPS and latency decisive. Because AI pipelines mix these patterns, select and compare storage using the real request sizes, queueing, concurrency and measurement point, then verify the choice with the application’s own performance metric.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
