iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Distributed AI training has no universal bandwidth requirement such as “100 Gb/s per GPU.” The right amount depends on the tensor bytes exchanged in each training step, the collective operation (AllReduce, AllGather, ReduceScatter or AlltoAll), the number of participating ranks, and the bandwidth your software actually achieves. A useful first estimate is communication time ≈ bytes moved per rank ÷ achieved application bandwidth + latency/overhead.
For a ring AllReduce of a tensor containing S bytes across N ranks, each rank moves approximately 2(N−1)S/N bytes. That approaches 2S as the cluster grows. It is a traffic estimate, not a guaranteed port speed: bucketing, overlap with computation, compression, topology, congestion and implementation details can materially change the result.
Is there one bandwidth number for distributed AI training?
No. Bandwidth demand is a property of the workload and cluster design, not just the model’s parameter count. Two jobs with the same model can generate different network loads when one uses data-parallel gradient synchronization and the other shards parameters or exchanges activations between model-parallel stages.
The practical target is sustained, measured bandwidth for the dominant collective while retaining headroom for protocol overhead, competing traffic, congestion and tail latency. A nominal link rate is only the starting point. Job completion time, training throughput and accelerator utilization are the outcomes that determine whether the network is adequate.
#1 Best Overall
- The 25Gb dual-port SFP+ network card is based on the Mellanox ConnectX-5 Ex controller, which provide the highest performing and most flexible interconnect solution.
- Technical Support:PXE、 RDMA、UEFI、SR-IOV、1588 PTP、Jumbo Frames（9.5KB）
- Windows 10/11、Windows Server 2016/2019/2022、Deepin 15.11/20/20.6/20.9、VMware ESXi 6.5/6.7、Ubuntu 18.04.5/20.04.1、Ubuntu 22.04.2/22.04.3、RHEL/CentOS 7.6/7.9/8.2/8.3、ZTE New Fulcrum 3.2.2/5.0.5、SUSE 12.5/15.4、FreeBSD 13.2、NeoKylin 7.6、OpenKylin 0.7.5、Mikrotik、iKuai route、Galaxy Kylin v10、Zhongke Fangde desktop OS、Zhongke Fangde server OS、Tongxin UOS 20、Emind OS
- install the operating system with its driver CD, or download it from the official website. Includes low-profile and full-height stands to support standard and ultra-thin computers/servers.
- Enjoy 24/7 customer service, 30-day free returns, 1-year free warranty, and lifetime technical support for your peace of mind.
Why synchronization is on the critical path
Every training iteration performs forward, backward and optimizer work. During those phases, participating GPUs repeatedly synchronize gradients or updated parameters, often within milliseconds, and continue doing so until convergence. If communication takes longer than the computation it is supposed to overlap, GPUs wait at synchronization points and the cluster’s effective training rate falls.
What creates network traffic during each training step?
The parallelism strategy determines which tensors move and which collective operation carries them. Identify that strategy before converting model size into a network requirement.
Data-parallel training (DDP)
In DistributedDataParallel, each rank computes gradients for its local batch and synchronizes them with an AllReduce. Every rank contributes and receives the reduced gradient, so the amount of traffic is tied to the gradient bucket sizes and the number of ranks. Bucketing can overlap communication with backpropagation, changing how much link capacity is needed at any instant without changing the total bytes required by the collective.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Fully sharded training (FSDP)
Fully Sharded Data Parallelism distributes parameter, gradient and optimizer state shards. It commonly uses AllGather to materialize parameters for a computation and ReduceScatter to return reduced gradient shards. Traffic is therefore spread across multiple collectives rather than one full-gradient AllReduce, and the best network design depends on when those collectives run and whether they overlap with computation.
Model-parallel and ranking workloads
Tensor, pipeline or expert-style model parallelism can require AlltoAll or AlltoAllv exchanges. AlltoAllv is especially demanding because message sizes can differ by rank, creating full-mesh traffic and transient incast. A fabric that performs well for uniform AllReduce may show congestion or tail-latency problems under this pattern.
How do I estimate bytes moved by an AllReduce?
Use the tensor size, rank count and measured application bandwidth as a first-order model. Let S be the bytes in the tensor participating in a collective, N the number of ranks, and B_app the achieved application-level bandwidth.
- For ring AllReduce, estimate per-rank traffic with
bytes_per_rank ≈ 2(N−1)S/N. - Estimate collective time with
time_per_collective ≈ bytes_per_rank / B_app + latency and software overhead. - Repeat the estimate for every large collective in an iteration, then account for overlap and the frequency of each operation.
Worked example
Suppose an 8-rank job performs a ring AllReduce on an 8 GB tensor. The estimate is 2(8−1)×8/8 = 14 GB of traffic per rank for that collective. That 14 GB is not a requirement for a 14 GB/s link: the collective may overlap with computation, and the application may achieve less than the physical port rate because of protocol, software and topology overhead.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #2
- Host Interface: PCI Express 5.0 x16
- Total Number of Ports: 1
- Expansion Slot Type: OSFP
- Media Type Supported: Optical Fiber
- Maximum Data Transfer Rate: 400 Gbit/s
To convert a measured rate, keep units explicit. Network vendors quote bits per second, while tensor sizes are usually expressed in bytes. Multiply gigabytes per second by eight to obtain gigabits per second, and state whether decimal or binary units are being used. Use the measured NCCL rate at the message size and rank count that resemble the real job, rather than a peak specification from a NIC datasheet.
How many Gbps does each GPU need?
There is no defensible universal per-GPU figure. A per-GPU requirement emerges only after you specify tensor bytes, precision, batch size, collective type, rank count, overlap and topology. The same accelerator can need modest bandwidth for a communication-light data-parallel job or substantially more for sharded or AlltoAll-heavy training.
- Tensor volume: Larger gradients, parameter shards or activation exchanges increase bytes moved.
- Precision: Lower-precision communication can reduce bytes, but only if the training implementation actually communicates in that format.
- Rank count: Ring AllReduce traffic approaches two tensor sizes per rank as the number of ranks increases.
- Overlap: Effective throughput can remain acceptable when communication is hidden behind computation; a short burst can still require high instantaneous bandwidth.
- Topology: Oversubscription, limited bisection bandwidth, hop count and NIC placement can matter more than the port’s headline rate.
- Software efficiency: NCCL algorithm choice, bucket sizing, protocol behavior and CPU or PCIe bottlenecks determine achieved application bandwidth.
For these reasons, avoid promises such as “100 Gb/s per GPU” or “200 Gb/s per node” unless they are tied to a named model, precision, batch size, rank count, collective, topology and benchmark method.
Is 100GbE enough for distributed training?
Sometimes, but the speed label alone cannot answer the question. A 100GbE fabric may be sufficient when measured collective time is hidden by computation and the network is not oversubscribed. It may be inadequate when large collectives dominate each iteration, many GPUs share uplinks, or congestion causes long-tail delays.
Test the actual job shape at realistic message sizes and rank counts. Compare the communication time with the compute time that can overlap it, then verify end-to-end throughput and GPU utilization. If GPUs frequently sit idle at gradient synchronization, increasing link speed may help, but so might changing bucket sizes, placement, collective algorithms or the degree of oversubscription.
Do I need InfiniBand or RoCEv2?
Both can provide RDMA-based GPU-to-GPU communication, but neither guarantees a particular training rate. InfiniBand is designed for low-latency, high-bandwidth communication across cluster nodes; Azure guidance identifies RDMA-capable virtual-machine families for distributed training. RoCEv2 carries RDMA over Ethernet, so its results depend heavily on Ethernet switching, routing and congestion-management configuration.
| Design concern | Why it matters |
|---|---|
| Achieved NCCL algorithm and bus bandwidth | Shows what the application and fabric deliver, rather than the port’s theoretical maximum. |
| Collective and message-size scaling | AllReduce, AllGather and AlltoAll can stress different paths and produce different rates. |
| Per-GPU and per-node port speed | Determines local injection capacity and whether several GPUs compete for one uplink. |
| Bisection bandwidth and oversubscription | Reveals whether simultaneous cross-rack traffic can use the advertised aggregate capacity. |
| Median and tail latency under load | Long outliers can stall synchronization even when average bandwidth looks healthy. |
| Congestion behavior and loss recovery | Especially important for RoCEv2 and irregular AlltoAllv traffic. |
| NIC, switch, cable, driver and topology compatibility | A mismatch can prevent GPUDirect or force a slower data path. |
Choose the fabric that your team can operate and benchmark consistently. An InfiniBand network adapter or other RDMA-capable NIC is only one component; it must match the GPU server, switch, port speed, cabling, driver, firmware and NCCL topology.
What should I provision for a production cluster?
Provision for measured sustained demand, not for a single theoretical formula. Use this workflow before ordering links or hosts.
- Document the parallelism plan. Record whether the job uses DDP, FSDP, tensor or pipeline parallelism, expert routing, or a combination.
- List dominant collectives. Capture AllReduce, AllGather, ReduceScatter, AlltoAll and AlltoAllv operations, their tensor sizes and how often they run.
- Estimate traffic. Apply the ring formula where appropriate and calculate bytes per rank for other collectives from their communication pattern.
- Benchmark NCCL. Measure algorithm bandwidth and bus bandwidth at realistic message sizes, GPU counts and node counts, using the same drivers, firmware and topology planned for production.
- Check the fabric. Inspect oversubscription, bisection bandwidth, routing, congestion controls, NIC-to-GPU affinity and PCIe paths.
- Test under contention. Run simultaneous jobs or background flows to expose tail latency and transient incast that a quiet-cluster test hides.
- Validate the outcome. Track job completion time, training throughput, synchronization wait time and accelerator utilization. Increase capacity or change placement only when these measures show a material bottleneck.
How should I benchmark a network for AI training?
Microbenchmarks are necessary but not sufficient. NCCL tests reveal collective behavior, while an end-to-end training run shows whether communication actually limits progress.
Run representative collective tests
Benchmark AllReduce, AlltoAll and AllGather with the message sizes, rank counts and node counts used by the workload. Include both single-node and multi-node runs. Record algorithm bandwidth and bus bandwidth separately; the former reflects application progress, while the latter normalizes collective traffic for comparison.
Measure scale and soak behavior
Repeat tests as the world size grows and sustain them long enough to expose thermal, routing or congestion effects. The IETF AI Fabric Benchmarking draft calls for coverage of AllReduce, AlltoAll, RDMA/RoCEv2, congestion management, load balancing and scale/soak behavior, with job completion time, training throughput and accelerator utilization as key outcomes.
Interpret published numbers carefully
Results from a paper or vendor test are experiment-specific. They are useful for forming hypotheses, not for guaranteeing capacity in a different cluster. Reproduce the relevant topology, software stack, message sizes and rank count before using a figure for procurement.
Recommended Free Tools
Why are my GPUs waiting on gradient synchronization?
Waiting usually means the communication path is slower or less predictable than the computation schedule assumes. Check the following in order.
- Collective mismatch: Confirm that the job is using the intended NCCL collective and not falling back to a slower path.
- Incorrect topology: Verify GPU-to-NIC affinity, switch paths, cable placement and GPUDirect RDMA availability.
- Oversubscribed uplinks: Compare aggregate GPU injection bandwidth with the bisection capacity of the spine, leaf and host links.
- Message fragmentation: Review bucket sizes and the number of small collectives; excessive fragmentation raises per-operation latency.
- Congestion: Inspect queueing, packet loss, pause or priority-flow-control events and tail latency during concurrent traffic.
- Software and firmware: Align NCCL, GPU drivers, NIC drivers, switch firmware and RDMA settings across nodes.
- Uneven work: A straggling rank can make all other GPUs wait even when average network bandwidth is adequate.
What do published measurements actually show?
The following figures illustrate how strongly results depend on the experiment. They should not be treated as universal network guarantees.
Rank #4
- DUAL-PROTOCOL 100G: ConnectX-4 VPI (MCX456A-ECAT) runs EDR InfiniBand 100Gb/s or 100GbE per QSFP28 port with 100G/50G/40G/25G/10G auto-negotiation — one card serves IB and Ethernet fabrics.
- PCIe 3.0 x16, FULL BANDWIDTH: Dual ports sustain line-rate 100Gb/s each for HPC, AI training nodes and high-throughput storage fabrics.
- RDMA WITHOUT CPU COPIES: Native InfiniBand RDMA plus RoCE accelerate MPI, NVMe-oF and distributed storage; hardware offloads cut latency and free CPU cycles.
- HEAVY VIRTUALIZATION: SR-IOV with up to 127 VFs per port (254 per card) plus VXLAN/GENEVE/NVGRE overlay offload for multi-tenant clouds and dense VM hosts.
- DATA CENTER FEATURES: PXE/UEFI boot, NC-SI management, DCB, jumbo frames; Linux (MLNX_OFED), Windows (WinOF) and VMware ESXi support; brackets for any chassis.
| Published result | Context |
|---|---|
| 30% of TCP/IP-link bandwidth; 5–10% of RDMA bandwidth | AIACC-Training authors (2022), reporting the utilization ceiling of a single communication stream in their cited setup. |
| Scaling efficiency above 0.96 | AIACC-Training authors (2022), for multi-stream decentralized gradient-communication experiments. |
| 1.3× and 1.8× improvement over Horovod on ResNet-50 and VGG-16 at 32 GPUs; 3.3× at 256 GPUs | AIACC-Training authors (2022), in the experiments reported by the paper. |
| 4 to 512 nodes | Transactions on Machine Learning Research (2025), the NCCL measurement range shown on DGX H100 systems with InfiniBand. |
A single stream can therefore underuse a fast link, while multiple streams or a different collective can raise utilization substantially. Always attach the workload, software and topology to a number.
How do I choose between a faster link and a different cluster design?
Choose the smallest change that improves the metric limiting job completion time. If NCCL bandwidth is low because of a topology or configuration error, a faster port will not fix it. If the fabric is healthy but communication occupies a large fraction of each iteration, higher per-GPU bandwidth, more bisection capacity or fewer cross-node exchanges may help.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
For large language model training, evaluate the complete path: GPU memory and PCIe, NIC placement, host links, leaf-spine capacity, routing, collective algorithms and software versions. Compare candidate designs using the same collective suite and report median and tail latency, achieved bandwidth, GPU utilization and time to finish a fixed training workload. That evidence is more useful than a generic per-GPU rule.
Frequently Asked Questions
What is the difference between NCCL algorithm bandwidth and bus bandwidth?
Algorithm bandwidth describes the rate at which the collective moves useful tensor data from the application’s perspective. Bus bandwidth normalizes the collective’s internal traffic so results from different algorithms or rank counts can be compared; both should be recorded because a high value in one does not guarantee a high value in the other.
Can gradient compression eliminate a network bottleneck?
Compression can reduce bytes per collective, but it adds encoding, decoding and possible synchronization overhead. Its benefit depends on the algorithm, hardware support and whether the reduced payload still preserves convergence behavior, so measure end-to-end training throughput rather than assuming a proportional speedup.
What does bisection bandwidth mean in a GPU cluster?
Bisection bandwidth is the aggregate capacity available when the network is divided into two halves and traffic crosses between them. It indicates whether many simultaneous GPU-to-GPU flows can run at once; a cluster can have fast host ports yet poor bisection capacity because of oversubscribed uplinks.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesWhy can one communication stream use only a small fraction of a link?
A single stream may be limited by protocol processing, message size, software serialization or the sender and receiver path rather than the physical link. The AIACC-Training authors reported a single-stream ceiling of 30% of TCP/IP-link bandwidth and 5–10% of RDMA bandwidth in their 2022 setup, so multi-stream and application-level tests are important.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
