iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The reliable way to reduce a network bottleneck in multi-GPU training is to measure the communication path in layers, then change only the layer that fails. First test GPU-to-GPU links inside a node, then measure node-to-node bandwidth and latency, then inspect NCCL collectives under the real job. Check topology, link rates, errors, and congestion counters before changing NCCL variables. A low-level test that is already below the hardware design target points to a platform or fabric problem; a healthy low-level path with slow collectives points to placement, NCCL settings, or workload behavior.
Do not use a single peak-bandwidth number as proof that a cluster is healthy. Tail latency, retries, queue drops, and collective completion time can stall training even when a short throughput test looks normal.
Start with a layered diagnosis
Distributed training crosses several distinct paths:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Local GPU path: GPU-to-GPU traffic over NVLink or PCIe, including the PCIe root complex and peer-to-peer access.
- GPU-to-NIC path: movement from GPU memory to the network adapter, ideally using a supported GPUDirect RDMA path rather than unnecessary host-memory staging.
- Inter-node fabric: InfiniBand or RoCE links, switches, routing, congestion control, and error handling.
- Collective software and workload: NCCL topology selection, rank placement, collective type, message size, and job scale.
NVIDIA’s tuning guidance recommends checking the lower-level GPU and fabric paths before blaming NCCL configuration. Record the context with every result so comparisons are meaningful:
#1 Best Overall
- The 25Gb dual-port SFP+ network card is based on the Mellanox ConnectX-5 Ex controller, which provide the highest performing and most flexible interconnect solution.
- Technical Support:PXE、 RDMA、UEFI、SR-IOV、1588 PTP、Jumbo Frames（9.5KB）
- Windows 10/11、Windows Server 2016/2019/2022、Deepin 15.11/20/20.6/20.9、VMware ESXi 6.5/6.7、Ubuntu 18.04.5/20.04.1、Ubuntu 22.04.2/22.04.3、RHEL/CentOS 7.6/7.9/8.2/8.3、ZTE New Fulcrum 3.2.2/5.0.5、SUSE 12.5/15.4、FreeBSD 13.2、NeoKylin 7.6、OpenKylin 0.7.5、Mikrotik、iKuai route、Galaxy Kylin v10、Zhongke Fangde desktop OS、Zhongke Fangde server OS、Tongxin UOS 20、Emind OS
- install the operating system with its driver CD, or download it from the official website. Includes low-profile and full-height stands to support standard and ultra-thin computers/servers.
- Enjoy 24/7 customer service, 30-day free returns, 1-year free warranty, and lifetime technical support for your peace of mind.
- GPU model, count, and
nvidia-smi topo -moutput - NIC or HCA model, port speed, link layer, and negotiated rate
- CPU and NUMA layout, rank placement, and container or virtual-machine topology
- NCCL, CUDA, driver, kernel, and perftest versions
- Job size, number of nodes, collective type, message sizes, and whether the test used host or GPU memory
Compare like with like. A host-memory RDMA result cannot be compared directly with a GPU-memory result, and an AllReduce benchmark at one message size does not predict AllGather or AlltoAll behavior at another scale.
1. Verify the local GPU topology and peer path
Map the hardware connections
On each node, run:
nvidia-smi topo -m
This matrix shows how GPUs connect to one another and to NICs, CPUs, and PCIe root complexes. Prefer rank and NIC placement that follows the shortest, highest-bandwidth path shown by the topology. A virtual machine or container can expose a PCI topology that does not reflect the physical machine; NCCL may then select a suboptimal route.
Check peer-to-peer capability
Check PCIe peer access with:
nvidia-smi topo -p2p p
Check NVLink peer access with:
nvidia-smi topo -p2p n
These commands report capability, not measured performance. A peer relationship that appears available can still be slow or unusable because of an unhealthy link, a platform setting, or a driver and kernel interaction.
Measure GPU-pair bandwidth
Use nvbandwidth to measure representative GPU-to-GPU pairs. Test pairs that training will actually use, not only the pair with the best topology. Capture both directions where the tool supports it and note whether transfers use NVLink or PCIe.
If measured throughput is low, investigate the local path before changing network settings. NVIDIA documents bare-metal IOMMU and PCI Access Control Services (ACS) as possible causes of failed or degraded PCIe peer-to-peer memory transfers. In affected Linux bare-metal PCIe configurations, translated IOMMU mode is unsupported for CUDA/NVIDIA-driver PCIe peer-to-peer memory transfer. IOMMU and ACS changes alter host platform behavior, so validate the correct setting for the server, firmware, operating system, and security requirements instead of applying a generic kernel argument.
Interpret a local-path failure
- Unexpected topology: check GPU seating, PCIe bifurcation, root-complex placement, firmware, and whether virtualization is hiding the physical map.
- Peer access unavailable: verify driver and CUDA compatibility, then investigate IOMMU or ACS configuration where applicable.
- Peer access available but bandwidth low: inspect link generation and width, NVLink health, PCIe errors, and thermal or power throttling before touching NCCL.
- Only one GPU pair is slow: isolate that pair and its shared switch, root complex, cable, or bridge; do not average it away with faster pairs.
2. Measure node-to-node bandwidth and latency separately
Confirm InfiniBand port state
For an InfiniBand or RDMA link check, inspect port state, link layer, and negotiated rate with:
ibstatusibstat
The port should be active, use the expected link layer, and negotiate the rate designed for the adapter, cable, and switch port. A downshifted rate is a physical or configuration clue, not an NCCL tuning problem.
Run throughput and latency tests
Use ib_write_bw for node-to-node bandwidth and ib_write_lat for point-to-point latency. Run the client and server with the same message size, queue settings, and memory type, and repeat across the paths used by the training job. Follow the exact options supported by the installed build:
Rank #2
- Host Interface: PCI Express 5.0 x16
- Total Number of Ports: 1
- Expansion Slot Type: OSFP
- Media Type Supported: Optical Fiber
- Maximum Data Transfer Rate: 400 Gbit/s
ib_write_bw --help
GPU-memory options differ by perftest build and platform. Confirm the syntax locally rather than copying a flag from an unrelated installation. Compare host-memory and GPU-memory runs when validating GPUDirect RDMA.
Bandwidth and latency answer different questions. Throughput can remain near its peak while queueing creates long latency tails. If training pauses intermittently, collect latency distributions or repeated samples, not just the best or average result.
Inspect InfiniBand fabric health
Confirm that a Subnet Manager is running. Inspect port counters for symbol errors, link recovery, and link-down events. Where access permits, use ibdiagnet to examine broader topology, routing, and fabric errors. Rising retry or error counters, repeated link recovery, or a route that differs from the intended design warrants checking cabling, switch ports, routing, firmware, and driver configuration.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesInspect RoCE congestion signals
For RoCE, inspect NIC statistics with the appropriate ethtool -S command and correlate them with switch counters. Look for:
- packet drops and queue drops
- pause frames and Priority Flow Control (PFC) events
- Explicit Congestion Notification (ECN) marks
- Congestion Notification Packets (CNPs)
- unstable throughput or widening latency tails
When these counters rise during a slowdown, investigate congestion-control and lossless-fabric design before changing NCCL. PFC or ECN activity is a diagnostic signal, not proof of one specific fault; verify queue configuration, priorities, buffer behavior, routing, and the switch and NIC design together.
Use the measurements to narrow the fault
| Observation | Likely layer to investigate first | Next checks |
|---|---|---|
Low nvbandwidth result |
GPU, PCIe, or NVLink path | Topology, link width and generation, IOMMU/ACS, firmware, peer-access errors |
Low ib_write_bw with host memory |
Adapter, cable, switch, or fabric | Port state and rate, counters, routing, Subnet Manager, cabling |
| Host bandwidth normal but GPU-memory bandwidth low | GPUDirect RDMA path | Peer-memory mechanism, driver, kernel, adapter support, perftest GPU options |
| Bandwidth normal but latency tails or retries rise | Congestion or fabric errors | RoCE PFC/ECN/CNP and queue drops, InfiniBand recovery and error counters |
| Low-level tests meet design expectations but NCCL is slow | Placement, NCCL, or workload | NCCL topology logs, rank-to-GPU/NIC mapping, collective and message-size behavior |
3. Validate the GPU-to-NIC path and process placement
Confirm the GPUDirect RDMA method
GPUDirect RDMA can transfer between supported GPU memory and a network adapter without the ordinary host-memory staging path. It is conditional on the exact GPU, adapter, driver, kernel, and platform. NVIDIA documents nvidia-peermem as a module that may need to be loaded; supported recent-kernel DMA-BUF configurations can provide an alternative. Determine which method is supported by the installed stack and verify that the module or DMA-BUF path is actually active.
Do not infer GPUDirect operation from a successful job alone. Run matched host-memory and GPU-memory perftest cases, record the options and buffer type, and compare the result with the hardware design target. A difference between those cases can indicate an unsupported path, an incorrect module, a driver mismatch, or a platform limitation.
Recommended Free Tools
Align ranks, CPUs, GPUs, and NICs
On NUMA systems, bind each process and its host memory near the GPU and NIC it uses. Use the host CPU/NUMA map together with nvidia-smi topo -m and NCCL initialization/topology logs. A rank that uses a GPU on one socket and a NIC on another may add a CPU interconnect hop even when the network itself is healthy.
Check the mapping for every node. A single misplaced rank can make a collective wait for the slowest participant. In containers, expose the required devices and verify that the visible PCI topology is consistent with the physical host. In virtual machines, compare the guest’s topology with the host design; an artificial layout can cause NCCL to choose the wrong transport or path.
Capture a placement record
- GPU UUID and local rank for every process
- NIC or HCA port selected by that process
- CPU affinity and NUMA node
- PCIe and NVLink relationship from the topology matrix
- container or VM device and PCI exposure
Keep this record with benchmark output. It lets you distinguish a fabric regression from a placement change after a scheduler, image, or virtualization update.
4. Examine NCCL only after the paths pass
Collect a baseline
Run a representative NCCL test or a short slice of the real training job after the low-level checks. Use the same number of nodes, ranks per node, message-size range, and collective mix as production. Preserve NCCL initialization and topology logs, software versions, environment settings, rank placement, and GPU and NIC identifiers.
NCCL uses topology-aware defaults, and a setting that helps one benchmark can hurt a different collective or job scale. Treat every change as an experiment with a recorded before-and-after baseline.
Investigate RoCE queue-pair parallelism
On RoCE, multiple queue pairs per connection may be needed to saturate the fabric. NCCL exposes NCCL_IB_QPS_PER_CONNECTION for this investigation. Change it only when measurements suggest insufficient parallelism, then rerun the representative workload. More queue pairs can alter resource use and congestion behavior; a larger value is not automatically faster.
Evaluate buffer size and CTA trade-offs
NCCL buffer sizing and communication CTA counts change how communication uses GPU memory and compute resources. Larger buffers or more communication CTAs can improve overlap in one workload while consuming memory or reducing compute capacity in another. Change one parameter at a time, document the exact value, and watch both step time and GPU memory headroom.
Use logs as evidence, not as a substitute for tests
NCCL debug output can reveal the selected topology, environment settings, transport, and CPU affinity. Enable logging selectively because verbose output can add noise and storage overhead. Compare the log’s selected path with the physical topology and the intended rank-to-NIC mapping. Forcing a NIC or transport is useful as a diagnostic experiment; retain the default unless measurements show that the forced choice is appropriate for the production topology.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Validate with training metrics
Microbenchmarks isolate communication, but training exposes overlap and synchronization effects. Measure step time, time spent waiting at collective calls, utilization during communication, and variability between steps. A setting that raises a bandwidth number but increases memory pressure or causes compute starvation is not an improvement for the job.
Rank #4
- DUAL-PROTOCOL 100G: ConnectX-4 VPI (MCX456A-ECAT) runs EDR InfiniBand 100Gb/s or 100GbE per QSFP28 port with 100G/50G/40G/25G/10G auto-negotiation — one card serves IB and Ethernet fabrics.
- PCIe 3.0 x16, FULL BANDWIDTH: Dual ports sustain line-rate 100Gb/s each for HPC, AI training nodes and high-throughput storage fabrics.
- RDMA WITHOUT CPU COPIES: Native InfiniBand RDMA plus RoCE accelerate MPI, NVMe-oF and distributed storage; hardware offloads cut latency and free CPU cycles.
- HEAVY VIRTUALIZATION: SR-IOV with up to 127 VFs per port (254 per card) plus VXLAN/GENEVE/NVGRE overlay offload for multi-tenant clouds and dense VM hosts.
- DATA CENTER FEATURES: PXE/UEFI boot, NC-SI management, DCB, jumbo frames; Linux (MLNX_OFED), Windows (WinOF) and VMware ESXi support; brackets for any chassis.
5. Match the diagnosis to the collective and workload
Understand what each collective stresses
| Workload pattern | Communication behavior to examine | Useful comparison |
|---|---|---|
| Data parallelism with AllReduce | Repeated global aggregation and synchronization | Scaling efficiency as nodes and message sizes increase |
| Sharded or model-parallel jobs using ReduceScatter | Partitioned reductions and synchronization frequency | Per-rank completion time and imbalance |
| Parameter or activation exchange using AllGather | Many ranks receiving distributed data | Receive-side queueing and message-size sensitivity |
| AlltoAll-heavy workloads | Many concurrent peer-to-peer flows | Congestion, routing, and tail latency under full fan-out |
Message size and job scale can change the best path and algorithm. Test the sizes generated by the model rather than selecting one convenient payload. Include the number of nodes and ranks because a fabric that is adequate for two nodes may show contention or routing hot spots at larger scale.
Compare InfiniBand and RoCE on the actual system
Both InfiniBand and RoCE are viable RDMA approaches. The available evidence does not establish a universal speed winner. Compare them using the metrics that affect your job:
- measured bandwidth and latency, including tails
- behavior under concurrent training traffic
- topology, routing, and oversubscription
- congestion-control and lossless-fabric configuration
- adapter, switch, cable, driver, and kernel support
- operational tooling, monitoring, and the team’s ability to troubleshoot failures
- collective pattern, message sizes, and intended job scale
A SIGCOMM 2024 paper by Meta’s authors describes production RoCE clusters supporting distributed GPU training at the scale of thousands of GPUs. It emphasizes co-design of topology, routing, transport, operations, and collective-library behavior. The paper also discusses designs supporting up to 32,000 GPUs and analyzes approximately 30,000 randomly selected training jobs from Meta’s fourth quarter of 2023. Those figures describe Meta’s fleet, designs, and workload dataset; they are not a universal performance guarantee or a controlled comparison against InfiniBand.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Common symptoms and corrective branches
Training is slow from the first step
- Compare
nvbandwidthacross the GPU pairs used by each rank. - Check
nvidia-smi topo -mfor an unexpected GPU-NIC or CPU path. - Run matched host- and GPU-memory
ib_write_bwtests. - Inspect NCCL topology and affinity logs for a misplaced rank or wrong NIC.
A consistently slow result usually points to topology, link rate, placement, or an unsupported data path rather than transient congestion.
Training is fast, then stalls or varies between steps
- Measure latency repeatedly, not only peak bandwidth.
- Correlate stalls with RoCE PFC, ECN, CNP, and queue-drop counters or InfiniBand retry and recovery counters.
- Check whether another job is sharing the links or switch queues.
- Compare the affected collective and message size with a quieter baseline.
Intermittent behavior is consistent with queueing, congestion, or fabric errors, but counters and repeated measurements are needed to distinguish them.
Only multi-node runs are slow
Local GPU tests that pass while inter-node tests fail move the investigation to the NIC, HCA, fabric, routing, congestion control, or GPUDirect path. Do not spend time changing NVLink-related settings when the node-to-node bandwidth or latency test is already below the design expectation.
NCCL is slower than the RDMA test
That result does not automatically indicate a bad network. NCCL adds collective scheduling, synchronization, topology decisions, protocol choices, and workload-specific message patterns. Verify rank placement and selected topology, then test the actual collective mix and message sizes before changing environment variables.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallA repeatable remediation runbook
- Freeze the baseline: record hardware, software versions, topology, rank mapping, collective, message sizes, bandwidth, latency, and training step time.
- Clear the local path: use
nvidia-smi topoandnvbandwidth; resolve peer-access, PCIe, NVLink, IOMMU, or ACS issues that apply to the platform. - Clear the fabric path: verify InfiniBand port state and rate or the RoCE link configuration; run
ib_write_bwandib_write_lat; inspect errors, retries, drops, pauses, ECN, and CNPs. - Verify GPUDirect: confirm the supported
nvidia-peermemor DMA-BUF method and compare host- and GPU-memory tests. - Correct placement: align ranks, GPUs, NICs, CPU affinity, and NUMA memory; verify container or VM topology exposure.
- Run the real collective pattern: use the production node count, collective types, and message-size distribution.
- Change one NCCL setting: investigate
NCCL_IB_QPS_PER_CONNECTION, buffer sizing, or CTA counts only when the measurements support that hypothesis. - Re-run and retain evidence: compare step time, collective wait time, bandwidth, latency tails, memory use, and congestion counters against the frozen baseline.
- Promote cautiously: keep a tuning change only if it improves representative jobs without creating new congestion, memory pressure, or instability.
What to monitor after the fix
- Per-link bandwidth and negotiated rate
- RDMA retries, symbol errors, link recovery, and link-down events
- RoCE drops, PFC pauses, ECN marks, CNPs, and queue occupancy
- GPU-to-NIC and GPU-to-GPU bandwidth for scheduled node types
- NCCL-selected topology and CPU affinity after image or driver changes
- Collective duration, tail latency, and training step-time variance
Trend these values by node, port, switch path, and job. A cluster-wide average can hide one degraded link or one incorrectly placed rank that repeatedly becomes the slowest participant.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
