Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A slow GPU all-reduce is a symptom, not a diagnosis. Isolate the problem in layers: establish a repeatable baseline, measure the collective, test intra-node GPU paths, test GPU-to-NIC and inter-node fabric, inspect NCCL’s selected topology, and profile the application timeline. Change NCCL settings only after those measurements agree that the underlying paths are healthy.
Why is my GPU all-reduce slow?
An all-reduce can be delayed by almost any part of a distributed training system. The limiting work may be GPU-to-GPU communication inside a node, PCIe or NVLink placement, the path from a GPU to a network adapter, the inter-node fabric, collective synchronization, rank placement, host-side scheduling, or contention with computation. The same symptom—high all-reduce time—can therefore have very different fixes.
Do not diagnose a cluster from a single bandwidth number. A useful comparison identifies the path, traffic shape, scope, and evidence behind the measurement:
Recommended Free Tools
- Path: NVLink, PCIe peer-to-peer, GPU-to-NIC, InfiniBand or RoCE, or a multi-node NVLink domain.
- Scope: one GPU pair, a node pair, one network rail, the full collective, or the complete application.
- Traffic shape: message size, direction, concurrency, and whether computation overlaps communication.
- Evidence: expected topology, measured bandwidth or latency, NCCL’s selected transport, and an application trace.
- Repeatability: the same nodes and workload compared with a known-good peer or historical baseline.
There is no universal “good” all-reduce result. Hardware generation, topology, message size, collective algorithm, rank mapping, and overlap with computation all change the result. A hardware link’s advertised peak is not a guaranteed all-reduce throughput.
#1 Best Overall
- The 25Gb dual-port SFP+ network card is based on the Mellanox ConnectX-5 Ex controller, which provide the highest performing and most flexible interconnect solution.
- Technical Support:PXE、 RDMA、UEFI、SR-IOV、1588 PTP、Jumbo Frames（9.5KB）
- Windows 10/11、Windows Server 2016/2019/2022、Deepin 15.11/20/20.6/20.9、VMware ESXi 6.5/6.7、Ubuntu 18.04.5/20.04.1、Ubuntu 22.04.2/22.04.3、RHEL/CentOS 7.6/7.9/8.2/8.3、ZTE New Fulcrum 3.2.2/5.0.5、SUSE 12.5/15.4、FreeBSD 13.2、NeoKylin 7.6、OpenKylin 0.7.5、Mikrotik、iKuai route、Galaxy Kylin v10、Zhongke Fangde desktop OS、Zhongke Fangde server OS、Tongxin UOS 20、Emind OS
- install the operating system with its driver CD, or download it from the official website. Includes low-profile and full-height stands to support standard and ultra-thin computers/servers.
- Enjoy 24/7 customer service, 30-day free returns, 1-year free warranty, and lifetime technical support for your peace of mind.
1. Establish a reproducible baseline before changing anything
Write down the environment for every test. A result is useful only when you can reproduce it and compare it with an equivalent result.
Record the complete test context
- NCCL, CUDA, GPU driver, framework, and network software versions.
- GPU model, count per node, and the output of
nvidia-smi. - Network adapter inventory, interface names, rail count, and node hostnames.
- Node count, ranks per node, rank-to-GPU mapping, and process-launch command.
- Message sizes, number of iterations, warm-up behavior, and any algorithm or protocol variables.
- Whether the workload is bare-metal, containerized, virtualized, or running under a scheduler.
- The exact nodes used and a known-good result from the same hardware and topology, if one exists.
Run the same test several times and keep the distribution, not just the fastest run. Record median and outlier behavior, because intermittent contention or a late rank may be more important than the average.
Keep production and diagnostic configurations separate
Do not begin by copying a collection of environment variables from an unrelated cluster. Save the current environment, launcher options, container image, and scheduler allocation. If a later experiment improves performance, you need to know which single change produced it and whether it helps the real application rather than only a microbenchmark.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches2. Measure the collective at the same scale as the problem
Run an NCCL collective benchmark such as all_reduce_perf with the relevant number of GPUs, ranks, nodes, and message sizes. A one-node, two-GPU test cannot validate a problem that occurs only across eight nodes. Conversely, a large multi-node test can hide a faulty local GPU pair behind other traffic.
Use a message-size sweep
Small messages are dominated by latency, kernel-launch overhead, host scheduling, and synchronization. Large messages expose sustained bandwidth and often make a bad link easier to see. Test a range rather than selecting one convenient size. Record both the reported bus bandwidth and the time per operation, and use the same iteration and warm-up settings for every comparison.
NVIDIA recommends testing network performance before production and retaining a baseline because achievable performance depends on topology. Treat the benchmark as a comparison tool, not as a universal pass/fail threshold.
Retest suspicious node pairs
If one pair of nodes is slower, run that pair repeatedly and compare it with several other pairs using the same allocation and launch settings. A slow result that follows one node suggests a host, adapter, driver, or local topology issue. A result that appears only with one peer suggests a link, switch path, rail, or cabling problem. A result that changes with rank placement points toward mapping or topology selection.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Interpret a healthy benchmark correctly
If an equivalent collective benchmark is normal while the application is slow, the network is not automatically exonerated and the application is not automatically at fault. The application may expose communication because compute no longer overlaps it, wait for a late rank, perform extra host work, or use a different rank mapping. Move to an application timeline instead of repeatedly tuning NCCL.
3. Inspect the intra-node GPU paths
Before testing the fabric, verify that GPUs inside each node can communicate over the paths you expect. A topology mismatch can make every inter-node collective slower because each rank must first move data across a weak local connection.
Read the topology matrix
Run:
nvidia-smi topo -mThis shows the relative placement of GPUs, CPUs, and network adapters. Look for GPUs that share a high-quality NVLink route, GPUs connected only through PCIe, and adapters attached to a different CPU or PCIe root complex than the GPUs that use them. Compare the output with the rank-to-device mapping used by the launcher.
Check peer-to-peer capability
Inspect PCIe and NVLink peer-to-peer status with:
nvidia-smi topo -p2p p
nvidia-smi topo -p2p nThe first command helps examine PCIe peer access and the second checks NVLink-related peer access. A missing or unexpected P2P relationship is a lead, not a complete diagnosis. The matrix can report that a relationship exists while actual bandwidth is poor, and it does not prove that the application is using that path.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Rank #2
- Host Interface: PCI Express 5.0 x16
- Total Number of Ports: 1
- Expansion Slot Type: OSFP
- Media Type Supported: Optical Fiber
- Maximum Data Transfer Rate: 400 Gbit/s
Measure actual pairwise bandwidth
Use nvbandwidth to measure the GPU pairs that NCCL is likely to use. Compare the resulting bandwidth matrix with the topology output and a known-good node. Test both directions where the tool supports directed reads and writes. An asymmetric result can identify a particular bridge, link, or access direction even when an aggregate test looks acceptable.
Investigate PCIe, IOMMU, and ACS carefully
NVIDIA documents Linux bare-metal IOMMU translation and PCIe Access Control Services as possible causes of GPU P2P and GPUDirect problems. If P2P is absent or unexpectedly slow, inspect PCIe bridge placement and the platform’s IOMMU and ACS configuration. Do not change firmware or kernel settings simply because a suggested command appears in a forum post: those settings can affect isolation, device assignment, and the operational requirements of the host.
Use NCCL’s active P2P diagnostics
NCCL’s built-in active diagnostics can test the P2P links it expects to use. A failed check identifies affected GPU pairs and paths, which narrows the search. A passing check makes that local P2P path less likely to be the primary cause; it does not prove that the full collective, NIC path, or application schedule is healthy.
4. Verify the GPU-to-NIC and inter-node path
When local GPU tests are healthy, examine how data leaves the node. A fast NIC cannot compensate for a poor GPU-to-NIC path, and a healthy GPU path cannot compensate for a congested or misconfigured fabric.
Confirm the intended network interface
Read the NCCL log to see which network interface, physical device, and transport it selected. Do not assume that the interface name in your launcher is the interface NCCL actually uses. Check that all ranks see consistent device names and that the selected rail matches the topology and the scheduler allocation.
Check GPUDirect RDMA support
For GPU Direct RDMA, verify that the adapter, driver stack, and kernel support the path available on your system. NVIDIA documents nvidia-peermem as one supported module path; recent Linux and DMA-BUF configurations may use DMA-BUF instead. The correct mechanism depends on the kernel, driver, and deployment, so confirm it in the environment rather than loading both indiscriminately.
Measure the fabric independently
Use the low-level tests appropriate for the fabric:
ib_write_bwmeasures InfiniBand write bandwidth.ib_write_latmeasures write latency.- Multi-node
nvbandwidthcan measure GPU-aware transfers where supported. - A matching
nvloomsuite can reproduce a specific many-GPU traffic pattern.
NCCL’s network diagnostic uses ib_write_bw over selected physical InfiniBand devices when the tool is installed on participating nodes and hostnames resolve. Test the same node pairs, rails, and device selections that the collective uses.
Compare rails and peers
Retest a slow node pair against different peers. If one rail is consistently slower, compare its adapter, port, switch path, and cabling with a healthy rail. If only a particular peer is slow, investigate the shared fabric path. If all low-level tests are healthy but NCCL is slow, return to rank placement, software consistency, and NCCL’s selected topology rather than replacing hardware immediately.
Account for multi-node NVLink domains
Multi-node NVLink configurations have their own domain and service checks in NCCL’s troubleshooting guidance. Verify that the required services and domain configuration are consistent across participating nodes before interpreting an inter-node result as ordinary InfiniBand or RoCE traffic.
5. Use logs to see what NCCL actually selected
NCCL diagnostics can expose rank differences, transport selection, channel construction, and the paths NCCL expects to use. Capture logs from every rank, not only rank zero. A single rank may select a different interface or report a different topology relationship.
Look for differences, not just errors
- Different network interfaces or physical devices selected by different ranks.
- Unexpected transport fallbacks between GPUs, NICs, or nodes.
- Ranks assigned to GPUs that do not match the intended topology.
- One rank entering a collective later or reporting a different channel configuration.
- Warnings about missing peer access, unavailable modules, or failed diagnostics.
A clean log does not establish performance. It only tells you that initialization completed without the reported error. Pair the log with bandwidth measurements and a trace.
6. Profile the application timeline
When the application and an equivalent collective benchmark disagree, capture a full multi-rank timeline with Nsight Systems. Trace NCCL activity in the context of GPU kernels, CPU work, storage, and synchronization.
Distinguish communication time from waiting time
A long NCCL interval may represent data transfer, but it may also include time waiting for a late rank. Compare the start times of the same collective across ranks. If seven ranks reach all-reduce promptly and one arrives late, increasing network bandwidth will not remove the delay.
Check overlap with computation
Communication can be hidden behind computation in one configuration and exposed in another. Look for changes in stream ordering, bucket sizes, gradient readiness, or kernel scheduling. A benchmark that runs communication alone may be fast while the application becomes slow when compute, communication, host callbacks, and data loading contend for resources.
Include all ranks and the host timeline
Incomplete traces can obscure collective relationships. Capture every participating rank and enough CPU context to identify launch gaps, storage delays, process pauses, and scheduler effects. NVIDIA recommends profiling to identify CPU, storage, and GPU bottlenecks rather than assuming every delay is a network problem.
7. Separate synchronization failures from bandwidth limits
A collective hang, timeout, or highly uneven rank timing is not automatically a low-bandwidth condition. Ranks must enter compatible collectives in compatible order, and each rank must use the intended device.
Enable PyTorch distributed diagnostics
For PyTorch jobs, set:
TORCH_DISTRIBUTED_DEBUG=DETAIL
TORCH_SHOW_CPP_STACKTRACES=1These settings can provide additional detail when collective desynchronization is detected. Use the output to check whether ranks called different collectives, called them in a different order, or selected inconsistent devices.
Verify rank and device placement
- Confirm that each process owns the GPU assigned by its local rank.
- Check that world size, node count, and ranks per node agree across processes.
- Ensure that all ranks execute the same collective sequence.
- Compare the launcher’s mapping with
nvidia-smi topo -m. - Capture the full multi-rank context when a timeout occurs.
If the trace shows a late rank, fix the preceding computation, data loading, synchronization, or process scheduling before changing network parameters.
8. Match the diagnostic to the failure pattern
NVIDIA’s DCGM guidance emphasizes that a test proves only what it exercised. “Completion without a reported workload error is not, by itself, a performance qualification.” Use the narrowest test that answers the question, then validate the result against a known-good system or a site requirement.
| Observed pattern | Useful diagnostic | What it can and cannot establish |
|---|---|---|
| Slow only when computation and communication run together | mnubergemm |
Reproduces a combined workload; it does not isolate one transfer path or prove that the fabric is responsible. |
| A particular GPU pair or direction is suspicious | mnnvbandwidth |
Measures directed reads and writes and produces a bandwidth matrix; compare it with a known-good node. |
| A particular many-GPU traffic shape is implicated | A matching nvloom suite |
Tests the selected traffic pattern; a different pattern may expose different contention. |
| Inter-node network path is suspected | ib_write_bw, ib_write_lat, and a matching multi-node GPU test |
Separates fabric bandwidth and latency from the full collective; it does not reproduce all application scheduling. |
A broad test can reproduce a condition without identifying its source. A narrowly targeted test can miss contention that appears only in the full application. Use both when the symptom changes under load.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.9. Tune NCCL only after the path is healthy
NCCL is tuned across many systems, so production tuning is generally discouraged until topology, drivers, rank placement, and fabric tests are consistent. A setting that improves one benchmark can regress another workload.
Rank #4
- DUAL-PROTOCOL 100G: ConnectX-4 VPI (MCX456A-ECAT) runs EDR InfiniBand 100Gb/s or 100GbE per QSFP28 port with 100G/50G/40G/25G/10G auto-negotiation — one card serves IB and Ethernet fabrics.
- PCIe 3.0 x16, FULL BANDWIDTH: Dual ports sustain line-rate 100Gb/s each for HPC, AI training nodes and high-throughput storage fabrics.
- RDMA WITHOUT CPU COPIES: Native InfiniBand RDMA plus RoCE accelerate MPI, NVMe-oF and distributed storage; hardware offloads cut latency and free CPU cycles.
- HEAVY VIRTUALIZATION: SR-IOV with up to 127 VFs per port (254 per card) plus VXLAN/GENEVE/NVGRE overlay offload for multi-tenant clouds and dense VM hosts.
- DATA CENTER FEATURES: PXE/UEFI boot, NC-SI management, DCB, jumbo frames; Linux (MLNX_OFED), Windows (WinOF) and VMware ESXi support; brackets for any chassis.
Run controlled experiments
- Save the baseline command, environment, logs, and application trace.
- Change one NCCL variable at a time.
- Repeat the collective benchmark at the original message sizes.
- Repeat the real application, including its normal compute and communication overlap.
- Check median, tail, and memory use rather than accepting one faster run.
- Revert the change if the gain does not reproduce across the relevant nodes and workloads.
Understand the trade-offs
NCCL documents tuning variables such as CTA counts and buffer sizes. Buffer changes can trade GPU memory for throughput and can behave differently on PCIe topologies. More parallelism is not automatically better: it can increase contention, reduce compute overlap, or expose a weak bridge. Keep a per-workload record of the result instead of treating a variable as a cluster-wide rule.
Common wrong turns and their fixes
Changing NCCL variables first
Why it fails: tuning can mask a bad rail, missing P2P access, or a late rank and makes the original baseline harder to recover.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Better approach: capture topology, pairwise bandwidth, fabric tests, logs, and a timeline first.
Comparing unrelated benchmarks
Why it fails: different message sizes, rank counts, algorithms, warm-up behavior, or overlap produce different numbers.
Better approach: compare identical commands on the same topology, then compare with a known-good peer.
Assuming a successful test proves performance
Why it fails: a test may not exercise the problematic direction, pair, rail, or traffic shape.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallBetter approach: choose a diagnostic that matches the failure pattern and compare its measurement with a requirement or baseline.
Reading only rank-zero logs
Why it fails: one rank can select a different interface, arrive late, or report a different path.
Better approach: collect and compare logs from all ranks.
Replacing hardware before isolating the path
Why it fails: a new adapter cannot fix a bad GPU-to-PCIe route, rank mapping, or synchronization bug.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteBetter approach: use independent GPU-pair and fabric tests to establish which layer is below expectation before considering a hardware change.
A practical decision tree
- Is the slowdown reproducible? If not, record repeated runs and investigate contention, scheduling, and outliers.
- Is the equivalent
all_reduce_perfslow? If yes, continue with path isolation. If no, profile the application. - Are GPU P2P and
nvbandwidthresults abnormal? Inspect topology, PCIe bridges, IOMMU/ACS, and NCCL P2P diagnostics. - Are local paths healthy but fabric tests abnormal? Check adapters, drivers, GPUDirect support, rails, peer nodes, and switch paths.
- Are low-level tests healthy but NCCL abnormal? Compare rank placement, selected interfaces, software versions, and multi-node topology.
- Are ranks entering at different times? Investigate application scheduling and collective order before bandwidth tuning.
- Does one controlled NCCL change help both benchmark and application? Keep it only if the improvement reproduces and does not create a memory or stability regression.
Field checklist
- Baseline versions, inventory, topology, mapping, message sizes, and launch configuration.
- Run the collective benchmark at the actual node and rank scale.
- Sweep small and large messages and retain repeated-run statistics.
- Inspect
nvidia-smi topo -mand P2P status. - Measure GPU pairs with
nvbandwidth. - Check NCCL’s selected interfaces and transports in every rank’s log.
- Test the fabric with
ib_write_bwandib_write_latwhere applicable. - Compare slow node pairs with different peers and rails.
- Profile all ranks with Nsight Systems when the application and microbenchmark disagree.
- Use PyTorch distributed debug settings for desynchronization or timeout symptoms.
- Use DCGM diagnostics that match the direction, pair, or traffic shape in question.
- Change one NCCL setting at a time, and validate it on the real workload.
What a defensible diagnosis looks like
A strong diagnosis connects the symptom to a measured layer: for example, a specific GPU pair has lower directed bandwidth than its peers, the topology places those GPUs across an unexpected PCIe bridge, NCCL selects that path for the affected ranks, and the application trace shows the corresponding collective delay. Alternatively, the low-level tests may be normal while one rank reaches all-reduce late, demonstrating a synchronization or scheduling issue rather than a fabric limit.
Document the evidence, the exact commands, the nodes tested, and the conditions under which the result changed. Documentation and hardware compatibility can change, so verify command behavior against the NCCL, CUDA, driver, framework, and operating-system versions installed in the target environment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
