Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
To diagnose a network bottleneck in distributed AI training, correlate three kinds of evidence at the same incident time: NCCL’s per-rank behavior, independent InfiniBand or RoCE measurements, and the physical GPU-to-NIC path. Start with a reproducible run, capture focused NCCL logs, run NCCL’s active diagnostics, verify PCIe/NUMA and rail placement, test the fabric outside the training job, and then compare a minimal NCCL collective with the full workload. A network is a credible bottleneck only when a path, rank, rail, node, or transport counter is measurably worse than the installed hardware baseline or a known-good run and the slowdown follows that path under controlled placement.
What does a network bottleneck look like in distributed training?
A network bottleneck is not simply a slow training step. Compute imbalance, data loading, synchronization, GPU clocks, an inefficient collective algorithm, or an application communication pattern can all increase step time without a fabric fault. Treat the network as the cause only when communication evidence and an independently measured path point to the same failure.
- Low independent RDMA bandwidth: suspect link-rate downshift, PCIe limitations, GPU-to-NIC locality, cable or optic problems, switch paths, or transport configuration before changing NCCL settings.
- Normal average bandwidth but high or unstable latency: investigate congestion, retransmissions, PFC or ECN behavior, queue drops, and intermittent link degradation. Collective operations can stall on tail latency even when average throughput appears healthy.
- One rank, rail, NIC, or node is an outlier: follow that hardware path and its process affinity. An outlier is an imbalance to isolate, not proof that every node is faulty.
- RDMA tests are healthy but NCCL is slow: inspect topology, locality, rail balance, NCCL’s selected devices, algorithm and protocol choices, and the application’s collective pattern.
- The job fails only at larger scale: find the first failing node, rank, rail count, or message size by reducing scale systematically.
Use a known-good run from the same cluster whenever possible. There is no universal bandwidth or latency number that proves a bottleneck across different NIC generations, link rates, message sizes, topologies, or collectives.
How should I establish a reproducible baseline?
Record enough context that another engineer can reproduce the slowdown and compare it with a healthy run. Capture the exact incident time so logs, endpoint counters, and switch telemetry can be aligned.
#1 Best Overall
- Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
- With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
- Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
- Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
- Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.
Record the workload and software
- Training command, framework, model or benchmark, batch and tensor shapes, and whether communication overlaps computation.
- NCCL, CUDA, driver, MPI or launcher, and network-plugin versions. Include the gIB plugin version when it is installed.
- Node count, rank-to-GPU mapping, GPU-to-NIC mapping, rail assignment, process affinity, and whether the transport is InfiniBand or RoCE.
- Incident start and end times, the first affected step, and whether the symptom is a slowdown, timeout, hang, or initialization failure.
Measure more than end-to-end step time
Save step time, collective duration, training throughput, and rank-to-rank variance. A high step time with normal collective duration points away from the network; a collective that waits on one rank or rail deserves path-level investigation. Keep the same message or batch shape when comparing runs, because communication behavior changes with size.
Keep a controlled comparison
Use one known-good run with the same software and placement if available. Change one variable at a time—such as node count, rail count, rank placement, or a transport setting—and retain the output from every trial. This prevents a tuning change from masking the original fault.
Which NCCL logs and counters should I collect?
Begin with concise per-rank logs, then increase detail only on a small reproducer. Collect every rank, not just rank zero, because a single slow rank can hold a collective while other ranks appear idle.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsStart with initialization and network evidence
- Set
NCCL_DEBUG=INFO. - Set
NCCL_DEBUG_FILEto a distinct file for each rank through your launcher or job wrapper. Do not let ranks overwrite one another. - Add
NCCL_DEBUG_SUBSYS=INIT,NETto capture communicator setup and network transport selection. - Include
GRAPHwhen topology detection or path selection is suspect. - Include
TUNINGwhen you need the selected algorithm, protocol, channel count, or thread choices.
Google Cloud’s NCCL troubleshooting guidance also recommends logs from all ranks and verification of NCCL, the gIB plugin, and their versions. Keep the launch environment and the resulting files together so version mismatches are visible.
Use verbose traces sparingly
DEBUG or TRACE output can become extremely large and can add I/O overhead. Enable it only for a short, repeatable reproducer, store it on local or high-throughput storage, and return to INFO for longer runs. A trace is useful when it answers a specific question, such as which NIC a rank selected or where a collective stopped progressing.
What to look for in the logs
- Whether all ranks selected the expected network devices and transport.
- Different channel, protocol, or algorithm choices between a healthy and slow run.
- One rank reporting delayed progress, repeated retries, or a different topology path.
- Initialization failures that occur before application work; these narrow the problem differently from a bandwidth collapse during training.
How do I run NCCL’s active diagnostics?
NCCL 2.31 and later include passive RAS checks and active diagnostics. Run them before changing tuning knobs because they exercise the communication paths directly and can separate an intra-node problem from an inter-node one.
Rank #2
- Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
- Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
- Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
- Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
- Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
Run the communication-path check
Set NCCL_RUN_DIAGNOSTICS=1 and initialize the communicator with the same node and rank layout that shows the problem. A failed peer-to-peer check narrows the issue to a GPU, interconnect, or driver path. A passing peer-to-peer check makes an intra-node culprit less likely and shifts attention toward inter-node networking or the application.
Understand the active network result
The active network check uses ib_write_bw and streams 1,000 messages of 64 KiB. It reports minimum, median, and maximum bandwidth across participating ranks. A healthy link should complete this measurement well under one second; a severely degraded link can reach the tool’s timeout. The example diagnostic flags ranks more than 30 percent from the mode median as outliers. That is a diagnostic rule for the example, not a universal performance threshold.
Save the complete output, including which ranks participated and the minimum, median, and maximum values. An aggregate average can hide a single bad rail that is sufficient to stall a collective.
How can I verify GPU, PCIe, NUMA, and NIC topology?
Distributed training performance depends on the path between each GPU and its assigned NIC, not merely on the nominal NIC speed. A GPU connected through a remote NUMA node, a narrow PCIe link, or an unintended rail can underperform even when fabric counters are clean.
Inspect peer and local paths
- Run
nvidia-smi topo -p2p nto see direct GPU peer relationships. - Use
nvbandwidthornvloomto measure local and multinode communication paths. - Record each GPU’s NIC locality, PCIe generation and link width, NUMA relationship, assigned rail, and process affinity.
Check for imbalance
Compare all GPUs and NICs rather than inspecting only the first device. Look for one GPU with a different PCIe path, one NIC carrying more traffic, a rank assigned to a remote NUMA node, or a rail that consistently reports lower bandwidth. Move the rank or rail assignment in a controlled test: if the slowdown follows the hardware path, investigate that path; if it follows the workload placement, inspect affinity and configuration.
Free tools Windows power users keep installed
One-click scans. No signup required.
Topology output should be kept with NCCL logs. NCCL can select a different network device or channel arrangement after a hardware, driver, or topology change, so a topology snapshot from a healthy run is valuable evidence.
Rank #3
- 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
- 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
- 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
- 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
- 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.
How do I test an InfiniBand fabric independently?
Test the fabric without the training application so NCCL and framework behavior cannot obscure a link or transport problem. Run tests between representative node pairs, then repeat on any node or rail identified as an outlier.
Check port state first
Use ibstatus or ibstat. Ports should be Active, physically LinkUp, on the expected link layer, and operating at the expected rate. A port that is up at a lower rate is already a capacity problem even if communication succeeds.
Measure bandwidth and latency
ib_write_bwmeasures endpoint bandwidth. Compare representative node pairs and, when GPUDirect RDMA is involved, compare host-memory and GPU-memory tests.ib_write_latmeasures latency. Use it when stalls, synchronization delays, or tail latency are suspected rather than relying only on throughput.ibpingchecks basic connectivity between InfiniBand endpoints.
Compare each result with the expected rate for the installed hardware and a known-good pair. Do not apply a cross-platform bandwidth target to a different link generation, message size, or topology.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Inspect fabric health counters
Use perfquery for symbol errors, link-error recovery, and link-down events. When the problem appears broader than one endpoint, ibdiagnet can provide topology and routing reports. Rising errors or link transitions during the same time window as the training slowdown strengthen a physical-path hypothesis.
How do I diagnose RoCE congestion and packet loss?
RoCE requires endpoint and switch evidence because congestion can reduce collective performance without taking a link completely down.
Read NIC and switch counters
Inspect ethtool -S and switch counters for errors, drops, pause frames, Priority Flow Control (PFC), Explicit Congestion Notification (ECN), CNP packets, and queue drops. Correlate counter changes with the incident timestamp and affected rail. A counter that rises only on an unused interface is not an explanation for the job’s slowdown.
Rank #4
- 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
- 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
- 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
- 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
- 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.
Check RDMA transport statistics
Use rdma statistic and look for retransmissions or timeouts. Increasing retransmission or timeout counters support a transport-loss or congestion hypothesis, especially when latency becomes unstable. Pair those counters with NCCL rank logs; a transport symptom without a corresponding affected path may be unrelated background traffic.
Separate congestion from link degradation
Stable link rate with queue drops, PFC or ECN activity points toward congestion or configuration. Link errors, downshifts, or intermittent carrier events point toward the physical link, optic, cable, or port. Test the suspect rail independently before changing NCCL protocols.
What does each test result mean?
| Observation | Most useful next investigation |
|---|---|
| Low independent RDMA bandwidth | Verify link rate, PCIe generation and width, GPU-to-NIC locality, cable or optic health, switch path, and transport configuration. |
| Normal bandwidth with high or unstable latency | Inspect congestion, retransmissions, PFC or ECN behavior, queue drops, and intermittent link events; examine latency tails rather than only averages. |
| One rank, rail, NIC, or node is an outlier | Trace its hardware path, NUMA and process affinity, rail assignment, and topology placement. Re-run after moving the rank or rail. |
| RDMA tests healthy but NCCL is slow | Compare NCCL-selected devices, topology, rail balance, algorithm and protocol choices, channel count, and the application collective pattern. |
| Failure appears only at larger scale | Reduce nodes, ranks, or rails to find the first failing scale, then compare synchronized NCCL and switch evidence at that boundary. |
| P2P diagnostic fails | Focus first on the GPU, local interconnect, PCIe path, or driver; do not begin with a fabric-wide tuning change. |
| P2P passes but inter-node check is poor | Shift attention to NIC locality, rail assignment, InfiniBand or RoCE transport, and the switch path. |
Why does distributed training hang only at scale?
A scale-only hang often exposes an imbalance or a path that smaller jobs never use. More ranks can select additional rails, traverse different switch routes, increase synchronization sensitivity, or expose a single weak node.
Find the first failing scale
- Run the same minimal reproducer on a small known-good node set.
- Add nodes or ranks in controlled increments until the first slowdown or hang appears.
- At that boundary, vary one dimension—node membership, rank count, rail count, or placement—to identify the failing path.
- Keep the same message sizes and collect logs from every rank in the failing and immediately smaller runs.
If the first failing run always includes one node, test that node in a smaller job. If failure follows a rail count or rank placement, inspect affinity and device selection. If no single node explains it, compare switch and transport counters from the same time window; scale can turn mild congestion into collective-wide waiting.
Distinguish an application-scale issue
A minimal NCCL test that passes while training hangs means the application differs in collective pattern, tensor sizes, overlap, version, placement, or configuration. Compare those variables one at a time. A minimal test that also fails is infrastructure evidence; continue down the physical or transport path rather than attributing the hang to the model.
How should I compare two remediation options?
Evaluate a hardware, topology, placement, or configuration change against the same baseline. The fastest-looking change is not necessarily the safest if it improves one rail while making another less predictable.
Best Value
- 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
- 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
- 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
- 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
- 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
| Comparison axis | What to measure | Decision question |
|---|---|---|
| Endpoint performance | Independent bandwidth and latency for the affected GPU, NIC, rail, and representative node pairs | Did the suspect path improve, including latency tails? |
| Transport health | Errors, retransmissions, timeouts, drops, pause, PFC, ECN, CNP, and queue counters | Did the evidence of loss or congestion disappear? |
| Locality | GPU-to-NIC mapping, PCIe generation and width, NUMA relationship, and process affinity | Is traffic using the intended local path? |
| Rail balance | Per-rail traffic and per-rank NCCL bandwidth | Did one rail remain an outlier? |
| Collective behavior | NCCL collective throughput, duration, and tail variance | Did the real workload improve, not just the microbenchmark? |
| Reproducibility | Repeated runs across nodes and at the failing scale | Does the result persist beyond one favorable run? |
| Operational risk | Driver, firmware, topology, configuration, and rollback impact | Can the change be isolated, monitored, and reversed? |
Change one setting at a time and retain both the pre-change and post-change evidence. A remediation is credible when the affected metric improves, the collective result improves, and the improvement follows the intended path across repeated runs.
What should a complete incident evidence bundle contain?
Assemble the bundle before clearing logs or rebooting a node. The goal is to let another engineer correlate software, hardware, and fabric behavior without repeating the incident.
- Incident timestamps, training command, framework and NCCL versions, transport type, node and rank list, and GPU/NIC/rail mapping.
- Per-rank NCCL
INFOlogs, plus focusedINIT,NET,GRAPH, orTUNINGoutput when used. - NCCL active diagnostic output, including minimum, median, maximum, and outlier ranks.
nvidia-smi topo -p2p noutput and relevantnvbandwidthornvloomresults.- InfiniBand
ibstatus/ibstat,ib_write_bw,ib_write_lat, and applicableperfquery,ibping, oribdiagnetresults. - RoCE
ethtool -S, switch counters, andrdma statisticoutput when that transport is used. - Step time, collective time, throughput, rank variance, and the matching known-good comparison.
Synchronize the timestamps across nodes as closely as your environment permits. A counter sampled hours after the hang cannot establish what happened during the collective.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →What is the shortest reliable diagnostic sequence?
When time is limited, preserve the evidence-first order rather than jumping directly to NCCL tuning.
- Reproduce the issue and record the exact time, placement, software versions, step time, and collective duration.
- Capture per-rank
NCCL_DEBUG=INFOlogs withNCCL_DEBUG_SUBSYS=INIT,NET; addGRAPHorTUNINGonly for a focused question. - Run
NCCL_RUN_DIAGNOSTICS=1and save the active network result. - Verify GPU-to-NIC, PCIe, NUMA, rail, and rank topology.
- Run the appropriate InfiniBand or RoCE endpoint tests and inspect transport counters.
- Compare a minimal NCCL test with the real job, reducing scale if necessary to locate the first failing boundary.
- Change one variable, repeat the measurements, and keep the baseline for rollback and comparison.
This sequence identifies whether the limiting factor is a physical path, transport, topology, NCCL choice, or application behavior before a configuration change obscures the diagnosis.
Frequently Asked Questions
Can a healthy bandwidth test still leave training slow?
Yes. A point-to-point test may use different message sizes, routes, memory types, or concurrency than the model’s collectives. Check NCCL collective duration and rank variance with the same placement and workload shape before declaring the fabric healthy.
Should I run NCCL TRACE on every production rank?
Usually not. TRACE can generate very large files and add storage overhead, which can perturb a long run. Capture it on a short reproducer after INFO-level logs have identified the specific operation or rank that needs more detail.
How do I tell whether the problem is a bad node or a bad switch path?
Repeat the same endpoint pair with the suspect node, then exchange one endpoint at a time while preserving message size and placement. A symptom that follows one node or rail points toward its local path; a symptom shared by multiple pairs that traverse a common route warrants switch and fabric-counter review.
What should I do after replacing a cable or optic?
Recheck port state and negotiated rate, clear or baseline the relevant error counters, and rerun the same endpoint and NCCL measurements. Confirm improvement over repeated runs at the scale that originally failed instead of relying on a single successful launch.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
