iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Use a layered, evidence-first workflow: capture the failing NCCL run, verify the transport and plugin it selected, test GPU-local and node-to-node paths separately, then correlate latency and throughput with NIC and switch counters. Follow the InfiniBand or RoCE branch that matches your fabric. Change NCCL settings only after the path and fabric have been checked.
What congestion and packet loss do to a training job
Distributed training depends on repeated collective operations such as all-reduce. A damaged link, queue drop, pause storm, or incorrect transport choice can therefore appear as a slow step, an intermittent hang, a transport error, or a job that completes only after retries. The impact is workload-dependent: Juniper’s Introduction to Congestion Control in Juniper AI/ML Networks states that “Packet loss can significantly impact training completion times,” but the guide does not establish one percentage that applies to every model or cluster.
Do not treat every slow NCCL run as a network fault. The same symptom can come from an intra-node GPU peer path, a host NIC, a PCIe path, a plugin mismatch, CPU or memory affinity, or application behavior. The objective is to identify the first layer whose measurements diverge from a healthy baseline.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsBuild an evidence set before changing anything
Run the failing workload again if it is safe to do so, and preserve the configuration that produced the symptom. A useful case file contains:
#1 Best Overall
- 【Fiber Optic Cleaning Kit】 The fiber optic cleaning swab effectively cleans multiple surfaces. This swab functions as a cleaner for SC, FC, ST, APC, or LC fiber optics, as well as optical connectors. Capable of over 500 reusable cleanings, its ergonomic design ensures comfortable handling. Each swab provides a single-use cleaning solution, removing even stubborn contaminants in one pass.
- 【Stable Fiber Cleaning 800+】Our fiber optic cleaning pen delivers consistent anti-static cleaning performance, supporting over 800 cleaning cycles for lower per-use costs. Our LC cleaning pen thoroughly removes contaminants from fiber surfaces, helping you get the job done.
- 【Premium Exterior Design】This fiber optic cleaning pen seamlessly combines convenience with efficiency. Designed with the familiar shape of a standard pen, it offers comfortable grip and effortless handling during cleaning.This fiber optic cleaning pen is easy to operate. Its unique “click” confirmation sound signals completion of the cleaning process, ensuring thorough and efficient cleaning every time.
- 【Stable Cleaning】This cleaning kit effectively removes dust, grease, and other debris from fiber optic port surfaces without scratching the surface.
- 【Wide Application】This LC fiber optic cleaning set is the ideal choice for fiber optic cleaning. This highly effective cleaning pen is suitable for LC/SC/FC/ST/MU/MTP/MPO/D4/DIN connectors
- NCCL logs from every participating node and rank.
- NCCL, network-plugin, GPU-driver, NIC-driver, and relevant switch or fabric software versions.
- Host names, NIC names, link layer, negotiated rate, and the rails or interfaces selected by the job.
- Start and end times for the reproduction, so counter deltas can be compared with application latency.
- The exact command line, container image, environment variables, CPU and memory affinity, and any recent topology or firmware change.
Record counters before the test and immediately afterward. A static nonzero counter is a clue, not proof of the current failure; a counter that rises while latency tails or throughput instability appear is much stronger evidence.
Step 1: capture NCCL’s actual path and decisions
Save diagnostic output to files instead of relying on a console scrollback buffer. For a reproduction, set the following variables (the file path must be unique per VM and rank):
NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS=INIT,ENV,GRAPH,NET,COLL,TUNING
NCCL_DEBUG_FILE=/path/to/logs/nccl_%h_%p.logThese settings are documented in Google Cloud’s gIB guidance for its AI Hypercomputer environment. Other clouds and on-premises clusters may use different plugin packages or log conventions, so apply the equivalent instructions for the deployed platform.
Read the resulting logs for:
- The network interface, rail, and transport selected for each rank.
- Whether NCCL loaded the expected network plugin or fell back to a built-in Socket or InfiniBand transport.
- Environment variables that were accepted, ignored, or rejected.
- Topology and graph decisions that differ between nodes.
- Errors during communicator initialization, collective operations, or connection setup.
A plugin can be installed on one image but absent from another, or a path can be administratively up yet unable to communicate. A Google Cloud example shows that missing gIB plugin support can result in use of a built-in Socket or IB transport instead. That fallback must be established from the log, not assumed from the host’s interface state.
Step 2: separate local GPU paths from the inter-node fabric
Run NCCL’s built-in checks where supported
NCCL documentation describes passive RAS readiness checks and active communication checks. Active checks run during communicator initialization and report findings; they are informational and do not abort initialization. Use the P2P check to identify failed GPU peer paths within a host. A failed local path points toward GPU topology, PCIe or NVLink connectivity, or device configuration rather than a switch queue.
The built-in network bandwidth check runs ib_write_bw over selected InfiniBand devices when its prerequisites are met. Treat its output as a baseline for the selected device and path, not as a universal benchmark for every collective or model.
Rank #2
- 【Multi Testers Tool Sets】This fiber optic termination tool kit includes OPM, visual fault locator, fiber cleaning pen, fiber cleaner cassette, fiber cleaver, fiber stripper, Kevlar scissors, FC to LC adapter, cleaning core, liquid dispenser, fiber cleaning swabs, fiber cleaner wipes, fiber optic tool bag. This fiber optic tool kit is used for fiber stripping, cleaving, cleaning and testing. It provides the fiber optic tools you need to assemble mechanical splices and delivers greater work efficiency.
- 【Fiber Cleaning Kit】1.25mm & 2.5mm fiber optic cleaner pen & fiber optic cleaning box are used to clean LC/MU/SC/FC/ST/MTP/MPO/D4/DIN ferrule end face. The fiber cleaner can effectively clean stains and prevent dust from adhering to the ferrule again after cleaning.
- 【Visual Fault Locator Fiber Optic】Our fiber tester can intuitively and accurately detect and locate fiber breaks, poor connections, bends or cracks. It is designed for SC, FC, LC, ST interfaces both in circle and square shape of different fiber optic cables. The fiber optic tester has strong output power.
- 【Fiber Optic Meter with FC to LC Adapter】The fiber optic cable tester can measure 10 standdard wavelengths 850/980/1270/1300/1310/1490/1550/1577/1625/1650nm with a test range of -70dBm~+10dBm. The measurement error is less than ±0.2dB. Support simultaneous display of linear mW and nonlinear index dBm.
- 【Fiber Optic Cleaver】3-in-1 high precision fiber cleaver is used to cut bare fiber (0.25mm), pigtail (0.9mm) and leather cable. It has 16 cutting points, the cutting angle is about 0.5 degree, and it can cut about 36000 times.
Interpret the split
- If a GPU peer check fails, investigate the local topology and device configuration before examining multi-hop routing.
- If local P2P checks pass but node-to-node tests fail, vary by rail, or show errors, focus on the NIC, link, switch path, and fabric.
- If local and inter-node baselines are healthy while the workload is slow, inspect transport/plugin selection, software versions, CPU and memory affinity, and workload-specific behavior.
Step 3: establish a host-to-host baseline
First verify that the intended NIC ports are administratively and operationally up, have the expected link layer and negotiated rate, and are reachable from the intended peers. Then measure both bandwidth and latency between representative nodes. On InfiniBand, the commonly used low-level tests are:
Recommended Free Tools
ib_write_bw
ib_write_latRun the client and server according to the versions installed on your hosts, selecting the same devices and ports that NCCL is expected to use. A bandwidth result can look normal while latency tails are unacceptable, so collect both measurements and repeat them during the time window in which the training job reproduces its problem.
NVIDIA documents rdma statistic for RDMA retry or loss-like signals and mlxlink for physical-link health on supported hardware. The command syntax and available fields vary by driver and adapter; use the version-specific documentation for the installed tools.
| Observation | Most useful next question |
|---|---|
| Low or unstable bandwidth between one node pair | Does the affected NIC, cable, switch port, or rail show errors or a lower negotiated rate? |
| Normal average bandwidth but long latency tails | Do pause, ECN, CNP, queue-drop, retry, or timeout counters rise during the tail events? |
| All endpoint tests healthy, NCCL slow | Did NCCL choose the expected transport, plugin, interface, and topology? |
| Only one rail or path is affected | Does its physical link and switch route differ from the healthy rail? |
InfiniBand troubleshooting branch
Confirm state, rate, and link layer
Use ibstatus or ibstat to verify that each intended port is active, reports the expected link layer, and has the expected rate. Compare every participating host and rail; one port in a degraded state can make a multi-rail job appear intermittently slow.
Watch RDMA and port-error counters during reproduction
Use rdma statistic and look for counters that increase while the workload is failing, including:
rnr_nak_retry_errpacket_seq_errimplied_nak_seq_errlocal_ack_timeout_err
For port-level evidence, perfquery -x <lid> can expose relevant counters. NVIDIA identifies SymbolErrorCounter, LinkErrorRecoveryCounter, and LinkDownedCounter as examples. Capture values before and after the test and compare them with endpoint latency and switch telemetry. A growing counter narrows the search; it does not by itself identify the failed component.
Rank #3
- 【Comprehensive 13 in 1 cleaning kit】This set includes 13 professional cleaning tools that can meet the cleaning needs of various fiber optic connectors. Not only for daily maintenance but also for troubleshooting, this set of tools can provide solutions to ensure that your work is twice as efficient.
- 【400x high-definition microscope for precise detection】Built in high-precision 400x magnification fiber optic microscope allows you to clearly observe small scratches, dust, and contamination on the fiber end face. Visually inspect the cleaning effect to ensure the quality of the connection and reduce signal loss.
- 【Widely compatible with multiple interface types】Compatible with fiber optic interfaces such as SC, FC, ST, LC, etc. Not only 2.5mm (SC/FC/ST) and 1.25mm (LC) ceramic plugs, but even precision MPO/MTP multi-core connectors, this set has corresponding tools for efficient cleaning.
- 【Efficient "one click" cleaning pen, convenient and fast】Equipped with a one click quick cleaning pen, just press lightly to complete an efficient cleaning. The operation is simple and fast, which is very suitable for frequent use during installation, testing, and maintenance, greatly improving work efficiency.
- 【Professional cleaning tools to protect precision equipment】Equipped with professional tools such as fiber optic wiping clips and spare wiping cloths, it can effectively remove stubborn stains and oil stains, while avoiding secondary pollution or physical damage to precision ceramic plugs.
Check the Subnet Manager and endpoint reachability
Run sminfo to confirm that a Subnet Manager is present. If routing or topology is suspect, use ibping for endpoint connectivity and, where authorized, ibdiagnet for broader topology, routing, and fabric-error diagnostics. These tools can reveal a path or fabric-wide issue that a single host’s counters cannot.
What an InfiniBand result means
- Inactive ports, unexpected rates, or link-down events point to port configuration, optics or cabling, firmware, or switch-port health.
- Rising retry, sequence, or timeout counters with endpoint degradation point toward a transport or fabric reliability problem that needs fabric-owner investigation.
- Healthy links, counters, and endpoint tests shift attention to NCCL’s selected transport, plugin and version compatibility, affinity, or the workload.
RoCE troubleshooting branch
Collect host and switch telemetry together
On each participating NIC, run ethtool -S <nic_name> and record the error and drop fields exposed by that driver. From the switch, collect interface and queue telemetry for the corresponding ports. Examine errors, drops, pause frames, Priority Flow Control (PFC), Explicit Congestion Notification (ECN), Congestion Notification Packets (CNP), and queue drops. Field names differ by NIC, switch, and software version, so use the vendor’s counter definitions rather than mapping names by guesswork.
Correlate congestion signals with application symptoms
Repeat the workload while sampling counters. If ECN or CNP activity, PFC events, or queue drops rise at the same time as latency tails and unstable throughput, investigate the congestion-control and lossless-fabric configuration before changing NCCL. If counters remain flat while NCCL alone slows, return to transport selection, software versions, affinity, and workload behavior.
Test endpoint connectivity
Use rping when an endpoint-level RoCE connectivity check is needed. Run it across the same hosts, interfaces, traffic class, and address family used by the job; a test on a different path does not validate the training path.
Check GID and traffic-class behavior by version
The NVIDIA NCCL guide says dynamic GID selection is used in NCCL 2.21 and later and warns against setting NCCL_IB_GID_INDEX on those versions. Older releases may require manual GID selection. Follow the deployed vendor’s instructions for NCCL_IB_TC values and verify the result in NCCL logs. Do not copy a setting from an older cluster into a newer one without checking the version-specific guidance.
Physical-link checks belong at the end of the evidence chain
When a particular port shows link errors, recovery events, down transitions, or a lower negotiated rate, inspect the physical path: adapter seating, transceiver or cable compatibility, switch-port configuration, and connector condition. Use mlxlink where supported and follow the hardware vendor’s safety procedures. A compatible fiber-optic cleaning kit can be appropriate only when inspection identifies a connector or cabling issue and the kit matches the installed optics; cleaning cannot correct PFC, ECN, queue drops, or an NCCL transport mismatch.
Rank #4
- optical fiber cleaning box：CLEANS MULTIPLE SURFACES - This box works as an SC, FC, ST, APC or LC Fiber Cleaner, as well as being an Optical Connector Cleaner.
- Optical Fiber Cleaner Pen ：FC/SC/ST Cleaner Pen, universal 2.5mm; LC MU Cleaner Pen1.25mm Comfortable design with single action cleaning Low cost per clean, up to 800+ Cleans
- Precise mechanical action delivers consistent cleaning results Automatically advance ensures each clean is performed with fresh cleaning tape
- Storage bag-Our optical fiber cleaning tools with waterproof empty bags can store fiber cleansing tools, and protecting products will not be damaged
- Industry professionals use this Optical Fiber Cleaner daily on mission critical projects. A hybrid fabric leaves no lint, and cleans dirty fiber with a swipe.
Replace or reseat one component at a time, then rerun the same endpoint and application tests. Record the change so a later improvement can be attributed to the physical intervention rather than to an unrelated workload or routing change.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Use a decision matrix before touching NCCL settings
| Local GPU checks | Endpoint or fabric checks | Likely focus | Next action |
|---|---|---|---|
| Fail | Any result | GPU peer path, PCIe/NVLink topology, or device configuration | Resolve the local path and rerun the P2P diagnostic before judging the network. |
| Pass | Fail or varies by rail | NIC, physical link, switch path, or fabric configuration | Follow the InfiniBand or RoCE counter branch and correlate deltas with the failure. |
| Pass | Pass, but counters rise during the job | Congestion, retries, queueing, or loss under load | Have the fabric owner investigate congestion-control and lossless settings before NCCL tuning. |
| Pass | Pass and counters remain stable | NCCL selection, plugin/version mismatch, affinity, or workload behavior | Compare logs, versions, topology, CPU and memory placement, and collective configuration. |
Verify software and environment consistency
Compare the NCCL version, network plugin, GPU driver, NIC driver, firmware, and container image on every node. A single host with a different plugin or driver can force a fallback or produce asymmetric behavior. Check that the environment variables visible inside the training process are the ones you intended; a shell export on a login host does not prove that a scheduler or container passed it to every rank.
Review CPU and memory affinity as well. An otherwise healthy fabric can be starved by a rank placed far from its NIC or by host memory contention. Keep this check separate from a packet-loss diagnosis: affinity problems usually do not create switch drops, but they can produce the same application-level slowdown.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Change NCCL configuration only as a controlled experiment
NCCL documentation cautions that a tuning gain in one benchmark can make another workload suboptimal. Preserve the original logs and environment, change one variable at a time, and test the real training workload rather than a synthetic case alone.
- Write down the baseline transport, interface, plugin, versions, affinity, bandwidth, latency, and relevant counter deltas.
- Choose one hypothesis, such as an incorrect interface selection or a version-specific RoCE GID setting.
- Make the smallest supported change for that hypothesis.
- Repeat the same node pair, message pattern, and workload duration.
- Compare completion time, latency tails, throughput stability, NCCL logs, and fabric counters.
- Revert the change if the application improves only on a synthetic test, if another workload regresses, or if counters show the underlying fault remains.
Do not use a tuning variable to hide a physical or fabric problem. A setting that suppresses an error message without restoring link health leaves the cluster vulnerable to the next workload.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Common symptoms and targeted responses
NCCL is slow across nodes, but single-node training is normal
Confirm the selected network plugin and interface in the NCCL log, then run host-to-host bandwidth and latency tests on the same rails. If low-level results are poor, follow the fabric branch. If they are healthy, compare NCCL versions, plugin availability, topology, and rank-to-NIC affinity.
Best Value
- Complete 14-in-1 Fiber Optic Kit: Includes fiber optic magnifying glass, cleaning box with replacement core, SC/LC/MPO cleaning pens, 1.25mm and 2.5mm cleaning rods, cleaning paper, cleaning bottle, air blower, brush, and convenient tool bag for organized storage
- Multi-Interface Cleaning Box: Suitable for SC/FC/LC/ST interfaces with replaceable cleaning core, providing up to 500 cleaning cycles for maintaining optimal fiber optic connections
- Dual Cleaning Pen System: SC cleaning pen works with SC/FC/ST 2.5mm interfaces while LC cleaning pen handles LC 1.25mm connections, each offering up to 800 cleaning cycles for long-lasting performance
- MPO Cleaning Pen: Designed for MPO MTP interface male and female connectors, featuring automatic cleaning thread replacement and delivering up to 600 cleaning cycles
- 400X Fiber Optic Microscope: Handheld fiber optic microscope equipped with 2.5mm and 1.25mm adapters for versatile inspection capabilities, enabling detailed examination and verification of fiber optic connections across various devices
The job hangs intermittently during a collective
Capture logs from all ranks, not just the rank that reports the timeout. Check whether the same port, rail, or node appears in each incident. Correlate timeout or retry counters and switch queue events with the hang window. A single successful rerun does not clear a path; repeat the test long enough to reproduce the condition.
RoCE throughput oscillates and PFC counters climb
Measure latency tails and queue drops while reproducing. Rising PFC, ECN, CNP, or queue-drop activity with the oscillation points to congestion-control or lossless-fabric configuration. Involve the network owner and verify traffic class, buffer, and priority settings against the deployed versions before changing NCCL.
One InfiniBand port reports errors
Compare its rate and state with the other rails, sample RDMA and perfquery counters before and after a controlled test, and inspect the corresponding cable, optic, adapter, and switch port. Use ibping and, where permitted, ibdiagnet to determine whether the issue is local or fabric-wide.
Free tools Windows power users keep installed
One-click scans. No signup required.
All counters look clean but the model remains slow
Check whether the job selected a Socket path or an unexpected plugin, whether active NCCL diagnostics were informational rather than a pass/fail gate, and whether CPU or memory placement is limiting progress. Compare collective timing and workload behavior before declaring a network fault.
A repeatable incident checklist
- Save NCCL logs from every rank with
INFOand the relevant debug subsystems. - Record NCCL, plugin, driver, firmware, NIC, switch, and fabric versions.
- Confirm the selected interface, transport, plugin, rail, link layer, and rate.
- Run local GPU P2P diagnostics where supported.
- Run matching node-to-node bandwidth and latency tests.
- Capture counters before, during, and after reproduction.
- For InfiniBand, inspect state, rate, RDMA retries/timeouts, port errors, Subnet Manager status, and topology.
- For RoCE, inspect NIC and switch errors, drops, pause/PFC, ECN, CNP, and queue drops.
- Check physical components only when link evidence points there.
- Change one NCCL setting at a time, test the real workload, and retain a rollback baseline.
This sequence turns “packet loss” from a broad suspicion into a bounded diagnosis: local GPU path, endpoint, physical link, fabric congestion, transport/plugin selection, or workload configuration.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
