iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Diagnose an inter-datacenter link by correlating training-step or collective time with point-to-point latency, throughput, RDMA counters, switch telemetry, and physical-link status during the same incident window. Bandwidth by itself is not a health check: NVIDIA’s NCCL documentation warns that “Bandwidth can look healthy while tail latency is still poor.” A repeatable comparison of unloaded and loaded traffic, followed by transport-specific counters, separates packet loss and retries from congestion, routing errors, and physical degradation.
The workflow below is designed for distributed GPU training over RoCE or InfiniBand. It gives you a way to localize the problem to a host pair, rail, route, port, or component without assuming a universal latency or error threshold. Set alert limits from your own healthy baseline and workload sensitivity.
What a useful diagnosis must establish
A slow training job does not automatically mean a bad inter-datacenter circuit. Collective operations can wait on a slow rank, a congested queue, retransmissions, a route collision, a switch software change, or a link that has fallen to a lower speed or width. The useful question is whether the slowdown tracks network behavior and, if it does, which layer explains it.
- Transport loss or retries: packets are dropped, arrive out of sequence, or fail to receive acknowledgements, causing recovery traffic and stalls.
- Fabric congestion: queues, buffers, pause mechanisms, or congestion-control signals rise under load while the physical link remains healthy.
- Routing or software behavior: a rail, path, static route, switch image, or configuration change creates collisions or inconsistent performance.
- Physical degradation: bit errors, link recovery, flaps, a dirty connector, a bad transceiver, or a negotiated speed/width problem affects one port or path.
Meta’s ACM SIGCOMM 2024 deployment report summarizes the operational reason for this depth of checking: “RDMA is very sensitive to network issues and hence it affects GPU training efficiency.”
#1 Best Overall
- 【Fiber Optic Cleaning Kit】 The fiber optic cleaning swab effectively cleans multiple surfaces. This swab functions as a cleaner for SC, FC, ST, APC, or LC fiber optics, as well as optical connectors. Capable of over 500 reusable cleanings, its ergonomic design ensures comfortable handling. Each swab provides a single-use cleaning solution, removing even stubborn contaminants in one pass.
- 【Stable Fiber Cleaning 800+】Our fiber optic cleaning pen delivers consistent anti-static cleaning performance, supporting over 800 cleaning cycles for lower per-use costs. Our LC cleaning pen thoroughly removes contaminants from fiber surfaces, helping you get the job done.
- 【Premium Exterior Design】This fiber optic cleaning pen seamlessly combines convenience with efficiency. Designed with the familiar shape of a standard pen, it offers comfortable grip and effortless handling during cleaning.This fiber optic cleaning pen is easy to operate. Its unique “click” confirmation sound signals completion of the cleaning process, ensuring thorough and efficient cleaning every time.
- 【Stable Cleaning】This cleaning kit effectively removes dust, grease, and other debris from fiber optic port surfaces without scratching the surface.
- 【Wide Application】This LC fiber optic cleaning set is the ideal choice for fiber optic cleaning. This highly effective cleaning pen is suitable for LC/SC/FC/ST/MU/MTP/MPO/D4/DIN connectors
The six-stage diagnostic workflow
1. Scope the symptom before collecting counters
Start an incident record for the exact training run and time window. Capture:
- job, framework, model, world size, rank-to-host mapping, and the collective phase or operation that stalls;
- affected host and rank pairs, data-center endpoints, rail or fabric identifier, and whether all paths or only a subset are involved;
- step time, collective time, and application logs for healthy and affected runs;
- start and end times in UTC, whether the issue is persistent or intermittent, and any concurrent jobs;
- recent changes to cabling, NIC firmware, switch software images, routing, quality-of-service, congestion-control, or GPU and PCIe configuration.
Compare the incident with the same workload’s own healthy run. A slowdown limited to one rail, one host pair, or one route is valuable localization evidence even when aggregate fabric dashboards look normal.
2. Measure throughput and latency together
Run a controlled RDMA test between the affected endpoints, using the same message sizes, queue-pair (QP) counts, traffic class, and path or rail used by training. Measure both an unloaded condition and a loaded condition that approximates the production collective. Record throughput at each message size and QP setting, then obtain point-to-point latency while the job is actually stalling.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Preserve the latency distribution, not just its average. Record minimum, mean, P50, P95, P99, P99.9, and maximum. NVIDIA specifically recommends point-to-point latency when jobs stall or slow intermittently because a long tail can coexist with apparently healthy bandwidth.
Rank #2
- 【Multi Testers Tool Sets】This fiber optic termination tool kit includes OPM, visual fault locator, fiber cleaning pen, fiber cleaner cassette, fiber cleaver, fiber stripper, Kevlar scissors, FC to LC adapter, cleaning core, liquid dispenser, fiber cleaning swabs, fiber cleaner wipes, fiber optic tool bag. This fiber optic tool kit is used for fiber stripping, cleaving, cleaning and testing. It provides the fiber optic tools you need to assemble mechanical splices and delivers greater work efficiency.
- 【Fiber Cleaning Kit】1.25mm & 2.5mm fiber optic cleaner pen & fiber optic cleaning box are used to clean LC/MU/SC/FC/ST/MTP/MPO/D4/DIN ferrule end face. The fiber cleaner can effectively clean stains and prevent dust from adhering to the ferrule again after cleaning.
- 【Visual Fault Locator Fiber Optic】Our fiber tester can intuitively and accurately detect and locate fiber breaks, poor connections, bends or cracks. It is designed for SC, FC, LC, ST interfaces both in circle and square shape of different fiber optic cables. The fiber optic tester has strong output power.
- 【Fiber Optic Meter with FC to LC Adapter】The fiber optic cable tester can measure 10 standdard wavelengths 850/980/1270/1300/1310/1490/1550/1577/1625/1650nm with a test range of -70dBm~+10dBm. The measurement error is less than ±0.2dB. Support simultaneous display of linear mW and nonlinear index dBm.
- 【Fiber Optic Cleaver】3-in-1 high precision fiber cleaver is used to cut bare fiber (0.25mm), pigtail (0.9mm) and leather cable. It has 16 cutting points, the cutting angle is about 0.5 degree, and it can cut about 36000 times.
A February 2026 IETF BMWG draft for AI-fabric benchmarking proposes measuring throughput for at least 60 seconds at each rate, repeating latency characterization at least 20 times, and repeating burst-absorption tests at least 50 times for each burst length. That document expired on 2026-08-30 and is a proposed methodology, not an adopted standard or a production pass/fail requirement. Use its measurement structure when it helps, but set duration and repetition appropriate to your environment.
| Observed pattern | What it suggests | Next check |
|---|---|---|
| Throughput falls and latency tail expands only under load | Congestion, queue drops, pause/PFC, ECN/CNP behavior, or a route collision | Per-port utilization, queue and buffer telemetry, PFC/ECN/CNP counters, and route usage |
| Average latency is normal but P99/P99.9 or maximum spikes during stalls | Intermittent queueing, retransmission, out-of-order delivery, or a flapping path | RDMA retry and sequence counters, switch drops, link-flap history, and time-correlated logs |
| Throughput and latency degrade even when the path is unloaded | Physical errors, negotiated speed/width, a bad port, persistent route problem, or host/PCIe fault | Port state and errors, mlxlink where supported, cabling, NIC and PCIe telemetry |
| Only one rail or host pair is slow | Localized port, cable, transceiver, route, or rail imbalance | Compare the same test on a healthy rail and inspect the exact path |
3. Watch RDMA counters while the test runs
Take a baseline, run the controlled test for a fixed interval, and compare counter deltas rather than lifetime totals. On systems that provide it, run rdma statistic before and after the interval and collect switch, NIC, PCIe-switch, and GPU telemetry at the same timestamps. NVIDIA highlights these RDMA statistics:
| Counter | Why a rising delta matters | How to interpret it |
|---|---|---|
rnr_nak_retry_err |
Receiver-not-ready negative acknowledgements and retries | Can indicate receiver pressure, loss, or a stalled path; correlate with queue, pause, and workload timing |
packet_seq_err |
Packets arrived with an unexpected sequence | Useful evidence of drops, reordering, or a path problem; check switch and NIC observations together |
implied_nak_seq_err |
Sequence gaps inferred by the receiver | Supports a loss or ordering hypothesis when it rises during the slowdown |
local_ack_timeout_err |
Expected acknowledgements did not arrive before timeout | May reflect congestion, loss, a degraded link, or a remote endpoint that stopped progressing |
Meta also calls out NIC-observed out-of-sequence packets, link flaps, and local ACK timeouts. A single counter is not a verdict: a counter that rises only during a known test and matches latency tails or switch drops is much stronger evidence than a large lifetime value with no timing relationship.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors4. Follow the transport-specific branch
RoCE (RDMA over Converged Ethernet)
Inspect the NIC’s hardware counters with:
ethtool -S <nic_name>Counter names differ by NIC and driver, so first identify which fields represent errors, drops, pause frames, PFC priority pauses, ECN marks, CNPs, and queue or buffer drops. Collect the corresponding switch-port counters for both directions. A rise in PFC pauses or queue drops alongside a latency tail points toward congestion or lossless-fabric behavior; a rise in symbol, CRC, or other physical errors points elsewhere.
Rank #3
- 【Comprehensive 13 in 1 cleaning kit】This set includes 13 professional cleaning tools that can meet the cleaning needs of various fiber optic connectors. Not only for daily maintenance but also for troubleshooting, this set of tools can provide solutions to ensure that your work is twice as efficient.
- 【400x high-definition microscope for precise detection】Built in high-precision 400x magnification fiber optic microscope allows you to clearly observe small scratches, dust, and contamination on the fiber end face. Visually inspect the cleaning effect to ensure the quality of the connection and reduce signal loss.
- 【Widely compatible with multiple interface types】Compatible with fiber optic interfaces such as SC, FC, ST, LC, etc. Not only 2.5mm (SC/FC/ST) and 1.25mm (LC) ceramic plugs, but even precision MPO/MTP multi-core connectors, this set has corresponding tools for efficient cleaning.
- 【Efficient "one click" cleaning pen, convenient and fast】Equipped with a one click quick cleaning pen, just press lightly to complete an efficient cleaning. The operation is simple and fast, which is very suitable for frequent use during installation, testing, and maintenance, greatly improving work efficiency.
- 【Professional cleaning tools to protect precision equipment】Equipped with professional tools such as fiber optic wiping clips and spare wiping cloths, it can effectively remove stubborn stains and oil stains, while avoiding secondary pollution or physical damage to precision ceramic plugs.
Check whether ECN marking and congestion notification packets (CNPs) increase on the same traffic class. Excessive marking or CNPs can indicate that queues are persistently filling; absent or asymmetric signals can expose a configuration mismatch. Do not change collective-library settings first when the evidence shows a congested or dropping fabric.
If counters appear normal but reachability or connection setup remains suspect, use rping between the relevant endpoints. Treat a successful rping as a connectivity check, not proof that production-size collectives have acceptable tail latency.
InfiniBand
Use perfquery to inspect port status and error counters. NVIDIA identifies symbol errors, link-recovery events, and link-down counters as useful clues for physical or fabric trouble. Use ibping for node-to-node reachability, then ibdiagnet for a wider view of topology, routing, and fabric errors.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Compare the affected port with a healthy port carrying the same class of traffic. A port that repeatedly recovers, goes down, or accumulates physical errors while its peer remains clean deserves hardware and cabling inspection before tuning software.
Rank #4
- optical fiber cleaning box：CLEANS MULTIPLE SURFACES - This box works as an SC, FC, ST, APC or LC Fiber Cleaner, as well as being an Optical Connector Cleaner.
- Optical Fiber Cleaner Pen ：FC/SC/ST Cleaner Pen, universal 2.5mm; LC MU Cleaner Pen1.25mm Comfortable design with single action cleaning Low cost per clean, up to 800+ Cleans
- Precise mechanical action delivers consistent cleaning results Automatically advance ensures each clean is performed with fresh cleaning tape
- Storage bag-Our optical fiber cleaning tools with waterproof empty bags can store fiber cleansing tools, and protecting products will not be damaged
- Industry professionals use this Optical Fiber Cleaner daily on mission critical projects. A hybrid fabric leaves no lint, and cleans dirty fiber with a swipe.
5. Inspect hardware, path selection, and recent changes
When transport counters or unloaded tests point away from ordinary congestion, inspect the physical and control-plane layers.
- Link negotiation and state: where supported, run
mlxlinkand verify expected speed, lane width, port state, and reported health. A lower-than-expected speed or width can cap throughput without producing obvious packet loss. - Cables and transceivers: inspect both ends, patch panels, and connectors for seating, contamination, damage, or incompatible optics. Cisco’s AI/ML networking guidance identifies faulty or dirty cable and transceiver connections as possible causes of bit errors.
- Flaps and recovery: align link-up, link-down, recovery, CRC, symbol, and physical-error timestamps with the training stall.
- PCIe and host path: check NIC, PCIe-switch, and GPU counters for the same interval. A network symptom that follows one host or PCIe path rather than one fabric path may not be an inter-datacenter fault.
- Routing and utilization: map the exact route or rail, check per-port utilization, and look for static-route collisions or an imbalanced path. Meta describes static-routing traffic collisions that produced inconsistent performance in its deployment.
- Software images and configuration: review switch-image rollouts, QoS, MTU, PFC, ECN, buffer, and routing changes. Meta also reports a performance regression associated with switch software-image scheduling, showing why a clean congestion dashboard does not prove that software and latency are healthy.
Do not clean or replace a connector merely because a link is slow. A fiber-cleaning kit is relevant only when inspection and error evidence implicate contamination; follow site procedures, record the intervention, and verify the link afterward.
6. Compare, isolate, and document the result
Repeat the same measurements on a healthy path and, if safe, under both isolated and concurrent traffic. Hold endpoints, message sizes, QP counts, traffic class, and workload phase constant. Record:
Recommended Free Tools
- throughput or goodput by message size and QP count;
- unloaded and loaded latency distributions, including tail percentiles;
- RDMA retransmission, sequence, and timeout deltas;
- per-port utilization, queue and buffer occupancy, PFC, ECN, CNP, and drop signals;
- physical errors, link-recovery events, and flaps;
- job and collective time with the suspect rail isolated and with concurrent traffic restored;
- route, configuration, and switch-image changes in the incident window.
A finding is strongest when an intervention changes the suspected signal and the training symptom at the same time—for example, moving a host pair to a healthy rail removes both the latency tail and the RDMA timeout increase.
Best Value
- Complete 14-in-1 Fiber Optic Kit: Includes fiber optic magnifying glass, cleaning box with replacement core, SC/LC/MPO cleaning pens, 1.25mm and 2.5mm cleaning rods, cleaning paper, cleaning bottle, air blower, brush, and convenient tool bag for organized storage
- Multi-Interface Cleaning Box: Suitable for SC/FC/LC/ST interfaces with replaceable cleaning core, providing up to 500 cleaning cycles for maintaining optimal fiber optic connections
- Dual Cleaning Pen System: SC cleaning pen works with SC/FC/ST 2.5mm interfaces while LC cleaning pen handles LC 1.25mm connections, each offering up to 800 cleaning cycles for long-lasting performance
- MPO Cleaning Pen: Designed for MPO MTP interface male and female connectors, featuring automatic cleaning thread replacement and delivering up to 600 cleaning cycles
- 400X Fiber Optic Microscope: Handheld fiber optic microscope equipped with 2.5mm and 1.25mm adapters for versatile inspection capabilities, enabling detailed examination and verification of fiber optic connections across various devices
How to distinguish the main failure classes
| Evidence combination | Most likely class | Confirmatory action |
|---|---|---|
| Loaded-only tail latency, rising queue drops or PFC, stable physical counters | Congestion or lossless-fabric configuration | Reduce competing traffic or isolate the rail; verify queue, ECN/CNP, PFC, and buffer behavior |
| Sequence errors, implied NAKs, ACK timeouts, and switch drops aligned to stalls | Packet loss or retransmission | Trace the affected hop and direction; compare NIC and switch counters and route symmetry |
| Symbol/CRC errors, link recovery, flaps, or reduced speed/width even when unloaded | Physical link or optics problem | Inspect and reseat according to procedure, test the cable/transceiver, and verify negotiated state |
| One rail is slow with no corresponding physical errors; route utilization is asymmetric | Routing collision or rail imbalance | Compare route tables and path utilization; test the same endpoints over another rail |
| Regression begins after a switch-image or QoS change and affects multiple paths | Software or configuration change | Correlate rollout time, configuration diff, and before/after latency distributions |
| Network counters are clean, but one host remains slow and PCIe/GPU telemetry is abnormal | Host, PCIe, or accelerator path | Repeat the test with another NIC, host, or PCIe path before attributing the issue to the fabric |
Numbers you may encounter—and how not to misuse them
Some published figures are useful as examples of measurement practice, not universal alert limits.
- Meta reports up to a 15% performance improvement for small message sizes from a specific latency-sensitive hierarchical collective tuning change in its deployment. It is not an expected gain for every topology or workload.
- Meta used a PFC pause longer than 200 ms as a watchdog trigger on its RTSW and CTSW devices. That is a deployment-specific trigger, not a generally “bad” PFC value.
- Meta raised alarms above 80% buffer utilization on RTSW devices and reported that the threshold had not been breached in practice. Buffer size, traffic mix, and switch behavior differ elsewhere.
- The February 2026 IETF draft’s proposed minimum of 60 seconds per throughput rate, latency percentiles from min, mean, P50, P95, P99, P99.9, and maximum under zero-load and full-utilization conditions, and repeated runs for latency and burst tests are benchmarking procedures. The draft expired on 2026-08-30 and does not establish production requirements.
Neither a single latency number nor a single counter value can classify every link. Build a baseline for each fabric, traffic class, message-size range, and training pattern, then alert on statistically meaningful deviation from that baseline.
Common diagnostic mistakes
- Looking only at average bandwidth: a high average can hide P99 or P99.9 stalls that dominate synchronous collectives.
- Reading lifetime counters without deltas: old errors can obscure whether the current incident is generating new failures.
- Testing an idle path only: many failures appear when queues, PFC, ECN, or burst absorption are stressed.
- Changing NCCL or application settings before proving the network cause: this can mask a physical or fabric fault and make comparison harder.
- Assuming a clean aggregate dashboard clears every route: a single rail, static route, switch image, or port can be bad while totals remain normal.
- Applying another site’s threshold: published watchdog values and performance gains are context-specific.
- Cleaning or replacing hardware without evidence: physical intervention should follow error and inspection evidence and must be followed by verification.
Close the incident with a reproducible record
Store the healthy and incident measurements together, including the exact endpoint pairs, rail, message sizes, QP settings, traffic class, test duration, latency percentiles, counter deltas, route, switch image, and configuration snapshot. State which hypothesis was supported, which was ruled out, and what changed after remediation. A later regression can then be compared against a known distribution rather than an informal “it felt slower” report.
Free tools Windows power users keep installed
One-click scans. No signup required.
The practical decision rule is simple: if training stalls coincide with a new latency tail and transport or fabric counters, follow those counters to the affected hop; if unloaded tests already show physical errors or a reduced link state, inspect the port and cabling; if only one rail or route diverges, investigate path selection and recent changes. Keep the workload constant, preserve distributions, and let site-specific baselines—not generic thresholds—define abnormal behavior.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
