Free tools Windows power users keep installed
One-click scans. No signup required.
Verify a Kubernetes GPU cluster in layers: confirm the job can reach the intended nodes and resources, measure local GPU paths, test the inter-node fabric, then run a multi-node NCCL collective. A healthy single-node test cannot establish that cross-node rails work, and no single bandwidth threshold applies to every GPU, NIC, rail layout, collective, and message size.
What to verify—and in what order
Each test answers a different question. A topology or peer-access check can expose local connectivity issues, but it does not measure achieved bandwidth. A fabric test isolates network performance, but it does not prove that an NCCL workload uses the intended paths. The final multi-node collective validates the combination under the job’s actual placement and configuration.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
NVD RTX PRO 6000 Blackwell Professional Workstation Edition Graphics Card for AI, Design,... | $19,999.99 | Buy on Amazon |
| 2 |
|
NVIDIA RTX PRO 4000 Blackwell Graphics Card - 24GB GDDR7 ECC Memory, PCIe 5.0 x16, 4X DisplayPort... | $3,134.14 | Buy on Amazon |
| 3 |
|
PNY NVIDIA RTX A6000 | $6,169.96 | Buy on Amazon |
| Diagnostic layer | What it helps establish | What it does not establish by itself |
|---|---|---|
| Kubernetes placement and operator readiness | Whether the required components are available and the job is placed on its intended GPU nodes and network resources. | That the GPUs, NICs, or rails deliver expected performance. |
nvidia-smi topo and nvbandwidth |
Local GPU topology, peer-access status, and measured GPU-to-GPU bandwidth. | Inter-node fabric performance or multi-node NCCL behavior. |
ib_write_bw or ib_write_lat |
Fabric bandwidth or latency along the selected physical network path. | That an NCCL collective uses every intended rail or meets an application’s performance needs. |
| Multi-node NCCL tests | Collective correctness and performance across the GPUs, job placement, and network paths used by the test. | That a different workload, message size, or configuration will perform identically. |
1. Confirm Kubernetes prerequisites and placement
NVIDIA’s DGX Kubernetes validation example uses the MPI Operator, GPU Operator, and Network Operator for multi-node NCCL tests. Treat these as prerequisites for that example, not as a universal installation recipe: operator names, supported versions, network integration, and resource names depend on the cluster.
- Check the relevant operator deployments with
kubectl get deploymentin the namespaces used by your installation. Confirm the components are healthy using the readiness checks appropriate to your deployment. - Verify that the intended GPU nodes are schedulable and that the multi-node job runner is available. Confirm the job actually lands on the intended nodes and receives the expected GPUs and network resources.
- On DGX systems using InfiniBand, check that the compute-side InfiniBand interfaces are up. Confirm the corresponding interface state on the nodes that will participate in the test.
Use a cluster-supported job template rather than copying an environment-specific manifest. NVIDIA’s example establishes a validation approach, but it does not provide one generic YAML configuration or performance target suitable for every Kubernetes distribution.
#1 Best Overall
- PLEASE NOTE: Exporting an NVIDIA RTX Pro 6000 GPU outside the US requires strict adherence to the U.S. Export Administration Regulations (EAR) and issuance of an export license from the Bureau of Industry and Security (BIS). Compliance and Know Your Customer (KYC) screening may be required as a condition of order acceptance. [NVIDIA Blackwell Streaming Multiprocessor] The new SM features increased processing throughput, and new neural shaders that integrate neural networks inside of programmable shaders | DLSS 4: Multi Frame Generation ensures ultra-smooth frame pacing for lifelike simulations.
- [Double-Flow-Through Design] The RTX PRO 6000 Blackwell features a double-flow-through cooling design, optimizing efficiency and airflow to sustain peak performance under 600W power loads. | [5th Gen Tensor Cores] Deliver up to 3X the performance of the previous generation and support for FP4 precision for faster AI model processing times with reduced memory usage, enabling local fine-tuning of LLMs and generative AI | [4th Gen Ray Tracing Cores] Double the ray-triangle intersection rate of the previous generation to create photoreal, physically accurate scenes and immersive 3D designs with RTX Mega Geometry, which enables up to 100X more ray-traced triangles.
- [PCIe Gen 5] Support for PCIe Gen 5 provides double the bandwidth of PCIe Gen 4, improving data-transfer speeds from CPU memory and unlocking faster performance for data-intensive tasks like AI, data science, and 3D modeling. | [GDDR7 Memory] With 96 GB of GPU memory and 1.8 TB ps bandwidth, it can tackle massive 3D and AI projects, fine-tune AI models locally, explore large-scale VR environments, and drive larger multi-app workflows.
- [DisplayPort 2.1] Achieve unparalleled visual clarity and performance, driving high resolution displays at up to 8K at 240 Hz and 16K at 60 Hz. Increased bandwidth enables seamless multi-monitor setups while HDR and higher color depth support ensures superior color accuracy for precision work, such as video editing, 3D design, and live broadcasting.
- [Universal MIG] Divide a single RTX PRO 6000 Blackwell into multiple isolated instances, each with dedicated resources, allowing for concurrent execution of multiple workloads, optimized GPU utilization, and secure isolation of different applications or users. [WARRANTY] 3 YR Manufacturer's Warranty. Bulk OEM Packaging. Retail Packaging is NOT included.
2. Check local GPU topology and bandwidth
Run these checks on each participating node. They help separate an intra-node GPU path problem from a cross-node fabric problem.
nvidia-smi topo -mshows the node’s GPU and device topology.nvidia-smi topo -p2p ninspects NVLink peer-to-peer access; usenvidia-smi topo -pto inspect PCIe peer access.- Run NVIDIA’s
nvbandwidthto measure GPU-to-GPU bandwidth on the paths relevant to the node’s topology.
A P2P matrix reports access status; it is not a bandwidth benchmark and does not, by itself, establish correctness under a workload. NCCL can use GPU P2P when CUDA reports that peers can communicate directly, typically over NVLink or PCIe, subject to topology and driver support. Keep the topology and measured bandwidth results together so a connectivity result is not mistaken for a performance result.
Check the GPU-to-NIC path separately
GPU-to-NIC direct communication is a separate path from GPU-to-GPU P2P. Verify that the NIC and driver support the intended GDRDMA route and that the deployed configuration supports it. NVIDIA documents nvidia-peermem as one route; supported DMA-BUF configurations can operate without that module. A successful local GPU P2P check does not validate GPU-to-NIC access.
3. Measure the inter-node fabric and rails
Before interpreting a slow collective, validate node-to-node connectivity with fabric tools. NVIDIA’s NCCL diagnostics can run ib_write_bw over the physical InfiniBand devices selected by NCCL when the communicator spans at least two hosts. The ib_write_bw utility from perftest must be available on each participating node, and hostnames must resolve across the participating nodes. NVIDIA’s NCCL guidance also identifies ib_write_lat for fabric latency checks.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #2
- Professional GPU with Blackwell Architecture
- Blackwell Architecture
- 24GB GDDR7 with PCIe 5.0 & Ray Tracing
- AI Workstation
When the endpoints and installed tool support it, the bandwidth measurement uses GPU memory; otherwise, it falls back to host memory. Record which memory path was used. A host-memory result should not be presented as validation of GPU-direct transport.
Compare same-NIC and cross-NIC results
When scheduled, NCCL diagnostics report same-NIC and cross-NIC modes separately. Preserve the NIC identity and rail mapping with each result, and interpret “cross-NIC” in light of the physical topology and the job’s NCCL_CROSS_NIC setting. The diagnostic tests paths selected through its topology and NCCL behavior; it does not prove that arbitrary physical paths were exercised.
Capture at least the participating nodes, GPU-to-NIC locality, rail-to-NIC mapping, interface state, selected memory path, and per-rank results. This context helps distinguish an unused or unexpectedly selected path from a poor measurement on a path the test actually exercised.
4. Run a multi-node NCCL correctness and performance test
After the independent GPU and fabric checks, run an NCCL test workload across the intended nodes and GPUs. Use the cluster’s supported multi-node job workflow and high-speed links. Check correctness first, then record performance across message sizes relevant to the real workload. Results are meaningful only alongside the test’s placement, collective, message size, and configuration.
Rank #3
- NVIDIA Ampere Architecture-based CUDA Cores - Double-speed processing for single-precision floating point (FP32) operations and improved power efficiency provide significant performance improvements for graphics and simulation workflows, such as complex 3D computer-aided design (CAD) and computer-aided engineering (CAE), on the desktop.
- Second-Generation RT Cores - With up to 2X the throughput over the previous generation and the ability to concurrently run ray tracing with either shading or denoising capabilities, second-generation RT Cores deliver massive speedups for workloads like photorealistic rendering of movie content, architectural design evaluations, and virtual prototyping of product designs. This technology also speeds up the rendering of ray-traced motion blur for faster results with greater visual accuracy.
- Third-Generation Tensor Cores - New Tensor Float 32 (TF32) precision provides up to 5X the training throughput over the previous generation to accelerate AI and data science model training without requiring any code changes. Hardware support for structural sparsity doubles the throughput for inferencing. Tensor Cores also bring AI to graphics with capabilities like DLSS, AI denoising, and enhanced editing for select applications.
- Third-Generation NVIDIA NVLink - Increased GPU-to-GPU interconnect bandwidth provides a single scalable memory to accelerate graphics and compute workloads and tackle larger datasets.
- 48 Gigabytes (GB) of GPU Memory - Ultra-fast GDDR6 memory, scalable up to 96 GB with NVLink, gives data scientists, engineers, and creative professionals the large memory necessary to work with massive datasets and workloads like data science and simulation.
Do not substitute the DCGM NCCL Tests plugin for this step. NVIDIA’s current DCGM documentation says the plugin runs only single-node NCCL tests; multi-node tests are unsupported. It can help check local NCCL behavior when the required NCCL library, test binary, and executable path are installed and configured.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.5. Interpret results and isolate the bottleneck
Use diagnostic status as a lead, not a service-level target
NCCL diagnostics use [OK] when a check completes without reporting an issue and [INFO] when a condition needs review, such as a failed verification or an incomplete check. A passing P2P check makes an intra-node or NVLink issue less likely; investigate other components, including the inter-node network or application. Failed checks can identify affected GPU pairs and connection paths.
The diagnostics provide per-rank minimum, median, and maximum bandwidth for same-NIC and cross-NIC modes. They may flag a rank more than 30% from that mode’s median. That is an outlier-reporting rule in this diagnostic—not a universal bandwidth acceptance threshold. NVIDIA’s example output showing 52 of 56 GPU-to-GPU peer accesses passing is illustrative, not a recommended pass rate.
Follow the evidence to the next layer
- Local peer access or GPU bandwidth is unexpectedly poor: investigate GPU topology, peer connectivity, driver support, and the measured GPU-to-GPU path before attributing the result to the fabric.
- Local GPU checks look sound but fabric bandwidth or latency does not: inspect interface state, node-to-node reachability, rail-to-NIC mapping, and whether the fabric test used GPU or host memory.
- Standalone GPU and fabric measurements match expectations but NCCL remains slow: examine job placement and NCCL configuration. NVIDIA’s performance guidance identifies
NCCL_CROSS_NIC, QPs per connection, chunk sizing, and CPU/memory affinity as variables that can affect results.
Change one setting at a time and compare under the actual workload. NVIDIA cautions that a tuning choice that helps one benchmark can be suboptimal for another; there is no universally optimal setting established for unspecified hardware and workloads.
Recommended Free Tools
What to record for a repeatable comparison
- Node and GPU identities, Kubernetes job placement, and the operators and versions in use.
- GPU topology and P2P status from each node, plus
nvbandwidthresults for the paths being evaluated. - NIC identity, rail mapping, interface state, hostname resolution, and fabric bandwidth or latency results.
- Whether a fabric bandwidth test used GPU memory or host memory.
- NCCL and CUDA versions,
NCCL_CROSS_NICvalue, collective, message sizes, per-rank same-NIC and cross-NIC measurements, and correctness outcome. - The hardware- and workload-specific baseline used for comparison.
NVIDIA’s NCCL 2.31.2 performance guidance recommends comparing standalone GPU and fabric performance with expected hardware performance before changing NCCL tuning. The appropriate baseline depends on the deployed GPUs, NICs, rail count, collective, message size, and software versions; the cited guidance does not establish a universal numeric threshold.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
