The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Adding GPUs does not automatically make one inference request finish proportionally faster. Multi-GPU inference partitions model work, then pays for communication and synchronization between devices. The result can be higher aggregate throughput or enough memory to run a model that cannot fit on one card, while single-request latency improves only modestly—or not at all.
To compare a multi-GPU result with a single-GPU result, match the model, precision, runtime, hardware topology, parallelism method, batch or concurrency, input and output lengths, and the exact metric. A GPU count by itself is not a benchmark specification.
What changes when inference uses multiple GPUs?
On one GPU, most layer computation and the intermediate data it needs stay on one device. With multiple GPUs, the runtime must decide how to divide that work. It may split individual layers, place different groups of layers on different devices, or run independent copies of the model. Each choice optimizes a different outcome.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesCapacity is not the same as speed
Sharding a model across devices can provide the memory capacity required for its weights, activations, and key-value cache. This is often the first reason to add GPUs: a model that exceeds one card’s memory can run when its state is distributed across several cards. The ability to run it does not imply that one request will complete faster than on a smaller model or a faster single device.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
vLLM’s current stable parallelism guidance recommends tensor parallelism when a model is too large for one GPU but fits within a multi-GPU node. It also describes hardware situations in which pipeline parallelism can provide higher throughput and lower communication overhead. Those are recommendations for the stated configurations, not a universal ranking of parallelism methods.
Three different goals are commonly called “faster”
- Capacity: fitting a model and its working state in available memory.
- Single-request latency: how long one request takes, including prompt processing and generated tokens.
- Aggregate throughput: how many tokens or requests the system completes per second across a batch or concurrent clients.
Adding GPUs can improve one of these while leaving another nearly unchanged. A serving system may use extra devices for replicas, increasing the number of simultaneous requests, without reducing the latency of any individual request.
How parallelism creates communication overhead
Tensor parallelism
Tensor parallelism divides operations within a layer among GPUs. After a partitioned operation, ranks often need to exchange partial results before the next operation can continue. NVIDIA’s TensorRT-LLM model-sharding documentation describes an all-reduce before proceeding when each GPU needs the full output of a preceding layer. The collective operation introduces a synchronization point: every rank must participate and wait for the required data.
Free tools Windows power users keep installed
One-click scans. No signup required.
The useful compute performed in parallel is therefore only part of the elapsed time. The total includes launching kernels, moving intermediate tensors, coordinating ranks, and waiting for the slowest participant. As GPU count rises, the amount and frequency of coordination can rise as well.
Pipeline parallelism
Pipeline parallelism assigns groups of layers to different GPUs. A request’s activations travel from one stage to the next. The approach can reduce some all-reduce traffic compared with tensor parallelism, but it introduces stage-to-stage transfers and pipeline scheduling effects. With too little concurrent work, later stages may sit idle while earlier stages process a request. With enough in-flight work, stages can overlap and raise throughput.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Data parallelism and replicas
Data parallelism keeps a complete model copy on each assigned GPU or GPU group and sends different requests to different copies. This can scale request throughput without making a single request’s model computation shorter. A benchmark that reports more requests per second after adding replicas is answering a different question from a tensor-parallel latency test.
Why the interconnect and system layout matter
GPU count does not describe the path between GPUs. Devices may communicate through high-bandwidth links inside one node, through a switch fabric, or across a network between nodes. The latency and bandwidth of that path affect every collective operation and activation transfer.
NVIDIA’s technical discussion of NVLink and NVSwitch explains that GPUs can leave Tensor Cores idle while waiting for exchanged results. Its hardware examples are specific to the systems and software discussed there; they should not be converted into a fixed speedup expectation for unrelated machines.
That article gives a concrete, configuration-specific example: up to 20 GB of tensor-parallel synchronization data per GPU for one Llama 3.1 70B query with an 8K-token input and a 256-token output (NVIDIA, 2024). This is not a universal traffic figure. A different model, sequence length, parallelism degree, or implementation can produce a very different volume.
What to record about topology
- GPU model, memory capacity, and number of devices.
- Whether all devices are in one node or spread across nodes.
- The intra-node connection, such as PCIe, NVLink, or an NVSwitch arrangement where applicable.
- The inter-node network and its configuration when multiple servers are involved.
- How many GPUs belong to each model shard or replica.
Two tests using the same GPU model and count can therefore produce different results if one uses a fast intra-node path and the other crosses a slower link.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Why batch size changes the apparent speedup
Batching puts more independent work in flight so the GPUs can perform more useful computation per launch and amortize fixed costs. Larger batches often increase aggregate tokens per second. They also consume more memory and can increase the time a particular request waits for service.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Single-request latency, offline batch throughput, and live-service throughput are separate measurements. A two-GPU system might be slower for one request at batch size 1 but faster at a large offline batch. A serving benchmark with continuous batching may report high total throughput while individual users experience a different latency distribution.
Load variables that change results
- Static batch size versus continuous or dynamic batching.
- Number of concurrent requests and request-arrival pattern.
- Prompt length and generated output length.
- Whether requests share similar sequence lengths or are highly variable.
- Warm-up state, queueing time, and the duration of the measurement window.
When a benchmark says “tokens per second,” verify whether it means aggregate tokens across all requests or per-request decode speed. Those values cannot be compared directly.
Fixed overhead is most visible at low load
For small workloads, communication and launch costs can be a large fraction of total time. NVIDIA’s Triton documentation for its TensorRT multi-device backend describes replication of rank-0 inputs, collective-operation launches, and kernel-launch overhead. In a documented low-batch, single-attention-layer case, these costs can make multi-GPU performance neutral to modest and noisy.
This example is not a statement about every model. It illustrates why a workload that does little computation can fail to amortize the setup and synchronization work required by multiple devices. Increasing batch or sequence length may change the balance, but it can also increase memory pressure and latency.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Metrics that must not be mixed
| Metric | What it measures | Why it can disagree with another result |
|---|---|---|
| Time to first token (TTFT) | Time from request submission until the first generated token. | Prompt processing, scheduling, and queueing dominate; decode throughput may have little effect. |
| Per-token decode latency | Time between generated tokens, often expressed as tokens per second per request. | It reflects autoregressive synchronization and may worsen when communication occurs every layer. |
| End-to-end latency | Total time for prompt processing and all requested output tokens. | It changes with both input and output length and includes startup or queueing costs. |
| Aggregate throughput | Total tokens or requests completed per second across a batch or concurrent load. | Parallel work and replicas can raise this number even when individual requests are not faster. |
A benchmark comparison is invalid if one result is TTFT and the other is aggregate decode throughput, even when both are labeled “inference speed.”
A like-for-like comparison checklist
Before attributing a difference to GPU count, align every material variable:
Workload
- Use the same model name, version, architecture, and parameterization.
- Match precision or quantization, including any calibration or kernel-specific format.
- Use the same input-token and output-token lengths, or report distributions.
- Describe the request mix rather than relying on one synthetic sequence.
Hardware path
- Record GPU type, memory, count, and clock or power settings when relevant.
- State the node count and the intra- and inter-node interconnect.
- Identify whether extra GPUs are shards, pipeline stages, or independent replicas.
Software and method
- Name the inference framework and exact version.
- Record CUDA, NCCL, driver, and relevant communication-stack versions.
- State tensor, pipeline, data, or hybrid parallelism and the degree used.
- Explain warm-up, measurement duration, synchronization, and timing boundaries.
- Say whether the test is offline, fixed-batch, or a live serving test with arrivals and queueing.
Reported result
- Label TTFT, per-token latency, end-to-end latency, or aggregate throughput.
- Include batch size, concurrency, and whether batching is continuous.
- Report averages and useful tail percentiles for serving tests; a single average can hide synchronization stalls.
No single cross-framework standard covering every item above is established by the cited documentation. This is a practical reporting checklist, not a formal certification protocol.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to read a surprising benchmark result
Two GPUs are barely faster than one
First check whether the test used batch size 1 or short sequences. Then inspect the interconnect and whether every layer performs a collective operation. If communication and launch overhead consume the saved compute time, a small gain is expected. Confirm that the two-GPU run did not use a different precision, runtime setting, or clock limit.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Two GPUs are slower for one request but faster overall
This usually indicates that the benchmark rewards throughput rather than latency. Larger batches, continuous batching, or independent replicas can keep both GPUs busy. Compare per-request latency and queueing separately from total tokens per second.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
More GPUs make the model fit but do not improve latency
That is a capacity win. If the model was memory-bound on one device, sharding removes the capacity limit; it does not remove the communication required by the shard. Measure whether the original single-GPU configuration was even able to run the same model and sequence lengths before declaring a speed regression.
Results differ between two servers with the same GPUs
Compare topology first: PCIe layout, NVLink or switch availability, NUMA placement, node count, and network fabric. Then compare framework, driver, CUDA/NCCL versions, parallelism mapping, and warm-up procedure. A nominally identical GPU list can hide a materially different communication path.
A reproducible benchmark procedure
- Define the question. Decide whether you need model fit, single-request latency, tail latency, or aggregate throughput.
- Freeze the workload. Specify model revision, precision, input length, output length, request mix, and stopping conditions.
- Document the machine. List GPUs, memory, node count, interconnect, CPU and system topology relevant to placement, and network details for multi-node runs.
- Choose and record parallelism. State tensor, pipeline, data, or hybrid mode, plus the number of GPUs per shard and per replica.
- Pin software. Record the inference framework, CUDA, NCCL, driver, kernel, and container or build identifiers.
- Warm up consistently. Run enough unmeasured iterations to initialize kernels, memory pools, and communication paths, and use the same warm-up policy for every configuration.
- Sweep load deliberately. Test the batch or concurrency levels that match your use case instead of reporting one convenient point.
- Measure the right boundary. Separate TTFT, decode timing, end-to-end latency, queueing, and aggregate throughput. Synchronize devices before stopping timers.
- Repeat and report variation. Include run duration, repeats, averages, and latency percentiles so noise from launches and collectives is visible.
- Publish the configuration with the number. A result without topology, parallelism, workload, and metric labels cannot support a reliable GPU-scaling conclusion.
Practical decisions: when more GPUs help
| Situation | Likely benefit | Main risk |
|---|---|---|
| Model does not fit in one GPU | Distributed capacity makes inference possible. | Single-request latency may stay flat or increase because of transfers and synchronization. |
| Many independent requests | Replicas or data parallelism can raise aggregate throughput. | Per-request latency and memory per replica still need separate measurement. |
| Large, steady batches | More parallel work can amortize launch and communication overhead. | Higher memory use and longer wait time for individual requests. |
| Fast intra-node fabric | Lower communication cost can make tensor or pipeline sharding more effective. | Results may not carry over to PCIe-only or multi-node systems. |
| Short, low-batch requests | Usually limited speedup unless the single GPU is capacity-bound. | Fixed launches, input replication, and collectives can dominate. |
The correct choice depends on the service objective. Select a sharding strategy to meet memory and latency constraints, or add replicas when the requirement is concurrent capacity. Do not use aggregate throughput as a substitute for user-visible latency.
What a trustworthy result should look like
A useful report might say: “Llama 3.1 70B, specified quantization, 8K-token input and 256-token output, framework and versions identified, two or eight named GPUs in one node, interconnect stated, tensor-parallel degree stated, batch and concurrency stated, and TTFT, end-to-end latency, per-token rate, and aggregate throughput reported separately.” That description lets another reader determine whether the comparison answers the same question.
A bare statement such as “eight GPUs are four times faster than one” omits the variables that determine whether the result is repeatable. Multi-GPU inference is a distributed system: compute partitioning, memory capacity, communication topology, synchronization, load, and measurement method all shape the number.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
