iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Sharing one GPU depends on whether you need models to run at the same time or take turns. Scheduler-level time-slicing can co-locate models, while ModelPool and a runtime router switch between them; exclusive assignment gives one model the card. In Christopher Maher’s August 10, 2026 LLMKube tests on DGX Spark systems, active time-slicing sharply reduced throughput under load, GPU-offloaded copies used roughly a full model’s size each, and the tested router switched quickly but failed to pass GPU-offload settings to its child servers.
What does it mean to share one GPU?
“Sharing” can describe different mechanisms at different layers. A scheduler can assign the whole GPU to one model or allow multiple model pods to contend for it; an operator can unload one model and load another; or a serving runtime can switch models within one server. These approaches do not provide the same concurrency, isolation, memory behavior, or transition time.
| Approach | Where it operates | What happens | Best fit in Maher’s conditional guidance |
|---|---|---|---|
| Exclusive | Scheduler | One model owns the GPU. | One model at a time, when predictable access matters more than co-residency. |
| Shared (time-slicing) | Scheduler | Multiple model workloads share GPU execution over time; they can be active concurrently and contend for resources. | Genuinely concurrent demand when memory is sufficient, with prefill monitored. |
| ModelPool | Operator | One member is resident at a time; changing members replaces the serving pod and loads another model. | Alternating roles when the workload can tolerate the measured swap delay. |
| llamacpp-router | Runtime | One server switches models in process rather than replacing the pod. | The measured switch was quick, but the tested configuration was not suitable for serving because GPU offload did not reach child servers. |
| Partitioned (MIG) | GPU hardware | On supported GPUs, hardware instances partition GPU resources for separate workloads. | Consider when supported hardware and hardware isolation are requirements; Maher did not benchmark it on his systems. |
Maher’s comparison covers the first four approaches as tested or discussed in his August 10, 2026 article. MIG is a separate mechanism, not another name for scheduler time-slicing.
What happened to inference speed under time-slicing?
The results depend on whether another tenant is merely present or actively demanding GPU time. Maher’s baseline was a 27B dense model at Q4 on one GB10 with no competing workload. The figures below are from his LLMKube test, not general performance expectations for other GPUs, models, runtimes, or prompts.
#1 Best Overall
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- 0dB technology lets you enjoy light gaming in relative silence
- Dual BIOS switch lets you toggle between Quiet and Performance BIOS profiles
- Dual ball fan bearings last up to twice as long as sleeve bearing designs
| Test condition | Prefill | Decode | What the result indicates |
|---|---|---|---|
| 27B Q4 baseline on one GB10, no other workload | 768.85 tokens/s | 31.34 tokens/s | Single-model reference point in Maher’s test. |
| 12B model alone | 76.7–77.6 tokens/s | 27.3 tokens/s | Compared with the idle-co-tenant case. |
| 12B model with an idle 27B co-tenant | 77.9–80.9 tokens/s | 24.1–26.2 tokens/s | Maher reported the difference from the alone run as within measurement noise after suspending the co-tenant for the comparison. |
| Four active tenants | 16-fold decline versus the test’s comparison rate | 7.33 tokens/s; 4.3-fold decline | A 10,050-token prompt took 209 seconds in this four-tenant test. |
The practical distinction is between a model that is loaded but idle and several workloads that are actively competing. In this specific test, four active tenants hurt prefill more severely than decode. Maher therefore considered time-slicing more suitable for short-prompt chat than large-context work, while making clear that this is a conclusion from his setup rather than a guarantee for other inference stacks.
Does unified memory make duplicate GPU-offloaded models cheaper?
No—not in the controlled replication reported by Maher. He retracted an earlier claim that duplicate GPU-offloaded models cost about 0.24 times model size on unified memory, compared with 1.16 times on a discrete GPU. The earlier result had suggested that unified memory changed the usual advice; the replication did not support that conclusion.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
On the same node, the controlled test used a cached 15.66 GiB GGUF and three instances launched with --n-gpu-layers 99. The measured instance costs were 16,727 MiB, 16,433 MiB, and 16,444 MiB. Maher characterized this as about 1.03 times model size per instance, with memory growing linearly rather than being shared among GPU-offloaded copies.
Recommended Free Tools
| Configuration in Maher’s comparison | Reported memory per extra instance | Interpretation |
|---|---|---|
| GB10, 27B model, GPU offload | 1.03 times model size | Not shared between instances. |
| GB10, 3B model, GPU offload | 1.38 times model size | Not shared between instances. |
| RTX 5060 Ti, 3B model, GPU offload | 1.16 times model size | Not shared between instances. |
GB10, 3B model, CPU-resident weights using --n-gpu-layers 0 |
0.19–0.36 times model size | Sharing was observed while weights stayed CPU-resident. |
Maher attributed the CPU-resident result to memory-mapped page-cache sharing. In contrast, llama.cpp GPU offload allocates private CUDA device buffers and copies weights there, including on unified memory. In a same-box warm comparison, his GPU-offloaded run delivered 99.00 tokens/s decode, versus 11.36 tokens/s with zero GPU layers. That speed difference is specific to the compared runs; it does not make CPU-resident copies equivalent in performance.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
The retracted result came from comparing different instruments—node memory against nvidia-smi, which reported [N/A] for GB10 memory—without replication of a single observation under four-tenant load, and from comparing a 3B model with a 27B model. For capacity planning based on these tests, budget roughly one full model size for each GPU-offloaded copy, regardless of whether the system uses unified or discrete memory.
How long do model swaps take?
ModelPool: replace the resident pod
ModelPool keeps one member resident at a time. Maher measured a 122.0-second cold swap and a 53.6-second warm swap. The transition includes pod teardown, scheduling, container start, and model load, so it is a poor fit for workloads that need seamless or frequent model changes.
Rank #4
- Powered by Radeon RX 9070 XT
- WINDFORCE Cooling System
- Hawk Fan
- Server-grade Thermal Conductive Gel
- RGB Lighting
The tested operator behavior also mattered: drain-before-unload preserved an in-flight 1,200-token generation; concurrent requests were coalesced into one swap; and a fail-closed path left the incumbent model resident when its successor could not be reached. These are behaviors of the implementation Maher tested, not guarantees for every model-pooling system.
llamacpp-router: switch inside the runtime
Maher measured an in-process router switch at 4.5–4.7 seconds, compared with 53.6 seconds for ModelPool’s warm swap. But the router configuration in this test achieved 34.07 tokens/s prefill and 3.46 tokens/s decode, against 768.85 and 31.34 tokens/s respectively for the exclusive llama.cpp baseline. He traced the shortfall to router mode not passing --n-gpu-layers to child servers, leaving llama.cpp at its default of zero GPU layers. The faster switch therefore did not make this tested router configuration a sound serving choice.
Best Value
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
Is MIG better than time-slicing?
MIG and time-slicing address different isolation and allocation needs, and Maher’s test cannot establish which is faster for LLM inference. The GB10 and RTX systems in his setup reported nvidia.com/mig.capable=false, so he did not test MIG or measure it against time-slicing.
NVIDIA’s Multi-Instance GPU documentation describes MIG on supported GPUs beginning with Ampere architecture. It can create up to seven GPU instances with isolated paths through cache and memory resources, intended to support predictable throughput and latency as well as fault isolation. NVIDIA describes dedicated compute, memory, and bandwidth for instances; those resources are sized for a partition rather than freely borrowed by a neighboring idle instance. NVIDIA also documents container, Kubernetes, virtualization, and MIG-backed vGPU deployments. These facts describe the mechanism and deployment options, not measured results for Maher’s GB10 setup.
NVIDIA’s technical blog separately distinguishes software time-slicing and MPS from hardware MIG, and reports about 1.00 request/s per GPU for MIG versus 0.76 for time-slicing in its own voice-AI pipeline benchmark. That vendor-reported result uses a different workload and configuration from Maher’s LLM tests, so it is not a direct comparison or cross-validation. The blog also notes that time-slicing does not provide hardware fault isolation.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhich approach should you choose?
- One model needs the GPU: use exclusive assignment, as in Maher’s conditional guide.
- Models need to run concurrently: consider scheduler-level time-slicing if memory is sufficient. Account for the active-tenant slowdown, especially when prompts are long or prefill throughput matters.
- Models take turns and transitions can be slow: ModelPool may fit if the workload can absorb a warm swap on the scale measured in Maher’s test.
- You need quick in-process switching: do not infer serving suitability from the router’s swap time alone; verify that the runtime actually applies the required GPU-offload configuration to its child processes.
- You require hardware isolation: check whether the exact GPU supports MIG and how its fixed resource partitions fit the workload. The cited LLM test offers no MIG performance comparison.
How can you avoid misleading sharing benchmarks?
Maher’s benchmark corrections and implementation notes point to several ways a test can measure configuration mistakes or workload artifacts instead of the sharing strategy itself:
- Verify the full serving specification. Maher’s first baseline reached only 12.11 tokens/s decode after an InferenceService spec was cloned without its
speculativeDecodingfield. He notes that MTP was a first-class CRD field, not anextraArgsentry. Diff the full configuration when cloning a benchmark setup. - Separate cold and warm requests. In his cited GB10 example, cold prefill was 93 tokens/s versus 743 warm. A first request after model load should not be treated as a steady-state result.
- Check for prefix caching. Reusing prompts can let prefix caching skip prompt work, making a prefill comparison misleading.
- Measure memory with comparable instruments. Maher’s initial memory claim relied on unlike measurements, including
nvidia-smireporting[N/A]for GB10 memory. Track the same memory metric consistently across runs. - Confirm who owns scaling. In the tested operator setup, manually running
kubectl scalecould be reconciled back to the operator’s configured state. Change the owning configuration rather than assuming a manual scale will persist. - Replicate the exact model and workload. The retracted claim compared different model sizes and relied on one observation under load. Hold model, runtime, prompt, tenant count, and measurement method constant before drawing an architecture-level conclusion.
- Validate the test harness itself. Maher describes a synthetic load test that missed defects in an agent-and-reviewer harness, including a reviewer receiving the base branch rather than the actual diff. Confirm that the harness submits and evaluates the intended workload and code.
Maher’s August 10, 2026 LLMKube article reports throughput and memory behavior, not output accuracy. Its results therefore do not establish that sharing preserves accuracy or quality.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
