iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Running a GPU workload on Kubernetes requires more than adding a GPU request to a pod. A GPU worker node must have a compatible vendor driver and a device plugin (or an operator that manages those components). The plugin registers an extended resource with kubelet, Kubernetes schedules that resource, and the workload requests it in the pod’s limits. You must then choose between exclusive GPUs, hardware partitioning such as NVIDIA MIG, or shared time-slicing, each with different isolation and cost consequences.
What Kubernetes does—and does not—provide
Kubernetes supplies the scheduling model for accelerator resources, but it does not install a GPU vendor’s kernel driver, container runtime integration, or device plugin on your worker nodes. Those node-level components are what make physical devices visible to kubelet.
When a device plugin starts successfully, it discovers GPUs on the node and registers an extended resource. Kubelet then reports allocatable capacity to the Kubernetes control plane. A scheduler can place a pod only after that resource exists in the node’s advertised capacity. Typical resource names are nvidia.com/gpu for NVIDIA hardware and amd.com/gpu for AMD hardware, but the exact name is determined by the plugin and its configuration.
Kubernetes documents GPU scheduling support for AMD and NVIDIA as stable since Kubernetes v1.26. The stability of the scheduling API does not guarantee that every driver, operating system, GPU model, container runtime, or plugin version is compatible; those combinations remain vendor-specific.
#1 Best Overall
- AI Performance: 767 AI TOPS
- OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis
The node components you need
Vendor driver
The driver lets the operating system communicate with the physical accelerator. It must be installed on every worker node intended to run GPU workloads, and its version must support both the GPU model and the software stack used by your containers. A node can be labeled as a GPU node while still being unusable if the driver failed to load, so treat driver health as a prerequisite rather than a scheduling signal.
Device plugin
A device plugin runs on GPU nodes and communicates device availability to kubelet. It registers the extended resource, reports how many devices are allocatable, and supplies the container runtime with the device information needed by a scheduled pod. Without a functioning plugin, a node may physically contain a GPU but show no schedulable GPU resource.
Operator-managed integration
An operator bundles and manages more of this integration than a stand-alone device plugin. Depending on the vendor and configuration, an operator can coordinate driver installation, the device plugin, runtime settings, monitoring components, and sharing configuration. This reduces the number of independently maintained manifests, but it does not remove compatibility checks. Verify the operator’s supported Kubernetes versions, operating systems, driver releases, and GPU families before deploying it.
NVIDIA’s GPU Operator documentation covers its managed components and GPU-sharing configuration. AMD documents an operator that can deploy a device plugin and also provides a Dynamic Resource Allocation (DRA) driver path. Do not assume that an NVIDIA resource name, installation sequence, or sharing behavior can be copied to AMD hardware.
How a GPU request reaches a pod
For extended GPU resources, place the request in the pod or container’s limits. Kubernetes uses a limit as the request when no separate request is supplied. If both fields are present, they must be equal; a request without a corresponding limit is not allowed for these resources.
This minimal manifest illustrates the scheduling contract. Replace the resource key with the name registered by your plugin and use an image that contains the libraries required by your application.
apiVersion: v1
kind: Pod
metadata:
name: gpu-example
spec:
containers:
- name: workload
image: your-registry/your-gpu-image:tag
resources:
limits:
nvidia.com/gpu: 1
The value 1 means one allocatable unit as exposed by the plugin. For an exclusive-GPU setup, that normally corresponds to one whole physical GPU. Under a sharing configuration, the same-looking integer can represent a replica or partition, so read the plugin’s semantics before interpreting capacity or performance.
Free tools Windows power users keep installed
One-click scans. No signup required.
Preparing and validating a GPU node
- Select compatible hardware and an operating system. Confirm that the GPU model, kernel, container runtime, Kubernetes release, and vendor software are supported together. Record the intended resource name and whether the node will provide full GPUs, partitions, or shared replicas.
- Install and validate the vendor driver. Check the host with the vendor’s diagnostic tools and confirm that every intended device is visible. A driver that loads on the host but is incompatible with the container’s user-space libraries can still produce runtime failures.
- Deploy the device plugin or operator. Use the release and configuration documented for your vendor. Wait for its DaemonSet or operator-managed components to become ready on each GPU node.
- Inspect node capacity. Use
kubectl describe node <node-name>and check bothCapacityandAllocatablefor the expected extended resource. If the key is absent, scheduling cannot succeed regardless of pod YAML. - Run a small validation pod. Request one unit, observe that it is scheduled to a GPU node, and inspect the container logs and device visibility. Test the same image and runtime settings your production workload will use; a generic host check does not validate application libraries.
- Record placement rules. Document node labels, taints, tolerations, and any affinity rules that distinguish GPU types or environments. Keep this information with the workload manifests so a replacement node is configured consistently.
Scheduling GPUs in a mixed cluster
The ordinary scheduler considers the extended resource reported by kubelet along with CPU, memory, and other constraints. It does not infer that a node is suitable merely because its name or hardware description contains “GPU.” Resource registration is the authoritative capacity signal.
Node labels and affinity
Labels can steer a workload toward a particular accelerator family, generation, or node pool. For example, a cluster might label nodes with an organization-specific key such as accelerator: nvidia or gpu-class: inference, then use node affinity in the pod. The label names and values are not universal; use the labels that your cluster actually maintains.
Taints and tolerations
Dedicated GPU nodes are often tainted so ordinary workloads do not consume expensive capacity. A GPU workload must include a matching toleration, while the taint keeps unrelated pods away. A toleration permits scheduling onto a tainted node; it does not select a node by itself, so combine it with the GPU resource request and, when needed, affinity.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4. System Requirements: Minimum 850W PSU with 16-pin 12V-2x6 (12VHPWR) connector required. Verify before purchasing.
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability. Compatibility: 348mm (13.7") length, 3.6 slots, 4.3 lbs. Confirm case clearance and slot spacing. GPU bracket included.
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
Separate node pools
Separate pools make heterogeneous fleets easier to operate. They let you apply a driver and plugin configuration to a known GPU family, scale expensive capacity independently, and set different disruption policies. The scheduler still relies on the registered resource and your explicit constraints; there is no universal label or affinity recipe that fits every cluster version and fleet.
Recommended Free Tools
Choosing full-GPU access or sharing
Decide first whether a pod needs exclusive access to a complete GPU. Sharing can improve utilization for small or bursty jobs, but it changes what a resource request guarantees. Isolation, failure behavior, and performance must be evaluated for the actual workload rather than inferred from the number of replicas.
| Mode | What a pod receives | Isolation | When it fits | Important limitation |
|---|---|---|---|---|
| Exclusive full GPU | One whole device per allocated unit | Strongest practical separation between pods on that device | Training, latency-sensitive inference, or workloads that need the device’s full memory and compute | Idle periods leave capacity unused; each pod may require a costly node |
| NVIDIA MIG | A hardware-partitioned GPU instance | Provides memory and fault isolation between GPU instances | Known-size inference or batch jobs that can run within a selected partition profile | Available profiles and capacity depend on the specific GPU; a partition is not the same as a full GPU |
| Time-slicing | A software-scheduled replica that interleaves access to a device | Does not provide MIG’s memory or fault isolation | Interactive, development, or tolerant workloads with modest and variable demand | Requesting several replicas does not promise a proportional share of compute or memory |
NVIDIA describes MIG as providing memory and fault isolation between GPU instances. Its time-slicing mode exposes replicas that interleave workloads. Those replicas are a scheduling convenience, not a guaranteed performance fraction: two replicas can contend for the same physical resources, and a request for multiple replicas does not establish a linear throughput entitlement.
Understanding operator versus device-plugin responsibility
A stand-alone device plugin is the narrow component needed to advertise devices and inject them into containers. You remain responsible for installing and upgrading the driver, configuring the runtime, deploying the plugin, and coordinating any monitoring or sharing settings.
An operator automates a broader lifecycle. It may reconcile driver DaemonSets, plugin versions, runtime hooks, validation jobs, and sharing configuration. That can make upgrades repeatable across a node pool, but an operator adds its own version compatibility and failure modes. Before adopting one, decide whether your team wants its reconciliation model, whether it supports your OS and Kubernetes release, and how you will roll back a failed driver or operator upgrade.
AMD’s documented path illustrates why vendor-neutral assumptions are risky: its operator supports a device-plugin approach and a DRA driver alternative. The object types, resource names, sharing controls, and troubleshooting commands can differ from NVIDIA’s. Follow the documentation for the exact hardware and software combination deployed in your cluster.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What GPU capacity really costs
GPU-enabled worker nodes are expensive infrastructure, and kOps’ GPU setup guidance explicitly cautions operators to minimize workloads running on GPU instances. The available material does not establish a current provider price, a cheapest region, or a cost-per-workload figure, so a credible budget must use your own region, hardware choice, utilization, and billing terms.
| Cost driver | Questions to answer |
|---|---|
| GPU instance or server | What is the purchase or hourly rate for the exact GPU, host type, and region? |
| Utilization | How many GPU-hours are productive, and how much capacity is idle while pods wait for data or remain scheduled? |
| Power and cooling | For owned hardware, what electricity, rack, and cooling capacity does the system require? |
| Storage and networking | Will datasets, checkpoints, and model images require high-throughput disks or network links that add cost? |
| Operations | What staff time is needed for drivers, plugin or operator upgrades, observability, security patches, and incident response? |
| Sharing overhead | Can time-slicing or MIG raise utilization without violating latency, isolation, or reliability requirements? |
Calculate an internal cost per useful workload unit—such as a training run, inference request, or processed batch—using measured utilization rather than the nominal GPU count. Include idle time caused by queueing, data preparation, and failed jobs. Sharing may reduce idle capacity, but it can also increase contention and complicate troubleshooting; it is not automatically a cost reduction.
Troubleshooting common failures
The pod remains Pending with an insufficient GPU message
- Inspect node
Allocatableand verify that the exact resource key in the pod matches the plugin’s registered key. - Check whether all advertised units are already allocated, including hidden consumers in another namespace.
- Review node affinity, selectors, taints, and tolerations; a pod can have available GPU capacity but no node satisfying its other constraints.
- Confirm that the sharing mode has not changed the number or meaning of allocatable units.
No GPU resource appears on the node
- Check the plugin or operator pod status and logs on that node.
- Validate the host driver and confirm that the operating system detects the device.
- Compare the node’s Kubernetes, kernel, runtime, driver, and plugin versions with the vendor support matrix.
- After correcting the node, wait for the plugin to re-register and then inspect capacity again; editing the workload alone cannot create the resource.
The pod schedules but the application cannot use the device
- Verify that the image contains the user-space libraries expected by the driver and framework.
- Check container runtime integration and the device files or environment supplied by the plugin.
- Run the vendor’s diagnostic utility inside the same image and security context as the application.
- Compare a known-good validation image with the production image to isolate library or application errors.
Shared workloads interfere with one another
Measure latency, memory pressure, and failure behavior under realistic concurrency. If workloads require memory or fault isolation, move them to MIG partitions or exclusive GPUs where supported. Do not solve contention by merely increasing the requested number of time-sliced replicas; that changes scheduling admission, not the guaranteed fraction of physical compute.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesQuick Recap
A practical decision checklist
- Identify the exact GPU model, operating system, Kubernetes version, and container runtime.
- Choose a supported driver and decide whether a stand-alone plugin or an operator best fits your upgrade process.
- Verify the registered resource name and inspect node
CapacityandAllocatable. - Put the GPU quantity in
limits; if a separate request is used, make it equal. - Define labels, affinity, taints, and tolerations for heterogeneous or dedicated GPU pools.
- Select exclusive access, MIG, or time-slicing according to isolation and workload behavior—not just nominal utilization.
- Test the production image on a disposable validation pod before scaling out.
- Model total cost with your own hardware or instance prices, utilization, power, storage, networking, and operating effort.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
