Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Why is my Kubernetes pod stuck in Pending waiting for a GPU? Start with the Pod’s latest FailedScheduling event. It usually states whether no node has the requested GPU resource or whether labels, affinity, taints, topology, or another scheduling condition excludes otherwise capable nodes. Then verify the exact GPU resource name and quantity, compare node capacity with allocatable devices, and check the driver and device-plugin deployment. Buying more hardware will not correct a scheduler configuration error.
What “Pending” means for a GPU Pod
A Pod in Pending has been accepted by the API server but is not yet running on a node. For a GPU workload, the scheduler must find a node that simultaneously satisfies the Pod’s resource request and every placement rule. A cluster can contain GPU-equipped machines while the Pod remains pending if those machines advertise a different resource name, have no free allocatable device, carry an untolerated taint, or fail the Pod’s selectors or affinity rules.
Do not confuse a scheduling wait with a later runtime failure. Once a Pod has a node assignment, it has passed scheduling; failures such as a container runtime error, missing libraries, or an application crash require a different investigation.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesRun this evidence-first triage
- Identify the namespace and node assignment. Replace the placeholders with the actual values:
kubectl get pod <pod> -n <namespace> -o wideA blank
NODEcolumn normally means the Pod has not been scheduled. - Read the complete description and Events section.
kubectl describe pod <pod> -n <namespace>Look at the newest events first, especially messages beginning with
FailedScheduling. Kubernetes emits scheduling events while an unschedulable Pod remains pending. - Record the exact resource key and quantity in the Pod spec. Check the rendered object rather than relying on a Helm value or a local file:
kubectl get pod <pod> -n <namespace> -o yamlInspect
spec.containers[].resources.limitsand, if present,requests. - List node capacity and allocatable resources.
kubectl get nodes kubectl describe node <node>In the output, compare
CapacityandAllocatable, and inspect the node’s labels and taints. - Check the GPU driver and device-plugin Pods. Find the DaemonSet and its Pods in the namespace used by your platform, then inspect status and logs:
kubectl get daemonsets,pods -A kubectl describe pod <device-plugin-pod> -n <plugin-namespace> kubectl logs <device-plugin-pod> -n <plugin-namespace>Use the actual names from your installation; vendors and distributions choose different namespaces and labels.
Capture the event text before changing anything. A message such as Insufficient nvidia.com/gpu points to resource fit; a message naming NodeAffinity or TaintToleration points to placement filtering. The distinction determines the next command.
#1 Best Overall
- Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
- With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
- Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
- Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
- Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.
Verify the GPU resource request
Use the resource name the node actually advertises
In the traditional device-plugin model, a vendor plugin registers an extended integer resource with kubelet. Common names include nvidia.com/gpu and amd.com/gpu, but your cluster may expose a different vendor-specific key. Copy the key exactly, including punctuation and capitalization, from the node’s resource list. A request for gpu does not match nvidia.com/gpu.
Extended device resources are integer resources. They cannot be overcommitted, so a request for two devices requires two allocatable devices on one eligible node; Kubernetes cannot split a fractional device between Pods.
Put the GPU in limits
Kubernetes’ GPU scheduling documentation states: “GPUs are only supposed to be specified in the limits section.” A GPU limit can be used without a separate request. If both fields are set, they must be equal; a request without a limit is unsupported for this resource type.
apiVersion: v1
kind: Pod
metadata:
name: gpu-check
spec:
restartPolicy: Never
containers:
- name: workload
image: your-image:tag
resources:
limits:
nvidia.com/gpu: 1The example is valid only if a node advertises nvidia.com/gpu. Substitute the key reported by your cluster. Do not assume that changing the image, command, or CPU request will create a missing GPU resource.
Check quantity and hidden defaults
Compare the requested count with each eligible node’s allocatable count. A Pod requesting four GPUs cannot fit on a node with two, even when the cluster has several two-GPU nodes; standard extended resources are allocated on a single node. Also inspect admission-generated fields and the final Pod YAML, because a mutating webhook, chart value, or sidecar may add an unexpected GPU request.
Check capacity, allocatable devices, and health
Understand capacity versus allocatable
When a device plugin registers devices, kubelet reports them in the node status. Capacity is the total registered count; Allocatable is what the scheduler can assign after kubelet reservations and device health are applied. A quick view of node status is:
Rank #2
- Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
- Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
- Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
- Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
- Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
kubectl get node <node> -o jsonSearch the returned status.capacity and status.allocatable maps for the exact GPU key. You can also use kubectl describe node <node> for a readable report.
Interpret a capacity/allocatable mismatch
- Both fields show no GPU key: kubelet has not received a registered resource. Investigate drivers, the plugin DaemonSet, registration, and plugin logs before editing the Pod.
- Capacity is positive but allocatable is lower: some devices may be assigned, reserved, or marked unhealthy. Check existing Pods and plugin health reporting.
- Allocatable is positive but no free device remains: the scheduler may report
Insufficient <vendor>/gpu. Remove only workloads you intentionally no longer need; do not delete running GPU jobs as a diagnostic shortcut.
When a device is marked unhealthy, the allocatable count can fall while capacity stays unchanged. That state explains why a new Pod cannot obtain a device even though the node still appears to have the original hardware count.
Check every eligible node, not only one
Run the node inspection across the cluster. A GPU may be available on a node that fails another filter, so the event can still report an apparent shortage. Resource fit and placement rules are evaluated together; confirm both before adding machines.
Find placement rules that filter out GPU nodes
Node selectors and required affinity
Inspect spec.nodeSelector and spec.affinity.nodeAffinity in the Pod. Required rules must match labels exactly, including values such as GPU model, operating system, zone, or accelerator class. Compare them with:
kubectl get nodes --show-labels
kubectl get node <node> --show-labelsA typo, a label removed during node provisioning, or a requirement for a model absent from the cluster can leave zero eligible nodes even when GPUs are idle. Required node affinity is a hard filter; preferred affinity influences ranking but does not by itself make a node ineligible.
Taints and tolerations
GPU nodes are often tainted so general workloads do not consume accelerator capacity. Display taints with:
Rank #3
- 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
- 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
- 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
- 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
- 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.
kubectl describe node <node>For every NoSchedule or NoExecute taint that should be tolerated, add a matching toleration to the Pod. The key, effect, and operator semantics must match. A toleration does not force placement on that node; it only prevents the corresponding taint from filtering it out.
spec:
tolerations:
- key: "accelerator"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"Use the taint actually shown on your node instead of copying this illustrative value.
Other scheduler filters
The scheduler’s NodeResourcesFit, NodeAffinity, and TaintToleration plugins commonly explain GPU placement failures. Pod anti-affinity, inter-Pod affinity, topology spread constraints, host-port conflicts, and volume topology can also remove candidates. The event message may list several filter reasons; fix the one that applies to the current Pod rather than broadly disabling scheduling rules.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Interpret common scheduling messages
| Event or symptom | What it suggests | Next check |
|---|---|---|
Insufficient nvidia.com/gpu (or another vendor key) |
The requested integer quantity does not fit on any currently eligible node. | Verify the exact key and count, then compare allocatable devices on nodes that also satisfy selectors, affinity, taints, and topology. |
| No GPU key in node capacity | The resource was not advertised to kubelet. | Check vendor drivers, device-plugin registration, DaemonSet readiness, and plugin logs. |
node(s) didn't match Pod's node affinity/selector |
GPU nodes exist but their labels do not satisfy a required rule. | Compare the rule with live node labels; correct either the label source or the Pod requirement. |
node(s) had untolerated taint |
A tainted GPU node was filtered out. | Inspect the node taint and add a narrowly scoped matching toleration if the workload is allowed there. |
TopologyAffinityError |
A node-local or topology-manager constraint rejected the request, possibly because local resources are insufficient. | Inspect Pod events, kubelet logs, node topology-manager policy, and device locality. |
| Pod has a scheduling gate | The Pod was deliberately withheld from scheduling. | Inspect spec.schedulingGates and the controller or admission component responsible for removing the gate. |
The first row does not prove that the entire cluster lacks GPUs. It may mean that GPUs are available only on nodes excluded by another filter. Confirm the node configuration and the full event message.
Investigate scheduling gates and Pod Scheduling Readiness
A scheduling gate intentionally keeps a Pod out of the scheduler’s consideration until another component declares it ready. Inspect the field directly:
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.schedulingGates}'If gates are present, identify the controller, admission webhook, or workflow that added them. Removing a gate manually can bypass a required preparation step and create a different failure, so use the owning system’s documented procedure. Pod Scheduling Readiness is stable from Kubernetes v1.30; older clusters or older controllers may not support the same behavior. Confirm the cluster version before relying on this field.
Rank #4
- 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
- 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
- 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
- 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
- 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.
Resolve topology-related GPU waits
TopologyAffinityError is not the same as a generic cluster-wide GPU shortage. Kubernetes can reject a request when a node does not have enough resources in the required topology locality or when the kubelet’s topology-manager policy rejects the combination of devices. This is especially relevant when CPUs, hugepages, network devices, and GPUs must be aligned on the same NUMA domain.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- Read the Pod’s Events and status for the exact topology error.
- Identify the node candidates and inspect kubelet logs on those nodes.
- Check the node’s topology-manager policy and the device plugin’s topology information.
- Reduce or change the locality requirement only if the workload permits it; otherwise place the Pod on a node whose local device layout satisfies the request.
Do not “fix” a topology error by merely increasing the cluster’s total GPU count. Additional devices in another NUMA domain or zone may not satisfy the Pod’s locality requirement.
If the cluster uses DRA, inspect claims and preparation
Clusters using Kubernetes Dynamic Resource Allocation (DRA) do not necessarily represent an accelerator as a traditional extended resource in the Pod. The Pod can reference a ResourceClaim, while a DRA driver selects, binds, and prepares the device. Inspect the Pod and its claims:
kubectl get pod <pod> -n <namespace> -o yaml
kubectl get resourceclaim -A
kubectl describe resourceclaim <claim> -n <namespace>Look for an unbound claim, a driver that is not running, or a device-preparation condition that has failed or not completed. The claim’s events and conditions are more useful than changing a conventional nvidia.com/gpu limit when the workload is intentionally using DRA.
DRA device-binding conditions are documented as beta in Kubernetes v1.36 documentation. Feature gates, API versions, and driver support must match the actual cluster; do not apply a DRA manifest copied from a newer release to an older control plane without checking compatibility.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair missing or unhealthy device-plugin advertisement
Confirm the drivers are installed on every intended node
Kubernetes relies on the vendor’s host driver and a corresponding device plugin. A container image containing GPU libraries cannot create a node resource if the host driver is absent or unusable. Check the node’s driver status using your vendor’s supported diagnostic command and inspect kernel or kubelet logs when the plugin cannot initialize.
Best Value
- 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
- 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
- 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
- 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
- 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
Check the DaemonSet and registration
A device-plugin DaemonSet should run on each node intended to host that GPU type. Verify that its Pods are scheduled, ready, and not repeatedly restarting:
kubectl get daemonsets -A
kubectl get pods -A -o wide
kubectl describe pod <device-plugin-pod> -n <plugin-namespace>
kubectl logs <device-plugin-pod> -n <plugin-namespace> --all-containersThe device-plugin framework lets a plugin register with kubelet, list devices, and report health changes. Registration errors, permission problems, incompatible driver versions, or a failed health check can prevent the GPU key from appearing or can reduce allocatable devices. Resolve the plugin’s reported error, then verify node status again; restarting the workload alone does not refresh a missing registration.
Account for version-sensitive health fields
Device-resource health status is described as beta in Kubernetes v1.36 documentation. Field names and observability may differ on older clusters. Treat a missing health field as an availability limitation, not proof that every device is healthy; use node status, plugin logs, and the vendor’s diagnostics together.
Free tools Windows power users keep installed
One-click scans. No signup required.
Separate scheduling from post-assignment failures
Run kubectl get pod -o wide again after each change. If a node name appears and the Pod moves to ContainerCreating, CrashLoopBackOff, or another post-scheduling state, stop applying scheduler fixes. Examine container events and logs instead:
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers
kubectl logs <pod> -n <namespace> --previousAt this stage, likely areas include runtime hooks, driver/library compatibility, permissions, image architecture, or an application-level CUDA error. An unhealthy device can affect an already assigned Pod differently from the scheduler’s allocatable count for new Pods, so correlate the failure with the assignment time and node health.
A practical decision path
- No node assigned: read the newest
FailedSchedulingevent. - Event says insufficient GPU: match the resource key and integer quantity, then compare allocatable devices on eligible nodes.
- Event names affinity, selector, taint, or spread: compare the Pod rules with live node labels, taints, and topology constraints.
- GPU key is absent from node status: troubleshoot host drivers, plugin registration, DaemonSet health, and device health.
- A scheduling gate is present: identify the component that owns and removes it.
- DRA is in use: inspect the ResourceClaim and DRA driver conditions rather than adding a legacy GPU limit.
TopologyAffinityErrorappears: investigate kubelet topology policy and node-local device layout.- A node is assigned: switch to runtime, driver, and application diagnostics.
This sequence keeps the evidence tied to the scheduler decision. It also prevents a common mistake: adding hardware when the real issue is an incorrect resource key, an untolerated taint, or a device plugin that never registered.
Quick Recap
Version and feature checks
- The core GPU scheduling and device-plugin framework is stable in Kubernetes v1.26 and later, but vendor driver and plugin compatibility still matters.
- Pod Scheduling Readiness is stable from v1.30; verify the control-plane version and the controller that manages gates.
- DRA device-binding conditions and device-resource health status are described as beta in v1.36 documentation; confirm feature support before depending on those fields.
- Always compare examples with the API versions and feature gates enabled in your cluster. A command can succeed while a newer status field is unavailable, or a manifest can be rejected on an older release.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
