Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
When every request to a Kubernetes cluster fails, including the one meant to repair the problem, the first job is to determine which of two conditions you are facing. Either the API server could not reach the admission webhook, or the webhook was reached and explicitly refused the request. The recovery path is different for each, and applying the wrong one wastes time or weakens your policy controls.
Two conditions that look the same from the command line
An admission webhook can block a request in two very different ways. In a call failure, the API server never gets a usable answer: the connection is refused, the request times out, or the response is malformed. In an explicit denial, the webhook answers, and its answer says the object is not allowed. Kubernetes handles these differently, and that difference decides your fix.
The official Dynamic Admission Control documentation states the rule directly: “The API server does not apply a failure policy when the webhook is reached successfully, and the webhook implementation has explicitly rejected the request (by specifying allowed: false in the response).” In other words, failurePolicy only governs what happens when the call to the webhook fails. It never overrides a valid rejection.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →| Question | Call failure (webhook unreachable or unusable) | Explicit denial (webhook answered with allowed: false) |
|---|---|---|
| Typical error wording | Mentions failing to call the webhook, such as a timeout, connection error, or invalid response from the webhook service | Mentions a denial from the webhook, usually including the webhook name and a message the webhook supplied |
Does failurePolicy matter? |
Yes. Fail rejects the request; Ignore lets it continue |
No. The request is rejected whatever the policy says |
Does switching to Ignore help? |
Yes, it can clear the lockout while the endpoint is down | No. The webhook still refuses the object |
| Correct fix | Restore the webhook service, or temporarily change its failure handling | Change the object so it satisfies the policy, or change the policy with its owner’s approval |
Step 1: Read the exact API error
- Re-run the failing command with full output, for example
kubectl apply -f <file> -v=6, and copy the error text. - Look for wording about calling the webhook (a timeout, a connection problem, or a failure to call the webhook service). If that is present, treat the case as a call failure.
- If the message names a webhook and carries a denial reason supplied by that webhook, treat the case as an explicit denial.
- If the message is ambiguous, check the webhook’s Pods and Service first. A healthy, reachable webhook that still rejects the object is an explicit denial, and no failure-policy change will help.
Step 2: Find the webhook configuration that matches the request
Admission webhooks are declared in cluster-scoped objects. List them with:
#1 Best Overall
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfiguration <name> -o yamlFor each entry under webhooks, check the fields that decide whether your recovery request is sent to it at all:
- rules: the API groups, API versions, resources, and operations (CREATE, UPDATE, DELETE, CONNECT) that trigger the webhook. A fix that updates a Deployment or a ConfigMap is matched only if those resources appear here.
- namespaceSelector: which namespaces are covered. Confirm whether the namespace you are repairing, or the webhook’s own namespace, falls inside it.
- objectSelector: label-based matching on the object itself. An object you have relabelled may suddenly be in or out of scope.
- matchConditions: CEL expressions that must all evaluate to true before the webhook is called. Check these, because they can narrow the match beyond the rules.
- failurePolicy and timeoutSeconds: together they determine what happens when the call fails and how long the API server waits before deciding it has failed.
Kubernetes guidance is to keep webhook scope as narrow as the policy allows. A broad rule that matches every resource in every namespace is the most common reason a repair request, which is usually an ordinary object, still reaches a webhook that is failing.
Step 3: When the endpoint is unavailable, understand the failure policy
For a call failure, the webhook’s failurePolicy decides the outcome. The documented default for admission webhooks is Fail, which rejects the request whenever the webhook cannot be called. Ignore allows the request to continue as if the webhook had not been consulted.
| Setting | When the webhook cannot be called | Trade-off |
|---|---|---|
Fail (documented default) |
The request is rejected | Protects policy strictly, but a down webhook can block the whole cluster, including repair operations |
Ignore |
The request continues without that webhook’s check | Keeps the cluster writable during an outage, but unchecked objects can be admitted |
Kubernetes good-practice guidance also suggests that mutating webhooks consider failing open, and that a validating admission check verify the final object state. That arrangement means a temporary mutator outage does not block compliant resources, while the final state is still enforced. Use this as a design principle, not as a bypass: Ignore never changes the outcome of an explicit denial.
If you must act during an outage, first restore the webhook. Check that its Pods are running and that its Service has endpoints:
kubectl get pods -n <webhook-namespace>
kubectl get endpoints <webhook-service> -n <webhook-namespace>
kubectl describe pod <webhook-pod> -n <webhook-namespace>If the webhook cannot be restored quickly, temporarily set its failurePolicy to Ignore. The index in the JSON patch path depends on the position of the webhook in the list, so confirm it first:
kubectl patch validatingwebhookconfiguration <name> --type=json
-p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'Restore Fail once the webhook is healthy again, and record the change, because a silent Ignore setting removes enforcement for everything the webhook was meant to check.
Free tools Windows power users keep installed
One-click scans. No signup required.
Step 4: Look for recovery dependency loops
Some lockouts are self-inflicted. The webhook blocks the very resources needed to bring it back, and no repair can get through. Three patterns cause most of these cases.
The webhook blocks its own Pods
If a webhook runs inside the cluster and intercepts the creation or rescheduling of its own Pods, and those Pods lack a property the webhook requires, the webhook cannot start new replicas. Kubernetes recommends avoiding self-mutations of this kind. The practical fix is to exclude the webhook’s own namespace from matching, for example with a namespaceSelector that uses the automatically set kubernetes.io/metadata.name label:
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["webhook-system"]Make this change in the configuration, not during an outage with the same webhook still active, and review the scope change with the platform team before applying it.
Two webhooks validate each other
Two webhooks can each depend on the other to admit a change. If one of them fails, the other cannot admit the change needed to restart it. Break the cycle by narrowing the rules of one webhook so it no longer matches the other’s resources, or by making one of them fail open.
A webhook intercepts a cluster add-on it depends on
A webhook may depend on a cluster add-on, such as a DNS or metrics component, that its own rules intercept. Exclude the add-on’s namespace or resources from the webhook’s matching so the add-on can be updated and restarted while the webhook is down.
Best Value
Step 5: Recover by branch
- Explicit denial, object is legitimate: adjust the object to satisfy the policy the webhook enforces, or ask the policy owner to change the rule. Setting
Ignorewill not work here. - Explicit denial, the fix itself is blocked: check whether the fix matches a rule in the webhook’s configuration. If it does, you may need the policy owner to exclude the affected namespace or object from the webhook’s selectors.
- Call failure, webhook can be restored: restore its Pods and Service endpoints, confirm
kubectl describeshows them healthy, then retry the change. - Call failure, webhook cannot be restored: set
failurePolicytoIgnoretemporarily, apply the repair, then restoreFail. - Last resort, cluster is fully locked: remove the webhook configuration with
kubectl delete validatingwebhookconfiguration <name>(or the mutating equivalent). This removes enforcement entirely, so reapply the configuration, after the repair, from version control. Your cluster’s admin credentials and the object’s own RBAC still apply.
Whether a configuration object can itself be blocked depends on whether the webhook’s rules cover admissionregistration.k8s.io resources. If they do not, the configuration can usually be patched or deleted even when the webhook rejects other changes.
Preventing the next lockout
- Keep webhook rules as narrow as possible, and always use
namespaceSelectorto exclude the webhook’s own namespace andkube-systemunless the policy explicitly requires them. - Test changes to
failurePolicyandtimeoutSecondsagainst a non-production cluster that runs the same Kubernetes version. - Run the webhook with more than one replica and a PodDisruptionBudget so a single node drain does not take it offline.
- Keep a copy of every webhook configuration in version control so it can be restored quickly.
- Document which resources each webhook intercepts, so an operator can identify a dependency loop before it locks the cluster.
Verify defaults and field behavior against your cluster’s Kubernetes version and the webhook’s own documentation, because the exact error text and available fields vary between releases and between the webhook implementations you install.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
