Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

First identify where the failure occurs: kubelet startup, Pod sandbox/container creation, or the application after the container starts. On a systemd-based cgroup v2 node, Kubernetes normally needs a Linux kernel 5.8 or newer, a cgroup-v2-capable runtime, and matching systemd cgroup drivers in kubelet and the runtime. Verify those facts before changing configuration; a blind kubelet restart is not a reliable repair for a node that already ran Pods under the old hierarchy.

What “fails to start” means in a cgroup v2 migration

cgroup v2 changes how Linux exposes resource-control hierarchies. Kubernetes creates cgroups for the kubelet-managed Pod and relies on the container runtime to create the container-level cgroups. A problem in either layer can produce a similar-looking event, but the repair is different.

Failure layer Typical evidence What it usually means
Kubelet or node kubectl get nodes shows NotReady, or kubelet will not stay running; journalctl -u kubelet contains startup or cgroup errors. The node prerequisites, kubelet configuration, or cgroup hierarchy are wrong. Pods cannot be scheduled or managed normally.
Runtime and Pod sandbox Pod events show FailedCreatePodSandbox, a container-creation error, or a CRI error while the Pod is being created. The runtime cannot create the requested cgroups, or kubelet and runtime disagree about the cgroup driver or hierarchy.
Application process The container is created and then exits, restarts, or is marked OOMKilled. The image, application, agent, or memory-sizing logic may not understand cgroup v2. This is not the same as a sandbox-creation failure.

Record the exact event and timestamp before editing anything. The distinction prevents you from “fixing” kubelet settings when the application is the component that actually fails.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Confirm that the node really uses cgroup v2

Run this on every affected Linux node, not only on the control-plane host:

stat -fc %T /sys/fs/cgroup/
  • cgroup2fs indicates a cgroup v2 hierarchy.
  • tmpfs indicates cgroup v1.

A mixed cluster is possible: one node may still boot with v1 while another uses v2. Check each node that can receive the failing workload. If the result is tmpfs, you have not diagnosed a v2 runtime failure on that node; you have a v1 node that needs its own compatibility decision.

Capture the facts that determine support before changing a driver:

  • Linux distribution and release.
  • Kernel version, using uname -r. Kubernetes’ cgroup v2 requirements specify Linux kernel 5.8 or later.
  • Kubernetes server and kubelet versions.
  • Container runtime and version, such as containerd or CRI-O.
  • Whether systemd is the host init system.
  • The effective kubelet and runtime cgroup-driver settings, including command-line overrides and generated files.

The Kubernetes documentation lists containerd 1.4 or later and CRI-O 1.20 or later as examples of runtimes with cgroup v2 support. Those are examples, not a promise that every combination of runtime, Kubernetes release, distribution packaging, and configuration is compatible. Check the compatibility information for the exact versions installed on your cluster.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Collect evidence before making a repair

Use both the cluster view and the node view. These commands are a workflow to run on your cluster; their output will identify which branch below applies.

  1. Check node readiness with kubectl get nodes -o wide.
  2. Inspect the affected Pod and its events with kubectl describe pod <pod-name> -n <namespace>. Look for FailedCreatePodSandbox, CRI messages, image errors, or an application exit code.
  3. Read kubelet logs with journalctl -u kubelet --since "30 min ago".
  4. Read the runtime service logs. The unit is commonly containerd or crio; use the unit name installed on the node, for example journalctl -u containerd --since "30 min ago".
  5. Save the exact error, the node name, Kubernetes version, runtime version, and cgroup filesystem result. A driver mismatch can look different across runtime releases, so the version is part of the diagnosis.

Do not infer a root cause from the phrase “cgroup” alone. A failed image pull, an admission policy, a missing device, and a cgroup hierarchy error can all occur during Pod creation.

Align kubelet and runtime cgroup drivers

For a systemd-based host running cgroup v2, configure both sides to use the systemd driver. Letting kubelet use cgroupfs while systemd manages other cgroups creates two independent managers. Under resource pressure, they can produce conflicting hierarchies and unstable node behavior. A driver mismatch between kubelet and the runtime can also cause container-creation failures or inconsistent resource accounting.

Verify the kubelet’s effective setting

The kubelet configuration field is:

cgroupDriver: systemd

On kubeadm-managed clusters, kubeadm defaults an unset kubelet driver to systemd for clusters created with kubeadm 1.22 and later. The generated per-node configuration is normally /var/lib/kubelet/config.yaml. Treat that file as evidence, not as the only authority: a systemd unit argument, drop-in, distribution wrapper, or another generated file can override it. Inspect the running unit and the kubelet logs to confirm what is actually effective.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Do not edit every node simultaneously. Make the change on one cordoned test node, restart services during a maintenance window, and verify that a test workload can create a sandbox and enforce its requested resources.

Configure containerd 1.x

For containerd 1.x, Kubernetes documents the runtime option in the CRI plugin table:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

Place the option in the configuration used by the running service. Package defaults may use a different path or may generate the file at boot. After changing it, restart containerd in a planned maintenance window and immediately inspect the service log for configuration or plugin errors.

Configure containerd 2.x

Containerd 2.x uses a different plugin table:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

Using the 1.x table in a 2.x configuration can leave the effective setting unchanged. Confirm the installed major version and the active configuration before selecting a table. Also confirm that the CRI integration plugin is enabled; some distribution packages disable it, in which case kubelet cannot use that runtime configuration even if the TOML syntax is correct.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Check CRI-O

Kubernetes documentation states that CRI-O’s default cgroup manager is systemd. Verify the effective CRI-O configuration rather than assuming a package default survived an upgrade, and keep kubelet’s cgroupDriver aligned with it. A manually selected cgroupfs manager or a vendor-specific drop-in still needs to be changed deliberately.

Restart order and validation

Changing a runtime setting without restarting the service leaves the old process running with the old hierarchy rules. Plan the operation as a node maintenance task:

  1. Cordon the node so new Pods are not placed on it: kubectl cordon <node-name>.
  2. Follow your workload-availability policy to drain it, if the node is already serving workloads: kubectl drain <node-name> --ignore-daemonsets may need additional flags for local storage, unmanaged Pods, or disruption budgets.
  3. Apply the kubelet and runtime settings appropriate to the installed versions.
  4. Restart the runtime, then kubelet, using the service manager for that distribution. Watch both service logs while they start.
  5. Run stat -fc %T /sys/fs/cgroup/ again and verify that the expected hierarchy is still mounted.
  6. Uncordon only after the node is Ready: kubectl uncordon <node-name>.
  7. Create or recreate a small test Pod and inspect its events, status, and resource behavior before returning production workloads.

Do not treat a successful service restart as proof that the repair worked. The decisive test is a new Pod sandbox and container on the corrected node.

Why an existing node may need replacement

Changing the cgroup driver on a node that has already joined a cluster is more disruptive than changing a new node image. Existing Pods and their sandboxes were created under the old hierarchy semantics. After the runtime switches drivers, the runtime may be unable to recreate those sandboxes, even though both services start cleanly.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Kubernetes explicitly cautions that restarting kubelet may not solve errors caused by changing a joined node’s driver. The safer operational choice is usually:

  • Cordon and drain the node according to your disruption and stateful-workload procedures.
  • Replace or reinstall the node with the intended cgroup v2 and systemd configuration already present.
  • Rejoin it to the cluster and validate a test workload before allowing normal scheduling.

If replacement is impossible, use the version-specific kubeadm migration procedure for your cluster release. Expect to remove and recreate affected Pods and to plan for stateful workloads, local volumes, PodDisruptionBudgets, and daemon sets. The general runtime guidance does not guarantee that an in-place kubelet restart will repair old sandboxes.

Runtime cgroup-driver discovery: when it helps and when it does not

The CRI design allows kubelet to obtain the cgroup driver from the runtime’s RuntimeConfig. When that capability is supported and enabled, the runtime-reported value can take precedence over the cgroupDriver value in kubelet configuration. This can remove duplicate settings, but it is not universal.

  • Support depends on the Kubernetes release, feature gates, runtime implementation, and distribution packaging.
  • A runtime may support the API but not have the relevant feature enabled.
  • Removing an explicit kubelet setting before verifying discovery can create a new mismatch.

Check the documentation for the exact Kubernetes and runtime versions in use. Until support is confirmed and observed in logs or runtime output, keep explicit, matching systemd settings on both sides.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

If the container starts but the application exits or is OOM-killed

A successful sandbox and container creation moves the investigation above the runtime layer. Inspect:

  • kubectl get pod <pod-name> -n <namespace> -o wide and kubectl describe pod for the termination reason, exit code, restart count, and memory events.
  • The application’s own logs and startup flags.
  • Any sidecar, monitoring agent, or profiler that reads cgroup files directly.
  • The Pod’s memory request and limit, and whether the process sizes its heap from the cgroup limit.

Kubernetes warns that older releases in the Node.js 18 line do not reliably detect cgroup v2 memory limits. Such a process can read host memory, size its heap too aggressively, and then be killed when the Pod limit is enforced. Update to an application runtime with reliable cgroup v2 detection or set an explicit heap limit appropriate for the Pod’s memory limit. Apply the same scrutiny to agents that inspect /sys/fs/cgroup; a cgroup-v1-only parser can misreport limits even when the application itself is current.

Do not “fix” an application OOM by changing the kubelet driver. First establish that the container reached the running state and that the termination was caused by the process or the kernel’s memory controller.

Rootless kind has separate startup limits

Rootless kind environments have constraints that are independent of a conventional, system-level Kubernetes node. A workload can fail to start because of the user’s Podman PID limit or because the host has too few inotify watches or instances. Check the rootless kind guidance for:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Podman’s per-user PID limits.
  • fs.inotify.max_user_watches.
  • fs.inotify.max_user_instances.

Raise those limits only after confirming the corresponding error. Do not generalize a rootless kind PID or inotify problem to a managed cluster using a system runtime; the symptoms may overlap, but the fix and ownership are different.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

If the node is still on cgroup v1

If stat -fc %T /sys/fs/cgroup/ returns tmpfs, investigate why the node did not switch. cgroup v1 is deprecated in Kubernetes 1.35, and kubelet no longer starts on a v1 node by default. An administrator can temporarily set failCgroupV1: false as a fallback for a v1 node, but that does not make it a cgroup v2 node and is not the solution to a failed v2 migration.

Treat this as a platform migration decision: either complete the distribution and boot-parameter change to v2 with a supported runtime, or deliberately keep the node on a supported v1 configuration while planning the upgrade. Do not mix a v1 fallback setting into a v2 repair plan.

Choose the repair by failure mode

Observed condition Most useful next check Likely action
Kubelet will not start or node is NotReady Node cgroup filesystem, kernel version, kubelet logs, and effective kubelet configuration. Meet the kernel and distribution prerequisites; correct kubelet configuration and boot hierarchy before debugging Pods.
FailedCreatePodSandbox or container creation error Runtime logs, runtime version, CRI plugin status, and driver values on both sides. Align kubelet and runtime to systemd; verify the correct containerd major-version table or CRI-O setting.
Only old Pods fail after a driver edit Whether those Pods and sandboxes were created under the previous driver. Prefer cordon, drain, and node replacement or the release-specific migration procedure.
Container starts, then exits or is OOMKilled Termination reason, application logs, memory limits, and cgroup-v2 awareness. Update or reconfigure the application or agent; do not alter cgroup drivers without evidence of a runtime problem.
Rootless kind only Podman PID limits and host inotify limits. Adjust the rootless host limits following kind’s rootless requirements.
Filesystem reports tmpfs Distribution boot configuration and Kubernetes release behavior for v1. Complete the v2 migration or use the documented temporary v1 fallback; it is not a v2 fix.

A production-safe recovery checklist

  • Classify the failure as kubelet, sandbox/runtime, or post-start application.
  • Run stat -fc %T /sys/fs/cgroup/ on every affected node.
  • Confirm kernel 5.8 or later, distribution support, Kubernetes version, runtime version, and systemd usage.
  • Verify the effective kubelet driver, including command-line and generated-file overrides.
  • Verify the runtime’s active CRI configuration and cgroup manager.
  • Use matching systemd drivers on systemd-based cgroup v2 nodes.
  • Use the containerd 1.x or 2.x configuration table that matches the installed major version.
  • Confirm the CRI plugin is enabled and restart the runtime in a maintenance window.
  • Test a new Pod sandbox before returning the node to service.
  • For an existing node changed in place, prefer replacement or the documented migration procedure over assuming a kubelet restart is enough.
  • For post-start failures, inspect application cgroup-v2 support and memory sizing.

What Kubernetes’ automatic detection does—and does not—do

Kubernetes documentation says: “The kubelet automatically detects that the OS is running on cgroup v2 and performs accordingly with no additional configuration required.” That statement describes OS detection; it does not remove the requirements for a supported kernel, a compatible runtime, an active CRI integration, or matching driver configuration. Automatic detection cannot repair a runtime that still uses an incompatible hierarchy, a stale sandbox created under a different driver, or an application that reads cgroup-v1 files.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Frequently Asked Questions

Can I fix every cgroup v2 Pod error by restarting kubelet?

No. A restart may reload configuration, but it does not reliably recreate sandboxes made under an old cgroup driver. Kubernetes specifically warns that restarting kubelet may not solve those errors; a drained node replacement or release-specific migration can be safer.

What does cgroup2fs versus tmpfs mean in the verification command?

cgroup2fs means the node is mounted with cgroup v2. tmpfs indicates cgroup v1, so investigate the node’s boot and distribution configuration instead of treating it as a v2 runtime failure.

Do kubelet and containerd have to use the same cgroup driver?

On a systemd-based cgroup v2 node, yes: configure both to use systemd. A kubelet/runtime mismatch can cause container-creation errors and inconsistent resource accounting.

Why did my container start and then get OOM-killed after the migration?

The runtime may be working, while the application sizes memory incorrectly. Older Node.js 18 releases are specifically noted as unreliable at detecting cgroup v2 limits; update the runtime or set an explicit heap limit, and check agents that parse cgroup files.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.