Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A Kubernetes node can report only 58% disk usage and still fail to create files because byte capacity and inode availability are separate limits. In a reported CI-node incident, Sergey Shinder traced “no space left on device” errors to an exhausted inode supply, not a full disk. Kubernetes documents inode-free signals separately from available-byte signals, so both need to be monitored and accounted for in eviction settings.
How a node can run out of inodes before it runs out of bytes
Filesystems track at least two distinct resources: space for file contents and inodes, which record metadata for files and directories. A filesystem can have bytes available but no free inodes to create another file. In that case, an application may receive “no space left on device” even though a dashboard based on byte usage shows room remaining.
Shinder’s incident account says three CI nodes began failing to start new pods while a disk panel showed 58% used. According to the account, df -i showed the root volume at 100% inode use. The article’s indexed listing gives no publication year, and these figures are reported incident details rather than independently validated statistics. Read Shinder’s incident account.
Why an image with many small files can trigger the failure
Shinder describes a frontend build image containing node_modules and about 400,000 mostly small files. The image was rebuilt several times a day, and nodes unpacked each version when pulling it. Many small files can consume inode capacity while adding comparatively little byte usage, making a byte-only cleanup trigger a poor early warning for this particular failure mode.
#1 Best Overall
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
In the account, image cleanup ran when byte use reached 85%, so inode growth did not necessarily trigger it. One node reportedly retained 43 versions of the image. These are details of Shinder’s incident; the 85% value is not a Kubernetes default established by the official documentation.
What Kubernetes monitors—and what custom thresholds can change
Kubernetes node-pressure eviction documentation identifies Linux inode signals separately from byte-availability signals. The inode signals are nodefs.inodesFree and imagefs.inodesFree; the related byte signals include nodefs.available and imagefs.available. For Linux, the documented default hard threshold is 5% free inodes for both nodefs and imagefs.
Rank #2
- Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Those defaults apply only when threshold parameters have not been changed. Kubernetes warns that changing a parameter does not automatically retain the other defaults unless default merging is enabled or the desired thresholds are specified explicitly. A custom threshold configuration can therefore leave inode pressure unwatched if it replaces the full default set without adding inode signals. Check the documentation for your deployed Kubernetes release and inspect the effective kubelet configuration before changing it. Kubernetes: Node-pressure Eviction.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Check which filesystem each signal describes
The names do not guarantee that node files, image layers, and container writable layers live on separate disks. Kubernetes documents layouts where nodefs, imagefs, and optionally containerfs can refer to the same underlying filesystem or to distinct filesystems. Interpret the signals against the actual node layout, rather than assuming images always use a separate volume.
Rank #3
- Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
How the incident spread across the CI nodes
Shinder reports that failed pods moved to other nodes, prompting further image pulls. As the same workload and image pressure reached additional nodes, 9 of 20 nodes reportedly refused work by that afternoon. Those counts describe the author’s incident only; they should not be treated as typical Kubernetes failure rates.
What the author changed
Shinder describes several changes aimed at inode visibility, image file count, and cleanup. They are reported remedies, not a controlled comparison or a universal configuration prescription.
Rank #4
- Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
- Expose inode use: The node panels added inode usage alongside byte usage, with corresponding alerts.
- Make kubelet thresholds explicit: The configuration listed each intended eviction threshold, including free inodes, instead of relying on defaults that might be displaced by custom settings.
- Reduce files shipped in the image: The frontend image stopped including
node_modules; dependencies were obtained from a cache volume at job time. The author reports reducing the image’s file count from about 400,000 to fewer than 9,000. - Remove stale images by age: Unused images were removed after three days regardless of byte usage.
- Change the container-volume filesystem: The author says the node image was changed to format the container volume as XFS, which the article describes as allocating inodes as needed. The Kubernetes documentation cited here does not verify that filesystem-specific claim or establish that this choice suits every deployment.
A practical diagnostic sequence
- Confirm the failure and resource: When file creation fails with “no space left on device,” compare byte availability with inode availability on the filesystem involved. Shinder’s account used
df -ito identify an inode-full root volume. - Map the affected path to a filesystem: Determine whether the failing path is on the node filesystem, image filesystem, or another volume. Kubernetes filesystem roles may share an underlying filesystem.
- Inspect effective eviction thresholds: Verify that the deployed kubelet configuration includes the inode signals you intend to use, especially if any threshold parameters were customized. Confirm behavior against documentation for the deployed release.
- Find sources of file growth: Review image contents and the accumulation of old image versions. Large counts of small files can be more relevant to inode exhaustion than their byte footprint.
- Alert and clean against the right limits: Track free inodes as well as available bytes, and choose image cleanup rules that do not depend exclusively on byte usage. Test configuration and cleanup changes against the node’s filesystem layout and workload.
Why adding disk space may not solve it
More byte capacity does not necessarily provide more inodes on a filesystem whose inode supply is already exhausted. The relevant remedy depends on how the filesystem was created and configured, where Kubernetes stores images and other files, and what is creating the file count. The incident’s approach targeted inode monitoring, kubelet thresholds, image contents, and stale-image cleanup rather than treating byte capacity as the only constraint.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsQuick Recap
Best Value
- [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
- 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
- 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
- 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
- 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
