iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The reports failed because the PostgreSQL container had a 64 MB /dev/shm, not because the database volume was full. Month-end queries joined three years of ledger data with parallel hash joins and needed hundreds of megabytes of shared memory per query. The former virtual machine had an 8 GB shared-memory filesystem, so the migration silently reduced capacity. Mounting a 2 GB memory-backed Kubernetes emptyDir at /dev/shm fixed the capacity mismatch, but the pod’s memory limit also had to increase because tmpfs pages count against that limit.
What actually failed
Sergey Shinder’s 2026 incident report describes a PostgreSQL 15 reporting database moved into a Kubernetes StatefulSet in June. On the first working day of July, month-end finance reports failed roughly one time in three with this message:
could not resize shared memory segment, No space left on deviceThe phrase “No space left on device” was accurate but misleading. The database data volume was only 40% full. The exhausted filesystem was /dev/shm, the in-memory shared-memory mount used by processes in the container. The container runtime had provided only 64 MB there. The previous virtual machine had an 8 GB tmpfs, so the migration changed an important capacity limit even though the database files themselves moved successfully.
Daily queries used only a few megabytes and passed migration testing. Month-end reports were different: they joined three years of ledger lines and used parallel hash joins. Each query could request hundreds of megabytes of shared memory, and several reports could run at the same time. The small default mount was therefore exposed only by the peak workload.
#1 Best Overall
Why PostgreSQL reports fail only at month end
Routine traffic does not exercise the limit
A small /dev/shm can look healthy for weeks when ordinary queries allocate little temporary shared memory. A migration test that replayss a typical day may never approach the mount’s ceiling. The first month-end close introduces larger joins, longer-running operations and more concurrent reporting sessions.
Parallel hash joins amplify the demand
Parallel execution lets multiple workers cooperate on a query. Hash joins need memory for hash tables and coordination structures; when the planner chooses a parallel plan, the workers must use shared memory that all relevant processes can access. A report joining several years of ledger data can therefore require far more shared memory than a daily query over recent rows.
The exact allocation depends on the query plan, data distribution, work settings and concurrency. The incident’s “hundreds of megabytes” is a measurement from that workload, not a universal PostgreSQL requirement. Treat it as evidence that peak reports must be measured in your own environment.
Free tools Windows power users keep installed
One-click scans. No signup required.
Containers do not inherit the VM’s directory sizes
A container filesystem can look like a complete machine while individual mounts have very different limits. As Shinder put it, “A container gives you a filesystem that looks like a machine’s, and some of its directories are much smaller than the machine’s were.” The database volume, node disk and /dev/shm are separate capacity domains. Free space in one does not make space available in another.
Confirm that /dev/shm is the bottleneck
Check the mount from inside the running PostgreSQL container during a representative workload:
- List the pod and identify the database workload:
kubectl get pods -n reporting. - Display the shared-memory mount size and usage:
kubectl exec -n reporting deploy/reporting-db -- df -h /dev/shm. - Inspect how the mount was created:
kubectl exec -n reporting deploy/reporting-db -- mount | grep shm. - Check the pod’s termination history and memory events:
kubectl describe pod -n reporting <pod-name>and review the Events section forOOMKilledor eviction messages. - Compare the output with the database volume:
kubectl exec -n reporting deploy/reporting-db -- df -h. A healthy data volume does not disprove a full/dev/shm.
Run the checks while a month-end-style query is active if possible. A quiet pod can show plenty of free space even though the next concurrent report will exhaust the mount. PostgreSQL’s error identifies the immediate allocation failure; the mount inspection identifies which filesystem reached its limit.
Immediate service restoration: disable parallel query
The incident’s short-term workaround was to disable parallel query. This reduced shared-memory demand and allowed reports to complete correctly, but they ran more slowly. It is a way to restore service while capacity is being corrected, not a substitute for sizing the container.
When the workaround is appropriate
- Month-end reporting must complete before a configuration rollout is possible.
- The failures are caused by shared-memory exhaustion rather than corrupt data or an invalid query.
- Slower execution is preferable to intermittent report failure.
What to record while it is enabled
- Report completion time and concurrency.
- The query plans and PostgreSQL settings used during the constrained run.
/dev/shmusage and pod memory usage.- Whether reports remain correct when parallel workers are removed.
Do not permanently trade away parallel execution without measuring the operational effect. The workaround can turn a one-in-three failure rate into consistently late reports, especially as the ledger grows.
Rank #3
Durable Kubernetes fix: mount a larger memory-backed emptyDir
The durable change in the incident was a 2 GB memory-backed emptyDir mounted at /dev/shm. Add a volume and mount to the PostgreSQL pod template:
spec:
template:
spec:
containers:
- name: postgres
volumeMounts:
- name: dshm
mountPath: /dev/shm
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 2GiApply the change to the StatefulSet’s pod template, roll the pod in a controlled window, and verify the result from inside the new container:
df -h /dev/shm
mount | grep shmThe 2 GB value is the capacity used for this incident, not a universal PostgreSQL setting. Size it from observed peak demand, simultaneous report count and a safety margin. A mount that is too small recreates the error; an unnecessarily large mount reserves memory that the pod and node may need for other work.
The memory-limit trap: tmpfs is still container memory
A memory-backed emptyDir stores pages in RAM (and the kernel’s accounting for that memory). Those pages count toward the container’s memory limit. Merely changing /dev/shm to 2 GB while leaving the old pod limit unchanged can replace a PostgreSQL allocation error with a container termination.
Increase the pod’s memory limit by the amount needed for the new shared-memory capacity, while preserving headroom for PostgreSQL processes, connection memory, cache, background workers and the operating system. The incident explicitly increased the limit by the same 2 GB used for the emptyDir. The correct total depends on the existing limit and measured workload; it is not possible to infer a safe absolute value from the incident alone.
Validate both ceilings after rollout
- Inside the container, verify that
df -h /dev/shmreports the intended size. - During replay, watch pod memory usage against its limit.
- Confirm that the pod is not marked
OOMKilledand that the node has sufficient allocatable memory. - Keep a margin below the limit; using the entire limit for tmpfs leaves no room for PostgreSQL itself.
How to size shared memory for peak reporting
Use a workload-based process rather than copying the VM’s 8 GB or the incident’s 2 GB blindly.
- Measure the current baseline. Record idle and ordinary-day
/dev/shmusage, pod memory usage and the number of concurrent reporting sessions. - Replay the real peak. Run the previous month’s month-end queries, including their normal scheduling and concurrency. The migration test that replayed only one week of daily queries missed the failure.
- Capture the high-water mark. Observe
/dev/shmwhile parallel reports run. Record the largest use and which query combinations produced it. - Add operational headroom. Allow for growth in ledger history, an extra concurrent report, plan changes and connection bursts. Choose a limit that remains below the pod’s total memory budget.
- Repeat after schema or workload changes. New indexes, PostgreSQL upgrades, altered planner settings and additional report users can change allocation patterns.
Peak concurrency matters as much as the largest single query. A mount sized for one report can still fail when several month-end jobs overlap.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Compare the two remediation options
| Option | Report correctness and speed | Shared-memory capacity | Pod memory headroom | Operational complexity | Peak-workload test |
|---|---|---|---|---|---|
| Temporarily disable parallel query | Reports remain correct in the incident, but run slower. | Reduces demand; does not enlarge /dev/shm. |
Usually lowers query demand, but does not fix an undersized mount. | Low; can be applied as an emergency setting and later reverted. | Still required to learn whether the production workload is viable at the slower speed. |
Mount a memory-backed emptyDir at /dev/shm |
Preserves parallel plans and their potential speed, subject to capacity. | Provides the configured limit; the incident used 2 GB. | Must increase with the mount because tmpfs pages count against the limit. | Moderate; requires a pod-template rollout, sizing and monitoring. | Must be validated by replaying actual month-end queries and concurrency. |
The incident report supplies no benchmark timings, so it cannot establish how much slower a particular report becomes with parallelism disabled. Measure that trade-off in your own database.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Migration checks that prevent a repeat
Database migration checklists often verify data files, network access and persistent volumes while overlooking host resources that containers do not inherit automatically. Add these checks before declaring a PostgreSQL migration complete:
/dev/shm: record its size in the source environment, the container image and the final pod.- Peak query workload: replay actual month-end statements, not only a representative week of daily traffic.
- Concurrency: reproduce the number and timing of simultaneous reports.
- Memory accounting: include memory-backed volumes in pod-limit calculations and test for
OOMKilled. - Open-file limits: compare the source and container settings and validate them under load.
- Huge pages: check whether the database or extensions depend on them and whether the node and pod expose them.
- Process limits: verify that worker and connection limits are compatible with the container runtime.
Keep the measured results with the deployment configuration. A future operator should be able to see why the mount and memory limit were chosen instead of treating them as arbitrary numbers.
Troubleshooting after the change
The error continues immediately
Confirm that the new pod, rather than an old replica, is serving traffic; check df -h /dev/shm inside that pod; and inspect the StatefulSet template for a mount-path typo. A volume mounted somewhere other than /dev/shm does not change PostgreSQL’s target filesystem.
Recommended Free Tools
The error becomes an OOM kill
The shared-memory mount may now be large enough while the container limit is not. Compare memory usage with the limit, account for PostgreSQL’s ordinary memory needs, and raise the limit only when node capacity and scheduling policy can support it. Do not respond by making the tmpfs unlimited.
Reports succeed but are too slow
That is consistent with the temporary parallel-query workaround. Restore parallel execution after the mount and memory budget are validated, then replay month-end queries to confirm both completion and acceptable duration.
Only some reports fail
Compare their plans, date ranges and overlap with other jobs. The failure threshold is workload-dependent: a query joining three years of ledger data can cross it while a shorter report does not. Use the failing combination, not an average daily query, to size the mount.
Bottom line
This was a capacity mismatch introduced by containerization. PostgreSQL 15 was moved from a VM with an 8 GB /dev/shm to a container with 64 MB, and the mismatch surfaced when parallel month-end hash joins needed hundreds of megabytes. Disabling parallel query restored correctness at lower speed. The lasting fix was a 2 GB memory-backed emptyDir at /dev/shm, paired with an equivalent increase in pod memory capacity and a test that replayed real month-end concurrency. Treat /dev/shm as an explicit PostgreSQL resource during every Kubernetes migration, not as an inherited property of the old machine.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
