To limit a Linux multiprocessing job as a whole, run it inside a cgroup and set an aggregate CPU quota and a hard memory limit. Use a systemd scope or service for a host-launched job, or the container runtime’s resource controls for a workload already running in Docker. Then set the program’s worker count to fit both budgets; a CPU limit alone will not keep a pool from exhausting memory.
Choose a boundary that includes the whole job
A process pool is a tree: it includes the launcher, workers, and potentially helper processes. A cgroup can apply resource controls to that group and its descendants, rather than requiring you to cap each worker separately. Select the boundary that matches how you launch the job:
| Where the job runs | Practical boundary | What to check |
|---|---|---|
| Directly on a systemd-managed Linux host | A systemd scope or service | Host systemd and cgroup configuration, plus any tighter limits inherited from parent cgroups. Systemd documents resource-control properties at systemd.resource-control. |
| In an existing Docker workflow | Docker container resource controls | Docker version and host/runtime configuration, plus any constraints imposed outside the container. See Docker’s resource constraints documentation. |
These are configuration examples, not tested commands; confirm the effective settings in your environment. The systemd command uses a scope for the launched command:
systemd-run --scope -p CPUQuota=200% -p MemoryMax=4G python job.pyCPUQuota=200% represents a maximum CPU bandwidth equivalent to two CPUs’ worth of runtime. MemoryMax=4G sets a hard memory limit, subject to systemd version, unit parsing, cgroup configuration, and parent limits.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
- Powerful Linux Laptop: This IdeaPad Slim 3 Laptop comes pre-installed with Ubuntu Linux, offering fast performance, robust security, and a clean, user-friendly experience. Enjoy full customization, seamless hardware compatibility, and access to thousands of open-source apps. Whether you're working, creating, or coding, it's built to keep up with everything you do.
- A Multitasking Master: The latest AMD Ryzen 7 5825U processor (up to 4.5 GHz) delivers powerful performance with 8 cores and 16 threads for smooth multitasking. Integrated AMD Radeon Graphics provide crisp visuals for streaming, browsing, photo editing, and casual gaming. With smart machine intelligence, it adapts to your needs for a fast, responsive experience.
- 15.6" Full HD Display: The IdeaPad Slim 3 boasts an 88% screen-to-body ratio for a floating, edge-to-edge visual experience. TÜV Low Blue Light certification reduces eye strain, making it perfect for long work or study sessions.
- Military-Grade Durability: The smart IdeaPad Slim 3 combines portability and durability, letting you work, study, and play on the go. With a profile 10% slimmer than the previous generation, it's lightweight yet military-grade rugged, ready for anything, anywhere.
- Versatile Connectivity: Enjoy the security of a built-in webcam with a privacy shutter. Connect effortlessly with multiple ports: 2x USB A, 1x USB C, 1x HDMI, 1x SD Card Reader, 1x Headphone/Microphone combo. Bundle comes with Stylus Pen, 256GB Portable SSD and 5-in-1 Docking Station.
For Docker, the equivalent shape is:
docker run --cpus=2 --memory=4g IMAGE COMMAND--cpus sets a CPU access cap and --memory a container memory limit. Do not substitute --cpu-shares when you need a hard CPU cap: shares are a relative weight that affects allocation when CPU is constrained, not an equivalent quota.
Does a CPU quota limit cores or CPU time?
A CPU quota limits the amount of CPU time the workload can consume over the scheduler’s quota period. It does not select which cores the workers run on. In systemd, CPUQuota=200% sets a maximum bandwidth of two CPUs’ worth of runtime; the workload may run across different CPUs while staying within that aggregate budget.
Rank #2
- Intel Core i5-10210U (up to 4.2GHz) - 1TB PCIe NVMe + 1TB HDD - 32GB DDR4 SDRAM
- 17.3" HD+ (1600x900) Display, Intel UHD Graphics 620
- Built in HD 720p Webcam with Microphone - Bluetooth Version4.2
- I/O Ports: 2x USB 3.1 (Data Only), 1x USB 2.0, 1x HDMI, 1x Headphone/Microphone Combo Jack
- Linux Mint Cinnamon 64-Bit - 6-Row Keyboard w/ Full Numberpad
CPU placement is a separate control. Systemd’s AllowedCPUs= restricts execution to a CPU list, while EffectiveCPUs= shows the resulting list after parent restrictions are taken into account. Affinity can help with locality or keep a job off selected CPUs, but pinning workers does not itself cap their aggregate CPU time. See the systemd resource-control properties.
What happens when a cgroup memory limit is reached?
On cgroup v2, memory.high and memory.max serve different purposes:
Rank #3
- Intel Core i5-1335U Processor (12M Cache, 12 Threads, up to 4.6 GHz) - 256GB Solid State Drive - 16GB DDR4 SDRAM
- 15.6" FHD (1920x1080) Non-Touch Anti-Glare Display - Intel UHD 620 Integrated Graphics - Stereo Speakers
- 720p HD Webcam with Privacy Shutter. Integrated Microphone - Intel Dual Band Wireless-AC (2x2) 8265, Bluetooth Version 4.2
- I/O Ports: 2x USB 3.0, 1x USB 3.1 Type-C 3.1, Headphone/Mic Combo Port, 4-in-1 Card Reader, HDMI, Kensington Mini-Lock Slot
- Linux Mint (Cinnamon) 64-Bit - Keyboard with Full NumberPad - Fast Charging
| Setting | Effect |
|---|---|
memory.high |
A pressure and throttling boundary. Exceeding it causes heavy reclaim pressure and throttling; it does not invoke the OOM killer. |
memory.max |
The principal hard memory limit. If usage reaches it and cannot be reduced, the kernel invokes the OOM killer within the cgroup. Usage can temporarily exceed the limit. |
The Linux Kernel Documentation describes memory.max as the “Memory usage hard limit” and the “main mechanism to limit memory usage of a cgroup” in its cgroup v2 documentation. A hard limit can cause allocations to fail or processes to be terminated. Leave headroom for the parent process, workers, libraries, shared-memory objects, and other processes in the job.
How to keep a Python multiprocessing pool within both budgets
Set the worker count deliberately when you need predictable resource use. For CPU-bound work, keeping the pool within the usable CPU budget is a reasonable starting point, not a universal optimum. Memory requires a separate estimate: count the parent and workers, including their per-process allocations and shared resources, under representative workloads. There is no reliable workers-per-GiB rule that applies to every program.
Rank #4
In Python 3.13 and later, multiprocessing.Pool(processes=None) uses os.process_cpu_count() as its default worker count rather than os.cpu_count(). The former reports logical CPUs usable by the calling thread and can be lower than the machine-wide count. This can account for CPU affinity; it does not universally translate a cgroup CPU quota into the right worker count, and it says nothing about memory. See the Python multiprocessing documentation and Python’s os.process_cpu_count() documentation.
For an explicit pool size, use the processes argument:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Best Value
- 12th Intel Alder Lake N95 Processor – The GMKtec G3 S Mini PC is powered by the 12th Gen Intel N95 processor with 4 cores, 4 threads, 6MB cache and a burst frequency up to 3.4GHz. Compared with N100/N5105/N5100/N5095, the N95 delivers up to 36% overall performance improvement. Perfect for routine tasks, office work, and home entertainment, this compact mini desktop is more convenient than traditional bulky PCs.
- 8GB RAM & 256GB SSD Storage – Pre-installed with 8GB DDR4 memory and a fast 256GB M.2 2242 SSD, the G3 S mini desktop offers quicker startup, smoother multitasking, and faster file transfers. Enjoy seamless performance whether you’re working on multiple applications, browsing, or streaming content.
- Rich Interfaces & Connectivity – The G3 S mini computer comes equipped with USB 3.2 (up to 10Gbps), dual HDMI 2.0 (4K@60Hz), and a 3.5mm audio jack. With support for WiFi 5, Bluetooth 5.0, and Gigabit Ethernet (RJ45 1000MbE), it connects easily with monitors, projectors, printers, office equipment, and other peripherals, making it versatile for both home and business use.
- Dual 4K Display Support – Featuring upgraded Intel UHD Graphics (up to 1000MHz), the G3 S supports 4K video playback and AV1 decoding for a smooth viewing experience. With dual HDMI outputs, you can connect two 4K@60Hz displays simultaneously, enabling efficient multitasking for work and entertainment.
- GMKtec WARRANTY - GMKtec offers a 1-year limited GMKtec's warranty for each mini PC, starting from the date of the purchase. All defects due to design and workmanship are covered. With a professional after sales team always ready to attend to your needs, you can simply relax and enjoy your mini PC.
from multiprocessing import Pool
with Pool(processes=4) as pool:
results = pool.map(work, items)Choose 4 based on the job’s actual CPU and memory budget; it is an illustration, not a recommended count for every machine. Adjust it if measured per-worker memory use would push the job too close to its hard limit.
Clean up workers and account for helper resources
Use a pool as a context manager, as above, or explicitly close it when work is complete and join its workers. If you need to stop outstanding work, terminate the pool and join it. The maxtasksperchild option lets you replace workers after a chosen number of tasks, which can help release resources accumulated by long-lived workers; it does not replace an aggregate memory limit.
On POSIX systems, the spawn and forkserver start methods also run a resource tracker for named resources such as semaphores and SharedMemory. Include these resources in troubleshooting if usage or cleanup does not match what you expect. Python documents pool lifecycle, start methods, and resource tracking in its multiprocessing reference.
Why per-process limits are not a job-wide cap
Python’s Unix resource module offers limits such as RLIMIT_CPU and RLIMIT_AS, but they apply to an individual process, not as a simple aggregate ceiling for a multiprocessing tree. RLIMIT_CPU limits per-process processor time and sends SIGXCPU when the limit is crossed. RLIMIT_AS limits a process’s address space; it is not a substitute for a cgroup memory limit covering the whole job. Use these as supplementary controls where appropriate, rather than as the job boundary. See the Python resource module documentation.
Recommended Free Tools
Check the effective limits before relying on them
A configured value may not be the value the job can actually use: parent cgroups can impose tighter CPU, memory, or CPU-list limits. Verify the cgroup version and effective settings using the controls and inspection methods appropriate to the host or container runtime. If usage is still higher than expected, check that the launcher and descendants are inside the intended boundary, account for shared and helper resources, and confirm that a parent limit has not changed the effective budget.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
