iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
To drive a long-lived shell or interpreter from Python, give one thread sole ownership of the child’s stdout. That thread reads lines and places them on a queue. The caller writes a command followed by a unique end marker that carries the command’s exit status, then consumes the queue until that marker arrives. The marker is a protocol you design. Python’s subprocess module supplies the pipes and the waiting, not the framing, so the marker format, its uniqueness, and the flushing behavior are your responsibility. For a job that starts, does its work, and exits, run() or communicate() is simpler and should be the default.
When a long-lived child is worth the trouble
run() starts a process, sends optional input, collects output, waits for exit, and returns a CompletedProcess. communicate() does the same work on a Popen object you created yourself. Both assume the child’s lifetime ends when the exchange ends. That is the wrong shape for a shell whose working directory, environment variables, and loaded state must survive between commands.
| Situation | Use | Why |
|---|---|---|
| One command, known input, you need its output and exit code | subprocess.run(..., capture_output=True, text=True) |
Pipes, EOF, and waiting are handled in one call. |
| Commands arrive after the child has started, and state must persist between them | Reader thread plus sentinel, as described below | The child must stay alive between exchanges, which communicate() does not allow. |
| The child checks whether it is attached to a terminal | A pseudo-terminal from the pty module |
Pipes present non-terminal standard streams, which can change program behavior. |
A finite example with the simpler API looks like this:
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsimport subprocess
result = subprocess.run(
["sort"],
input="pearnapplen",
capture_output=True,
text=True,
check=True,
)
print(result.stdout) # apple, then pearWhy a bare pipe can deadlock
A child writes into an operating-system pipe that has a fixed-size buffer. If the parent is blocked reading a different stream, or is waiting for the child to exit, the child can block on a full write and never exit. The Python documentation states the rule in the Popen objects section of the subprocess reference, which covers Python 3.14:
#1 Best Overall
“Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.”
A persistent session cannot call communicate(), because that method reads to EOF and waits for the process to exit. You must drain every captured stream yourself. You have two options. You can merge stderr into stdout with stderr=subprocess.STDOUT, which gives one ordered stream and is what the class below uses. Or you can run a second reader thread for stderr. Merging keeps the protocol simple, but you lose the ability to tell error text from normal output.
The select/readline race
A common attempt to avoid blocking is to call select() on the child’s stdout and call readline() only when select() reports data. The program can stall even though the child has already printed the answer:
Free tools Windows power users keep installed
One-click scans. No signup required.
- The child writes two lines in one burst, for example
ok 1andok 2. - The first
readline()on the buffered text wrapper reads a chunk of bytes from the pipe into the wrapper’s internal buffer and returnsok 1. - The kernel pipe is now empty, so
select()reports the descriptor as not readable. - The program waits for
select()to report readiness. The second line is already sitting in Python’s buffer, and the child is waiting for the next command. Neither side moves.
The underlying cause is buffered I/O layering: select() reports the state of the file descriptor, while readline() works on the wrapper’s buffer. Python’s documentation describes Popen streams as file objects and explains how text and binary modes are configured, but it does not name this particular stall. Treat the sequence above as an explanation of how the layers interact, not as a quoted Python rule. The fix is structural: only one piece of code should perform buffered reads, and that code is allowed to block.
Designing the protocol
Four decisions carry the design. Popen enforces none of them.
One owner for stdout
A single reader thread calls readline() in a loop and puts each line on a queue.Queue. The coordinating thread never touches the pipe; it takes items from the queue. Queue is thread-safe, and its get() accepts a timeout, which provides the waiting behavior you wanted from select() without splitting the buffer. When the pipe closes, the thread puts a distinct EOF item on the queue.
EOF means the child closed its output or exited. It does not mean the current command succeeded, so the caller must treat it separately from a completion marker.
A marker that cannot be mistaken for output
Give each command its own marker, built from a random token such as uuid.uuid4().hex. A per-command token also protects against stale markers: a marker left over from a command you have already abandoned will not match the current one. Avoid a fixed prompt as the completion signal. Non-interactive shells usually print no prompt, and a program’s prompt text can appear inside ordinary output.
A child that flushes its output
The bufsize argument controls how the parent buffers what it writes to the child. It does not force the child to flush its own output. A child whose stdout is a pipe usually switches to block buffering, so its output can sit in memory until a buffer fills. If you control a Python child, call print(..., flush=True) when you emit the marker, or start the interpreter with python -u. Shells differ in how promptly they flush builtin output when stdout is a pipe, and the behavior depends on the shell and its version. Confirm that your shell emits the marker promptly in your environment. If it does not, choose a shell that flushes, or move the child to a pseudo-terminal.
Commands must not read the protocol
The shell reads your command script from stdin. Any command that reads stdin, such as cat with no arguments, will consume the marker line that follows it. The marker never arrives, and the call times out. Redirect stdin for commands that do not need it, for example command < /dev/null.
A working session class
The class below runs /bin/sh with merged output, one reader thread, and a per-command marker that carries the exit status. It is a starting point rather than a finished library. It has no locking, so one thread should drive a session at a time.
Recommended Free Tools
import queue
import subprocess
import threading
import uuid
class ShellSession:
"""A persistent POSIX sh whose stdout is read by exactly one thread."""
def __init__(self, argv=("/bin/sh",)):
self._proc = subprocess.Popen(
list(argv),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # one ordered stream, no second pipe to drain
encoding="utf-8",
errors="replace",
bufsize=1,
)
self._lines = queue.Queue()
self._reader = threading.Thread(target=self._pump, daemon=True)
self._reader.start()
def _pump(self):
try:
for line in self._proc.stdout:
self._lines.put(("line", line))
finally:
self._lines.put(("eof", None))
def run(self, command, timeout=10.0):
marker = "__END_" + uuid.uuid4().hex + "__"
script = (command + "n"
+ "printf '\n" + marker + " %s\n' "$?"n")
self._proc.stdin.write(script)
self._proc.stdin.flush()
chunks = []
while True:
try:
kind, line = self._lines.get(timeout=timeout)
except queue.Empty:
raise TimeoutError(
f"no end marker within {timeout} s; session state is unknown")
if kind == "eof":
raise EOFError("shell exited before the end marker arrived")
if line.startswith(marker + " "):
status = int(line[len(marker) + 1:])
# The printf added one newline before the marker; remove it.
return "".join(chunks)[:-1], status
chunks.append(line)
def close(self, timeout=5.0):
try:
self._proc.stdin.close() # EOF: sh exits when its script ends
except BrokenPipeError:
pass
try:
self._proc.wait(timeout=timeout)
except subprocess.TimeoutExpired:
self._proc.kill()
self._proc.wait()
self._reader.join(timeout)Sending commands and reading results
Each call to run() writes the command, then a printf that emits a newline, the marker, and $?, the exit status of the command just run. The newline guarantees the marker starts its own line, whether or not the command’s output ended with a newline. The code then removes exactly the one newline it added, so the returned text matches what the command printed.
sh = ShellSession()
out, status = sh.run("ls /etc | head -n 3")
print(status, out)
sh.run("cd /var && pwd") # state persists between calls
out, status = sh.run("pwd") # prints /var
sh.close()Because the shell itself stays alive, the second pwd reports the directory changed by the earlier command. A one-shot run() call could not do this.
Timeouts and stale output
A timeout cannot distinguish a slow command from a hung one. When run() raises TimeoutError, the shell may still be executing the command, and its marker will arrive later. Do not keep using that session. A later run() would read the old command’s output, and its marker, as part of its own result, because only the new token ends the call. Close the session and create a new one.
Shutdown
Closing stdin is the normal way to end a shell that reads its script from stdin: it exits when input ends. close() waits for that exit, kills the process if the wait times out, and then reaps it with a final wait(). Once the process is gone, the pipe reaches EOF and the reader thread finishes. Killing the process is only safe after you have decided the session is no longer needed, because it discards any command still in progress.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Quick Recap
Choosing between the alternatives
| Approach | Good fit | Main cost |
|---|---|---|
| Reader thread with a queue and a sentinel | A synchronous program that drives a persistent shell or REPL | You own the framing, the timeouts, and the stale-output handling. |
asyncio.create_subprocess_exec |
An application that already runs on asyncio | Cancellation and cleanup must be handled in the coroutine. Reads can use readuntil() with a separator, but you must still handle EOF and overlong lines. |
selectors |
Multiplexing several descriptors on POSIX | It reintroduces the buffering hazard if you mix readiness checks with buffered reads. On Windows, selectors works with sockets, not with Popen pipes. |
Pseudo-terminal (pty module) |
A child that needs terminal behavior, such as isatty()-sensitive output |
Available on Unix-like systems only. The terminal echoes your input into the output, and line endings are translated, so the parser has to account for both. |
Platform and safety notes
- Use an argument list. Launch a known executable with a sequence such as
["git", "status"]andshell=False, which is the default. Useshell=Trueonly when shell syntax or a builtin is genuinely required. The Python documentation’s security section warns that shell metacharacters must be quoted carefully and that untrusted input can cause shell injection. - Windows has no
/bin/sh. The class above assumes POSIX sh and itsprintfand$?syntax. On Windows, the reading design is the same, but the marker command must be written forcmd.exeor PowerShell, and each must be tested in place. - Verify against your exact environment. Process creation differs between POSIX and Windows, and the behavior of shells and child programs varies by version. Run the class with the Python version, operating system, and child program you intend to deploy.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
run() times out and the child is still alive with no output |
The child buffers its output, or the command is waiting for stdin | Flush the marker in a Python child or use python -u. Redirect stdin with < /dev/null. Check the shell’s flushing behavior. |
| A command swallows the next command or marker | The command read from the shell’s stdin | Redirect that command’s stdin from /dev/null, or pass its input explicitly. |
| A result contains text from an earlier command | A timed-out command finished later and its marker was read by the next call | Discard the session after any timeout and create a new one. |
EOFError is raised |
The shell exited, often because a command ran exit or the shell crashed |
Treat it as session loss. Create a new session and resend the command if it is safe to repeat. |
BrokenPipeError on write |
The child exited before the write, so its stdin pipe is closed | Check proc.poll() before writing, and recreate the session. |
| Hangs only when output is large, and stderr is used | A separate stderr pipe is not being drained | Merge stderr into stdout with stderr=subprocess.STDOUT, or add a second reader thread. |
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
