iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A cURL command that succeeds in a terminal can fail from Python because Python may pass different arguments, build a different URL, run with a different network environment, or send a different request altogether. Start by determining whether your code launches the curl executable or replaces it with a Python HTTP client; then compare the actual inputs and error signals at each boundary.
First, identify which client Python is using
There are two distinct setups. With subprocess, Python starts the curl executable, which still handles the transfer. With a library such as Requests, Python sends the request through a different client implementation. The troubleshooting checks overlap, but the things you inspect are not identical.
| Setup | Who interprets the request? | What to inspect |
|---|---|---|
| cURL typed in a terminal | The active shell parses the command, then passes arguments to curl. | Shell quoting and expansion, the arguments curl receives, and the terminal process’s environment. |
Python launches curl with subprocess |
By default, Python passes an argument list directly to the program without shell interpretation; curl handles the transfer. | The argument list, return code, standard output and error, and the Python process’s environment. Python subprocess documentation |
| Python HTTP library | The library constructs and sends the request; curl is not involved. | Method, URL, headers, authentication, body encoding, redirects, proxy and certificate settings, response status, and exception behavior. |
Gotcha 1: Terminal quotes and shell behavior do not automatically carry over
In a terminal, quotes and shell operators are processed before curl receives its arguments. For example, an unquoted & in a URL can be treated as a shell operator rather than as part of the URL. The curl project advises quoting URLs that contain such characters. curl FAQ
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Python’s subprocess uses shell=False by default. When you pass an argument list, Python sends those values directly to the program rather than asking a shell to interpret them. This means terminal quote characters should not usually be copied into the list: quotes that group text in a shell can become literal characters if passed as part of an argument.
#1 Best Overall
import subprocess
url = "https://example.com/search?q=red%20fox&page=2"
result = subprocess.run(
["curl", "--fail", url],
check=True,
capture_output=True,
text=True,
)
print(result.stdout)Use a list of arguments as the starting point. Log or inspect that list and the exact URL value. Set shell=True only if you intentionally need shell behavior; doing so makes the chosen shell interpret the command and its quoting.
Gotcha 2: The URL may not be the URL you think it is
Inspect the final URL value after your code has assembled it, not just the template or individual pieces. The curl project states: “A URL provided to curl cannot contain spaces.” Encode spaces and construct query values with a URL-aware encoder when they may contain reserved characters. curl URL syntax
Manual string concatenation can change the meaning of a URL when a value contains characters that have a special role in URLs. Compare the resulting string with the one used in the working terminal command, paying particular attention to spaces, ampersands, question marks, and other reserved characters.
Gotcha 3: The Python process may have different proxy or certificate settings
A terminal, IDE, notebook, service, and scheduler can start processes with different environment variables. So a request that works in an interactive terminal may use a different route or certificate configuration when launched from Python.
- Check relevant proxy variables in both environments. curl documents
http_proxy,HTTPS_PROXY,ALL_PROXY, andNO_PROXY; explicit proxy options override environment variables. curl man page - If the Python code uses Requests, check how its environment-based proxy settings interact with values supplied by the program. Requests also documents
REQUESTS_CA_BUNDLEandCURL_CA_BUNDLEas certificate-bundle overrides. Requests advanced usage - Compare the certificate and trust configuration seen by the exact failing process with the one used by the working terminal.
Do not treat disabling TLS verification as a general fix. Identify the certificate or trust-setting difference and keep verification enabled.
Gotcha 4: A Python HTTP request is not automatically equivalent to a curl command
When Python uses Requests or another HTTP library instead of starting curl, it is not merely translating curl syntax. Compare the details of the request: method, final URL, headers, authentication, body encoding, redirect behavior, proxy configuration, and certificate settings. Do not assume every curl option has a direct equivalent in a particular Python library.
Also distinguish three different outcomes: whether the program ran, whether the HTTP server returned a successful status, and whether your client code treated that status as an error. For example, curl’s --fail changes how certain HTTP error responses affect curl’s failure behavior. A process return code, an HTTP response status, and a library exception are separate signals; inspect the one your code is actually checking. curl man page
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesDebug the failure in order
- Choose the branch: determine whether Python starts the curl executable or sends the request through a Python HTTP library.
- If it starts curl: print or log the argument list. Prefer a list with the default
shell=False; capture standard output and error, and inspect the process return code. - Compare the URL: compare the final Python URL string with the one used in the terminal. Check for literal spaces and reserved characters that were not encoded or were interpreted differently.
- Compare the environment: check the relevant proxy variables and certificate-bundle configuration for the working terminal and the failing Python process.
- If using an HTTP library: compare the actual method, URL, headers, authentication, body, redirects, proxy, and certificate setup; then inspect both the response status and how the code handles errors.
These checks identify where the two paths diverge; a terminal success alone does not establish that Python used the same arguments, environment, or request.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
