Run one independent Selenium WebDriver session per concurrent worker. Let Selenium Manager find a compatible ChromeDriver when possible, avoid sharing a custom Chrome profile or driver port, and inspect Chrome and ChromeDriver logs when a browser exits during startup. For larger or distributed suites, use Selenium Grid and size its nodes for the number of sessions they can support.
What concurrency means in Selenium
A Selenium WebDriver object represents a browser session. To run tests at the same time, start a separate WebDriver session for each worker; do not have multiple workers issue commands through the same driver object. Local sessions use a ChromeDriver service that starts the browser, while remote sessions connect to a remote WebDriver endpoint. See the Selenium WebDriver driver documentation.
Independent sessions need independent browser state. ChromeDriver normally creates a temporary profile for a session. If you supply a custom Chrome profile, each active session must use a different directory. Reusing one profile can cause an “user data directory is already in use” error or state leaking between tests. ChromeDriver capabilities documentation describes Chrome options and profile handling.
Use Selenium Manager for local parallel runs
Selenium Manager ships with Selenium releases since version 4.6. It can discover the installed browser, resolve a compatible driver, download it, and cache it. Leaving the driver path unspecified lets Selenium Manager handle the normal local setup. It still needs network access when it must retrieve driver metadata or download a driver; corporate proxy restrictions may require proxy configuration or an intentionally managed driver binary. See Selenium Manager documentation.
Recommended Free Tools
#1 Best Overall
Python: one driver per worker
This example starts up to four concurrent sessions, navigates each to a URL, returns the page title, and quits each browser even if navigation fails. It relies on Selenium Manager’s default driver resolution. Pass a distinct profile path to each worker only when a custom profile is needed.
from concurrent.futures import ThreadPoolExecutor
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def run_case(url, profile_dir=None):
options = Options()
if profile_dir:
options.add_argument(f"--user-data-dir={profile_dir}") # unique per worker
# Leave the driver path unset so Selenium Manager can resolve ChromeDriver.
driver = webdriver.Chrome(options=options)
try:
driver.get(url)
return driver.title
finally:
driver.quit()
urls = ["https://example.com", "https://www.selenium.dev/"]
with ThreadPoolExecutor(max_workers=4) as pool:
titles = list(pool.map(lambda url: run_case(url), urls))
print(titles)The worker count limits simultaneous sessions, not the number of URLs in the list. If each worker needs its own persistent profile, create a distinct directory for each worker before calling run_case and pass it as profile_dir. If tests do not need persistent browser state, omit the argument and let ChromeDriver create temporary profiles.
When you must pin ChromeDriver
Some teams deliberately pin a driver binary for reproducible or air-gapped builds. Ensure the Chrome and ChromeDriver major versions are compatible, and construct a separate Service and WebDriver for each worker. Selenium’s Python API documents Service as the object that starts and stops ChromeDriver: Chrome Service API.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
def run_pinned(url, chromedriver_path, profile_dir=None):
options = Options()
if profile_dir:
options.add_argument(f"--user-data-dir={profile_dir}")
# Create this Service inside the worker; do not share it between workers.
service = Service(executable_path=chromedriver_path)
driver = webdriver.Chrome(service=service, options=options)
try:
driver.get(url)
return driver.title
finally:
driver.quit()The pinned-path approach makes driver updates your responsibility. Avoid configuring every worker with the same explicitly assigned debugging port or sharing a single running ChromeDriver service. For ordinary local sessions, instantiate WebDriver independently and allow each local Service to select its port.
Choose local drivers or Selenium Grid
| Approach | Best for | Main trade-off |
|---|---|---|
| Selenium Manager with local WebDriver | Small to medium parallel suites on one machine | Limited by that machine’s CPU and memory; first-time driver resolution needs network access |
| Manually pinned ChromeDriver with local WebDriver | Reproducible or air-gapped builds | The team must deliberately maintain a compatible browser and driver pair |
| Selenium Grid | Distributed execution or more sessions than one machine can support | Requires remote endpoint setup and node-capacity planning |
Selenium Grid provides remote nodes and configurable concurrent-session capacity. Its documentation illustrates an eight-session configuration on an eight-CPU node; that is an example, not a universal capacity rule or benchmark. Set node capacity based on the resources available to your browsers, and configure the node maximum sessions with --max-sessions. See Grid CLI configuration options.
Connect a Python worker to a remote Grid
For Grid, create one remote WebDriver session per worker and use the Grid URL supplied by your deployment. Chrome-specific startup arguments still belong in ChromeOptions.
from concurrent.futures import ThreadPoolExecutor
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
GRID_URL = "http://grid-host:4444"
def run_remote(url):
options = Options()
driver = webdriver.Remote(command_executor=GRID_URL, options=options)
try:
driver.get(url)
return driver.title
finally:
driver.quit()
urls = ["https://example.com", "https://www.selenium.dev/"]
with ThreadPoolExecutor(max_workers=4) as pool:
titles = list(pool.map(run_remote, urls))Replace GRID_URL with the endpoint configured for your Grid. A remote endpoint does not remove capacity limits: if the Grid has no available slots, requests may wait or fail according to its configuration.
Diagnose startup failures in order
First identify whether the failure is driver resolution, Chrome process startup, profile reuse, endpoint collision, or machine capacity. The error text and ChromeDriver log often distinguish these cases.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute1. session not created or browser/driver mismatch
- Remove stale manually downloaded ChromeDriver binaries from the path if you intend to use Selenium Manager.
- Check that the installed Chrome and pinned ChromeDriver have compatible major versions.
- If automatic management is blocked by a firewall or proxy, configure Selenium Manager’s proxy or provide a controlled driver path rather than mixing an old binary into the environment.
Selenium Manager’s browser discovery and driver resolution are intended to handle the common matching task, but an explicitly selected stale driver can defeat that default behavior. Details: Selenium Manager.
2. DevToolsActivePort or Chrome exits immediately
Try launching Chrome directly under the same operating-system account and environment before debugging WebDriver. Check that the account can execute Chrome, that required runtime libraries are present, and that the machine has enough memory. On Linux, running Chrome as root is a documented common cause of startup crashes. Do not treat --no-sandbox as a routine fix: Chrome documents it as unsupported and highly discouraged. See ChromeDriver guidance for Chrome startup failures.
Rank #2
3. Profile is already in use
Remove the shared custom --user-data-dir argument and let ChromeDriver allocate temporary profiles, or assign a unique directory to every session that runs at the same time. Do not point parallel tests at a developer’s active Chrome profile.
4. Driver port or process collisions
Start one local Service per worker and do not hard-code the same Chrome remote-debugging port or reuse one service process across workers. Let the local Service select its port unless your infrastructure explicitly allocates distinct ports. The local-session model starts a separate driver service for each session; see Selenium driver sessions.
5. Machine saturation or Grid slots unavailable
If Chrome starts slowly, crashes under load, or sessions queue on Grid, reduce the worker count and retest. Browser processes consume CPU and memory beyond the Python worker overhead. For concurrency beyond one host’s practical capacity, distribute execution over Grid nodes and configure their session limits to match their resources. The Grid eight-CPU/eight-session example is illustrative only: Grid configuration.
6. Capture logs that explain the failure
When startup still fails, retain the exception and enable ChromeDriver logging for a failing worker. The Python Service supports a log output path:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
service = Service(
service_args=["--verbose"],
log_output="chromedriver.log",
)
driver = webdriver.Chrome(service=service, options=options)Use a distinct log file per worker if several failures may occur at once. Compare the ChromeDriver log with Chrome’s direct-launch result; the former shows the driver startup conversation, while a Chrome crash outside Selenium points toward the browser environment or account.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Performance and reliability practices
- Begin with a modest worker count and increase it only while the machine or Grid remains stable. Concurrency is bounded by available browser capacity, not just the thread pool size.
- Always call
quit()in afinallyblock so completed or failed cases do not leave browser and driver processes behind. - Do not share WebDriver objects, custom profile directories, or fixed service/debugging ports across active workers.
- For reproducibility, pin a compatible browser/driver pair and update it deliberately; for less manual setup, let Selenium Manager resolve and cache the driver.
- For distributed workloads, size Grid nodes and configure maximum sessions instead of trying to force more local Chrome processes onto an overloaded machine.
Or skip the browser setup
If the goal is simply to capture a website rather than test browser interactions, ScreenshotNeo provides a screenshot API and MCP server. A single GET request returns a PNG, JPEG, WebP, or PDF. Its capture flow accepts cookie/consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and responses identify the page verdict and billing status in headers. AI agents can use its MCP server tools, including take_screenshot, get_page_info, and capture_pdf.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesFor a full list of parameters and response behavior, see the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpScreenshotNeo includes 1,000 shots a month free with no card; paid plans start at $5 for 3,000 shots. Sign up for the free plan to make a first capture.
Frequently Asked Questions
Can parallel Selenium tests use the same Chrome profile?
No. Give each simultaneously active session its own --user-data-dir, or omit the custom profile and use ChromeDriver’s temporary profile.
Does Selenium Manager start multiple ChromeDriver processes for a thread pool?
Selenium Manager resolves the driver binary; your code still creates a separate WebDriver session and local Service for each worker.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →When should I move from local Chrome to Selenium Grid?
Use Grid when execution needs remote nodes or the desired concurrent sessions exceed the practical CPU and memory capacity of one machine.
