What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To scrape a website value with Selenium, start a WebDriver session, open the page, locate the element, wait until its required value is available, then read the representation that actually contains the data. Use rendered text for visible labels, textContent for DOM text, and an attribute or runtime property for inputs and other controls. Close the driver in a finally block so the browser process is not left running.
This guide uses Python for complete examples and explains the same workflow for dynamic pages, repeated records, missing elements, and common failures. Selenium’s official documentation covers element finding, element information, wait strategies, and the first-script pattern.
What Selenium actually scrapes
Selenium does not download a page and guess which characters are interesting. It controls a real browser, finds a DOM element, and reads a specific representation of that element. The representation matters because a value displayed on screen may not be present in the element’s original HTML attribute.
- Rendered text: the user-facing text Selenium exposes through
element.text. Use it for headings, prices, status labels, table cells, and other visible content. - DOM text: text held by descendant nodes. Retrieve it with JavaScript when you need the element’s
textContent, including text that is not rendered exactly as the browser displays it. - Attribute or runtime property: values such as an input’s current value, a link’s
href, an image’ssrc, or a checked state. The current input value is commonly a property, so reading only the originalvalueattribute can return stale data.
Always identify where the target value lives before choosing the API. Selenium’s element-information documentation treats rendered text, text content, attributes, and properties as separate cases.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware match#1 Best Overall
Prerequisites and setup
A basic run needs three components: a Selenium language binding, a browser, and a browser driver. Install the Python binding in a virtual environment:
python -m venv .venv
# macOS/Linux
source .venv/bin/activate
# Windows PowerShell: .venvScriptsActivate.ps1
pip install -U seleniumInstall a supported browser such as Chrome, Firefox, or Edge. Current Selenium versions can often obtain a matching driver automatically through Selenium Manager when you create the driver. If your environment prevents that, install the browser’s driver separately and ensure it is on PATH. Selenium’s getting-started guide describes the supported setup and scaling options.
The minimal extraction workflow
- Import Selenium and create a browser driver.
- Navigate with
driver.get(url). - Locate the target with a stable selector.
- Read text, DOM text, or a property/attribute.
- Handle failures and always call
quit().
This runnable example extracts a visible heading:
from selenium import webdriver
from selenium.webdriver.common.by import By
url = "https://example.com"
driver = webdriver.Chrome()
try:
driver.get(url)
heading = driver.find_element(By.TAG_NAME, "h1")
print(heading.text)
finally:
driver.quit()find_element returns the first match and raises an exception when no match exists. For multiple records, use find_elements; Selenium documents that plural find methods return a collection and return an empty list when there are no matches.
cards = driver.find_elements(By.CSS_SELECTOR, "article.product")
for card in cards:
name = card.find_element(By.CSS_SELECTOR, ".name").text
print(name)Choosing a locator that survives page changes
Prefer a selector tied to meaning rather than presentation. A unique id, a stable data-testid, or a semantic element with a distinctive attribute is usually less fragile than a long chain of generated classes.
# ID
price = driver.find_element(By.ID, "total-price")
# CSS attribute
email = driver.find_element(By.CSS_SELECTOR, "input[name='email']")
# Link text (use only when the visible label is stable)
help_link = driver.find_element(By.LINK_TEXT, "Help")Keep the selector specific enough to identify the intended record, but not so specific that harmless layout changes break it. If a page contains repeated fields, first locate each record, then locate child fields relative to that record.
Scraping text, textContent, and input values
Visible text with element.text
Use text for text a visitor can see after the browser has rendered the page:
Rank #2
status = driver.find_element(By.CSS_SELECTOR, "#job-status").text.strip()
print(status)DOM text with JavaScript
When you need the DOM node’s text, including text that is visually hidden or formatted differently, retrieve textContent:
node = driver.find_element(By.CSS_SELECTOR, ".description")
dom_text = driver.execute_script("return arguments[0].textContent;", node)
print(dom_text.strip())Current values from form controls
An input’s live value is commonly a runtime property. Read it with get_property("value"):
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemssearch = driver.find_element(By.NAME, "q")
current_value = search.get_property("value")
print(current_value)If you need the markup attribute exactly as delivered, use get_attribute("value"). These can differ after JavaScript or user interaction changes the control. The same distinction applies to properties such as checked and attributes such as href or aria-label.
link = driver.find_element(By.CSS_SELECTOR, "a.download")
url_from_markup = link.get_attribute("href")
checked = driver.find_element(By.ID, "agree").get_property("checked")Waiting for JavaScript-rendered values
Navigation reaching the browser’s configured load state does not guarantee that application scripts have finished inserting or changing the target value. A page may return an empty element first, then populate it through a request. Selenium describes this as a common race condition.
Use an explicit wait for the condition your scraper needs. The following waits until a result element exists and contains non-empty text:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
wait = WebDriverWait(driver, 20)
result = wait.until(
EC.presence_of_element_located((By.CSS_SELECTOR, "#result"))
)
wait.until(lambda d: result.text.strip() != "")
print(result.text.strip())If the element must be visible and usable, use visibility or clickability instead:
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #3
button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more"))
)
button.click()For a value that changes after a click, wait on the value itself rather than sleeping for an arbitrary duration:
wait.until(
lambda d: d.find_element(By.ID, "total").get_property("value") == "100"
)Do not mix implicit and explicit waits. Selenium’s wait guidance warns that combining them can create unpredictable timeout durations. Choose explicit, condition-based waits for extraction jobs so each step states what “ready” means.
A complete, defensive scraper
This example extracts a dynamically populated price, records a clear failure, and always closes the browser:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, NoSuchElementException
URL = "https://example.com/product"
SELECTOR = "[data-testid='price']"
driver = webdriver.Chrome()
try:
driver.get(URL)
wait = WebDriverWait(driver, 20)
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, SELECTOR)))
wait.until(lambda d: element.text.strip() != "")
value = element.text.strip()
print({"url": URL, "value": value})
except TimeoutException:
print(f"Timed out waiting for {SELECTOR} on {URL}")
except NoSuchElementException:
print(f"Element disappeared before its value could be read: {SELECTOR}")
finally:
driver.quit()For production jobs, store the URL, selector, timestamp, and exception type with each failure. That makes a changed page distinguishable from a temporary load problem.
Common problems and fixes
NoSuchElementException
Cause: the selector is wrong, the element is inside an iframe, or JavaScript has not created it yet. Fix: verify the selector in browser developer tools, wait for presence, and switch to the correct frame before locating the element.
TimeoutException
Cause: the condition never became true, the page is slow, or the selector targets an element whose value remains empty. Fix: wait on the actual readiness condition, inspect the page after navigation, and increase the timeout only after confirming the condition is correct.
Empty text
Cause: the value is in an input property, is rendered later, or is represented by an attribute. Fix: try get_property("value"), get_attribute(...), or a textContent script as appropriate, then wait for the value to become non-empty.
StaleElementReferenceException
Cause: a framework replaced the node after you located it. Fix: wait for the update to finish and locate the element again immediately before reading it; do not keep old element references across re-renders.
Unexpected driver or browser errors
Cause: an unavailable browser binary, incompatible manually installed driver, or restricted execution environment. Fix: confirm the browser launches interactively, update Selenium and the browser, allow Selenium Manager to resolve the driver where possible, or install a matching driver and check PATH.
Frames, scrolling, and repeated data
An element inside an iframe is not in the top-level document. Wait for and switch to the frame, scrape its contents, then return:
frame = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.CSS_SELECTOR, "iframe.results"))
)
driver.switch_to.frame(frame)
value = WebDriverWait(driver, 20).until(
EC.visibility_of_element_located((By.CSS_SELECTOR, ".value"))
).text
driver.switch_to.default_content()For lazy-loaded lists, scroll or trigger the site’s “load more” control, wait for new records, and use a plural finder again. Re-querying avoids stale references when the framework replaces the list.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Performance, reliability, and scaling
Browser automation is heavier than an HTTP request because each session runs a browser engine. Reuse one session for related pages when isolation is not required, avoid fixed sleeps, and collect several fields from each located record before moving on. Keep explicit timeouts bounded so a broken URL cannot stall an entire batch.
When many browsers or URLs must run concurrently, Selenium’s documented scaling route is Selenium Grid. Grid is optional for a local script; it adds infrastructure for distributed, multi-browser execution. Record failures and retry only transient navigation or timeout errors, not a selector that is demonstrably wrong.
Or skip the browser setup
If your goal is a clean screenshot rather than extracting a structured value, ScreenshotNeo provides a single GET request. Before capture it accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers identify the page verdict and billing status. Its MCP server lets Claude, Cursor, and other MCP clients call take_screenshot, get_page_info, and capture_pdf.
See the ScreenshotNeo documentation for all options, including full-page and element capture, device and retina settings, waits, custom CSS or JavaScript, cookies and headers, blocking rules, PDF output, caching, signed links, asynchronous webhooks, bulk capture, and usage reporting.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 screenshots each month with no card. Paid plans start at $5 for 3,000 shots; every feature is available on every plan. Create a free ScreenshotNeo account to get started.
Recommended Free Tools
Frequently Asked Questions
Can Selenium scrape a value that is not visible?
Yes, when the value exists in the DOM or a property. Use textContent for DOM text or get_property/get_attribute for control and markup values; element.text is intended for rendered text.
Why does page load completion still produce an empty result?
Application JavaScript may populate the element after navigation. Wait for the target element and the specific non-empty value condition instead of relying on page-load readiness.
Should I use implicit waits for every scraper?
No. Use explicit, condition-based waits for predictable extraction and do not mix implicit and explicit waits.
When do I need Selenium Grid?
A local WebDriver is sufficient for a basic script. Grid becomes useful when you need distributed or multi-browser execution at larger scale.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
