To paginate with Selenium, collect the current results, click the site’s Next control, wait for a page-specific signal that the results have changed, and repeat until the site indicates there is no next page. The locator, change signal, and stopping condition depend on the target site; Selenium does not provide one universal pagination selector or end-of-list rule.
What a reliable pagination loop needs
Pagination automation has four parts: identify the current results, find a usable Next control, confirm the transition, and stop when the site reaches its final page. First inspect the target page’s markup and behavior. Choose a stable locator for the Next control and a separate, observable signal that identifies the new page—such as a changed page number, URL, or result content.
Selenium’s Python expected condition element_to_be_clickable checks that an element is visible and enabled. That makes it useful before clicking, but it does not prove that the next page has loaded. A click must be followed by a wait for the transition your next step depends on. Selenium’s waiting strategies documentation explains why: document readyState covers assets defined in the HTML, while JavaScript can continue changing the page afterward.
Choose a signal that matches the site
| Pagination behavior | Useful signal after clicking |
|---|---|
| Full document navigation | Wait for a page-specific state, such as the expected URL or page indicator, then locate results again. A click alone is not proof that the destination page is ready. |
| In-place JavaScript or AJAX update | Wait for the page indicator or result content to change, or for the previous result container to become stale if the site replaces it. |
Selenium’s Java API notes that a click that causes a new page load does not itself wait for that load to complete. The general practice applies across bindings: verify a meaningful state before using the new page’s data. The Python expected-conditions reference includes checks for presence, visibility, clickability, and staleness; use the documentation for your installed binding because APIs differ. In particular, Selenium 4 no longer supports Expected Conditions in .NET.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problems#1 Best Overall
Python example: collect pages until Next is unavailable
This pattern assumes the site exposes a visible page number that changes after each click and a disabled or absent Next control at the end. Replace the sample selectors and adapt the terminal check to the actual page. It is illustrative, not a tested recipe for a particular website.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
next_locator = (By.CSS_SELECTOR, "a.next")
while True:
# Read the current page afresh on every iteration.
rows = driver.find_elements(By.CSS_SELECTOR, ".result-row")
collect(rows)
next_buttons = driver.find_elements(*next_locator)
if not next_buttons or not next_buttons[0].is_enabled():
break
old_page_text = driver.find_element(
By.CSS_SELECTOR, ".page-number"
).text
wait.until(EC.element_to_be_clickable(next_locator)).click()
wait.until(
lambda d: d.find_element(
By.CSS_SELECTOR, ".page-number"
).text != old_page_text
)- On the first page, locate and collect the current results.
- Check whether Next exists and is enabled. If it does not, stop. Some sites keep the control visible but disable it on the last page; others remove it.
- Record the current page indicator, wait until Next is clickable, and click it.
- Wait until the indicator changes before collecting results on the next loop.
For a site that does not show a page number, replace the final wait with a condition that reflects its behavior—for example, changed result text, a changed URL, or staleness of the old result container. Selenium’s expected conditions describe the available state checks.
Rank #2
Wait for state, not elapsed time
A fixed sleep guesses how long a transition will take: it may end too early on a slow run and waste time on a fast one. An explicit wait instead polls a condition until it succeeds or the timeout expires. Python’s WebDriverWait evaluates its condition repeatedly, returning when it becomes truthy or timing out otherwise; see the Python WebDriverWait API reference.
Pick the condition for the next operation. If the next step needs results, wait for the result state to change or become available. If it needs a page indicator, wait for that value to change. If the application replaces a known container, waiting for the old container to become stale can confirm replacement. A generic wait for document readiness is not a substitute for the application-specific signal.
Avoid stale elements after a page change
A WebElement is a reference to a particular DOM element. Navigation or a JavaScript update can remove or replace that element, leaving the reference stale and causing a StaleElementReferenceException. Do not carry result rows, page indicators, or Next controls forward across a transition unless you know the site retains those exact nodes. Re-find them from the current DOM after the wait succeeds. Selenium’s troubleshooting guidance covers stale element references and related errors.
Adapt the stopping condition to the target
There is no universal last-page convention. The Next control might be absent, disabled, or replaced by another site-specific state. Inspect the actual page and use the terminal signal it provides; do not assume that a missing page number or a click failure means the list is finished. If the site has no reliable visible terminal state, the loop needs a site-specific rule based on its documented behavior.
The Python API reference searched for this tutorial identifies Selenium 4.49.0. Match API details to the Selenium version and language binding in your project; the example above uses Python-specific expected conditions.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
