How to Handle Pagination in Selenium

Answer: Treat pagination as a sequence of observable page-state changes. Read and process the current results, find and click the Next control, wait until the application shows that the intended transition completed, then locate the results and controls again. Stop when the site’s own end condition says there are no more pages.

Selenium has no universal pagination command: the markup, navigation behavior, and end signal belong to the application. The selectors and state-change condition in the example below are placeholders; inspect the target page and adapt them.

1. Identify the pagination behavior

Before writing the loop, inspect the page’s DOM and determine how it advances. Common patterns include:

A Next button or link that navigates to another URL.
A Next control that updates results in place and changes a current-page label.
A client-side rerender that replaces the result container.
Numbered links, a “Load more” button, or infinite scroll. These need a different advancement action, but the same principle applies: wait for evidence of a state change.

Choose stable locators for the results, current-page marker if present, and next control. Prefer attributes or accessible names that identify the control reliably. Avoid assuming that a particular CSS class, button label, or disabled-state representation applies to every site.

2. A reliable Python pagination loop

This example uses Selenium’s Python binding and explicit waits. It assumes the page updates a visible current-page marker after clicking Next. Replace the URL, selectors, and processing function with the ones for your application.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

START_URL = "https://example.com/catalog"
RESULTS = (By.CSS_SELECTOR, "table tbody tr")
NEXT = (By.CSS_SELECTOR, "button.next")
CURRENT_PAGE = (By.CSS_SELECTOR, ".current-page")


def process_row(row):
    # Replace this with your application-specific extraction or storage.
    print(row.text)


def next_is_unavailable(driver):
    """Example end check. Adapt it to the site's actual disabled state."""
    buttons = driver.find_elements(*NEXT)
    if not buttons:
        return True
    button = buttons[0]
    aria_disabled = button.get_attribute("aria-disabled")
    disabled = button.get_attribute("disabled")
    return not button.is_enabled() or aria_disabled == "true" or disabled is not None


driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)

try:
    driver.get(START_URL)
    wait.until(lambda d: d.find_elements(*RESULTS))

    # A safety bound prevents an unexpected page state from looping forever.
    max_pages = 500
    pages_seen = set()

    for _ in range(max_pages):
        # Re-find result rows on every iteration; a rerender can invalidate old elements.
        rows = driver.find_elements(*RESULTS)
        for row in rows:
            process_row(row)

        if next_is_unavailable(driver):
            break

        old_page = driver.find_element(*CURRENT_PAGE).text.strip()
        if old_page in pages_seen:
            raise RuntimeError(f"Pagination returned to an already visited page: {old_page}")
        pages_seen.add(old_page)

        # Reacquire the control immediately before clicking it.
        driver.find_element(*NEXT).click()

        # Wait for evidence that this application's page marker changed.
        try:
            wait.until(
                lambda d: d.find_element(*CURRENT_PAGE).text.strip() != old_page
            )
        except TimeoutException as exc:
            raise TimeoutException(
                f"Next was clicked, but the current-page marker did not change from {old_page!r}"
            ) from exc
    else:
        raise RuntimeError(f"Stopped after reaching the safety limit of {max_pages} pages")
finally:
    driver.quit()

The example’s `next_is_unavailable` function checks several common signals, but the application may use a different one. For example, it may expose `aria-disabled=”true”`, a CSS class, a final-page marker, or no Next control at all. Confirm the real DOM and implement the relevant condition. `is_enabled()` alone may not reflect a site’s custom disabled styling.

What the loop does

Waits for results on the initial page.
Processes the current results before advancing.
Checks the application’s end condition.
Records the current-page value as a loop guard.
Clicks a freshly located Next control.
Waits until the page marker changes, then starts the next iteration and reacquires the results.

If a valid page can have zero rows, do not use the presence of result rows as the only initial readiness or transition signal. Wait for a page marker, a result container, or another application-specific signal instead.

3. Choose a wait condition that proves the transition

A click only means Selenium sent an interaction. It does not prove that the application has finished updating. Selenium’s documentation explains that a document reaching the expected ready state does not guarantee that JavaScript-driven content is ready. Explicit waits poll for a chosen condition until it becomes true or the timeout expires. [Selenium waiting strategies]

Transition signal	Use it when	Watch out for
Changed page number or label	The page exposes a reliable current-page marker.	The marker must update for each successful transition. Include the expected old value in the condition.
Changed URL	Next navigates to a distinct URL for each page.	Some apps update content without changing the URL. A URL change by itself may also precede content readiness.
Old result container becomes stale	The app replaces the result container during a rerender.	Staleness proves the old node was detached, not necessarily that the new results are ready. Follow it with a wait for the new container or its contents.
Loading indicator disappears	The app shows a dependable loading state around each update.	A very brief or absent indicator can make this signal unreliable. Pair it with a changed page marker or content.
Result content changes	The results have a stable identity that changes across pages.	Repeated or empty content can make a text comparison ambiguous. Prefer a page marker or URL if available.

There is no universally best signal. Choose the observable that demonstrates that this application reached the intended page. Selenium documents expected conditions for visibility, text, and staleness; syntax and availability vary by language binding. The Selenium documentation notes that .NET no longer supports the Expected Conditions classes in Selenium 4, so check the API for your binding. [Selenium expected conditions]

Wait for a replaced result container

If a page rerender replaces the results, wait for the old container to go stale and then wait for the new results. The Python API documents `staleness_of` for this case. [Selenium Python expected conditions API]

from selenium.webdriver.support import expected_conditions as EC

old_container = driver.find_element(By.CSS_SELECTOR, "#results")
driver.find_element(*NEXT).click()
wait.until(EC.staleness_of(old_container))
new_rows = wait.until(lambda d: d.find_elements(*RESULTS))

Use this pattern when the container is actually replaced. If it remains in the DOM and only its contents change, staleness will never occur; wait for a page label, URL, or content change instead.

4. Handle other pagination patterns

When clicking Next changes the URL, capture the old URL and wait for a different one. Then wait for the destination page’s results or marker. This avoids treating navigation start as completed content readiness.

old_url = driver.current_url
driver.find_element(*NEXT).click()
wait.until(lambda d: d.current_url != old_url)
wait.until(lambda d: d.find_elements(*RESULTS))

Numbered page controls

If the interface has numbered buttons, use the current page and target page to select the next number, then wait for the current-page marker or URL to match that target. Do not assume page numbers are contiguous or that a particular numbered button remains in the DOM after each update.

Load more

For a “Load more” control, process the current results, record a stable identifier for the last result, click the control, and wait for the result count or last-result identifier to change. Stop when the control disappears or is disabled according to the site’s actual markup.

Infinite scroll

Scroll the page or result container to trigger loading, then wait for a new result identifier or a changed result count. A fixed delay after scrolling is not proof that more results arrived. Stop when the page exposes an end-of-results signal or repeated scrolling produces the documented terminal state.

5. End conditions and duplicate protection

Pagination has no universal terminator. Check for whichever signal the application provides:

Next is absent from the DOM.
Next is disabled through a native disabled attribute, `aria-disabled`, or another verified state.
A current-page marker matches a known last page.
The application displays a final-page or end-of-results marker.

Use a maximum-page bound or track visited page numbers and URLs to catch loops, repeated transitions, and unexpected redirects. If you are collecting data, make processing idempotent where possible: a retry after a timeout can otherwise store the same page twice. Record the page identifier alongside each result so you can detect gaps and duplicates.

6. Troubleshooting

Symptom	Likely cause	Fix
Selenium reads the same page after clicking Next	The code proceeds before the JavaScript update, the click did not take effect, or the chosen marker does not change.	Wait for a verified state transition. Check whether the control is interactable and inspect the page’s URL, page marker, results, and loading state.
Timeout waiting for the page marker	The selector is wrong, the marker is not updated by this interaction, Next is disabled, or the request failed.	Inspect the DOM after the click. Choose a signal that changes on this application and include the old and expected state in timeout diagnostics.
Stale element reference	A rerender detached the element previously located.	Discard the old reference and locate the results and controls again after the transition. Use a staleness wait only if the old node is expected to be replaced.
Element click intercepted or not clickable	An overlay, animation, sticky element, or off-screen control blocks interaction.	Wait for the overlay or animation to finish, ensure the control is visible and enabled, or scroll it into view. Avoid JavaScript-triggered clicks unless the application genuinely requires them.
Loop never stops	The end condition does not match the site’s disabled state, or the page marker repeats.	Inspect the final page’s markup and use its actual terminal signal. Add a page limit and visited-page detection.
Rows are missing or duplicated	Results were read during an update, a page was retried, or old elements were reused.	Wait for the transition before extraction, reacquire rows on every page, and deduplicate using a stable record identifier if the task permits.
Implicit and explicit waits behave unpredictably	Both wait strategies are active and their timeouts interact.	Prefer explicit waits for pagination transitions and avoid casually mixing them with an implicit wait. Selenium warns that mixing the two can produce unpredictable wait times. [Selenium waiting strategies]

7. Performance, reliability, and cost

Wait only for the state you need

Use a condition-based explicit wait with a bounded timeout. Fixed sleeps may be too short on a slow run or waste time when longer than necessary. Selenium recommends explicit waits for specific conditions and warns about mixing implicit and explicit waits. [Selenium waiting strategies]

Keep the timeout long enough for normal application latency, but report a clear failure when the expected state never appears. A timeout should preserve useful context such as the current URL, page marker, and page being processed. Do not silently skip a page after a failed transition.

Keep extraction and retries safe

Process one page at a time unless the application and your collection requirements support a different approach. Store a checkpoint after each completed page if the run is expensive to restart. Retry only after deciding whether the click may already have advanced the application; blindly clicking again after a timeout can skip a page.

Selenium’s documentation does not provide pagination-specific speed, reliability, or cost benchmarks. Actual runtime depends on the site, network, browser, and work performed for each result. Measure the workflow in its intended environment rather than assuming a fixed per-page duration.

8. Or skip the browser setup

If your goal is to capture screenshots of pages in a paginated workflow, ScreenshotNeo is a website screenshot API and MCP server from Yorker Media. Selenium remains useful when you need to interact with the page and collect data; for a screenshot, a single API request can return an image or PDF. See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/catalog?page=2 -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://example.com/catalog?page=2"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://example.com/catalog?page=2' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie and consent banners are accepted and removed before the shot, along with supported newsletter popups and chat widgets; each step can be turned off.
Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed. Responses identify the page verdict and billing status in headers.
An MCP server provides `take_screenshot`, `get_page_info`, and `capture_pdf` tools for Claude, Cursor, and other MCP clients.
1,000 screenshots a month are free with no card. Paid plans start at $5 for 3,000 screenshots.

9. FAQ

Can Selenium click Next until there are no more pages?

Yes. Put the site-specific Next availability check and transition wait inside a loop, process each page once, and include a safety bound or visited-page guard.

Should I use a fixed sleep after clicking?

Not as the main synchronization method. Wait for a condition that proves the application reached the next state; a fixed delay cannot confirm that it did.

Why does the page load but still show old results?

The document may be loaded while client-side JavaScript is still updating the results. Wait for the application’s page marker, URL, content, or container state to change.

Do I need to set an implicit wait?

Not to paginate. Explicit waits let you target the condition that marks each transition. If your session also uses an implicit wait, account for Selenium’s warning that combining the two can produce unpredictable timing.