How to Capture Screenshots of Paginated Search Results with Selenium

To capture every state in paginated search results with Selenium, save a screenshot of the current results, click the next-page control, wait for a page-specific change, and repeat until the site’s final-page condition is true. Use a driver screenshot for the current browsing context or a WebElement screenshot for just the results region. The selectors and completion condition must match the site you are automating.

The example below uses Python. Replace .results, a.next, and the terminal-state check with selectors verified on the target page. This is a pattern, not a universal locator set.

1. Set up Selenium and inspect the page

Install Selenium and a browser driver supported by your environment. Selenium Manager can manage drivers in many standard setups; consult the Selenium documentation for current browser and driver setup guidance.

python -m pip install selenium

Before writing the loop, inspect the page in browser developer tools and identify:

A selector for the results container or a stable result identifier.
A selector for the next-page control.
A condition that means the last page has been reached, such as a disabled next button or an explicit current-page indicator.
A state change that proves the new results are ready, such as a changed page number, URL, or first result identifier.

If the results are inside an iframe, switch into that frame before finding these elements. If a consent, sign-in, access-limit, or challenge page appears, handle it according to the site’s permitted access and terms; do not save it as though it were another results page.

2. Capture each page with Python

from pathlib import Path

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

START_URL = "https://example.com/search?q=selenium"
RESULTS = (By.CSS_SELECTOR, ".results")
NEXT = (By.CSS_SELECTOR, "a.next")
CURRENT_PAGE = (By.CSS_SELECTOR, ".pagination .current")

output_dir = Path("screenshots")
output_dir.mkdir(parents=True, exist_ok=True)

options = webdriver.ChromeOptions()
# Uncomment to run without opening a visible browser window:
# options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 10)
max_pages = 100  # Defensive bound; choose a suitable limit for your task.

try:
    driver.set_window_size(1440, 1000)
    driver.get(START_URL)

    for page_number in range(1, max_pages + 1):
        results = wait.until(EC.visibility_of_element_located(RESULTS))
        driver.save_screenshot(str(output_dir / f"results-{page_number:03}.png"))

        # Reacquire the control on each iteration. The DOM may have changed.
        next_button = wait.until(EC.presence_of_element_located(NEXT))
        aria_disabled = next_button.get_attribute("aria-disabled")
        disabled = next_button.get_attribute("disabled") is not None

        if not next_button.is_displayed() or not next_button.is_enabled() or disabled or aria_disabled == "true":
            break

        old_page = wait.until(EC.visibility_of_element_located(CURRENT_PAGE)).text.strip()
        next_button.click()

        # This works when the current-page label changes after the click.
        wait.until(lambda d: d.find_element(*CURRENT_PAGE).text.strip() != old_page)
    else:
        raise RuntimeError(f"Stopped at the safety limit of {max_pages} pages")
finally:
    driver.quit()

The code saves the currently visible browsing context to a numbered PNG file. The 100-page bound prevents a faulty or missing end condition from looping indefinitely. Pick a bound appropriate to the site and task. A timeout raises an error rather than silently treating an incomplete capture as a finished run.

Choose the right wait condition

A successful click does not prove that JavaScript-rendered results have updated. A browser’s document-ready state covers document loading, not every later application update. Selenium recommends synchronizing on observable states and warns that mixing implicit and explicit waits can lead to unpredictable wait times. Use explicit waits for the transition you need; avoid setting an implicit wait alongside them. See Selenium waiting strategies.

The example waits for the current-page label to change. Other sites may need a different condition:

Results container is replaced: save the old element reference, click next, and wait for EC.staleness_of(old_results); then reacquire the container.
Results update in place: wait for a page marker, URL, result count, or stable result identifier to change. Staleness will not work if the same container remains in the DOM.
Navigation changes the URL: record the old URL and wait until driver.current_url differs, then wait for the results to be visible.
Loading indicator appears: wait for it to disappear and then for the updated results or page marker. Disappearance alone may not prove the intended results loaded.

Do not reuse old result or pagination element references after a transition that rebuilds the DOM. Selenium reports a stale element when a reference no longer points to an attached element; find it again after the update. See Selenium’s stale element guidance.

3. Choose what each screenshot contains

Capture method	Use it when	Python
Current browsing context	You want the visible page, including surrounding navigation, headings, or status content.	`driver.save_screenshot("page.png")`
Results element	You want only the results region and your driver supports element screenshots.	`results.screenshot("results.png")`
Full document	You need one tall image of the entire document rather than the current view.	Check the browser-specific driver API; support is not uniform across the general screenshot API.

Selenium’s Python API also offers byte-returning screenshot methods, useful when you want to process or upload an image without first saving it to a file. The browser-context and element screenshot methods are documented in the WebDriver API and WebElement API.

# Save only the results element
results = wait.until(EC.visibility_of_element_located(RESULTS))
results.screenshot("results-only.png")

# Get screenshot bytes instead of saving directly
png_bytes = driver.get_screenshot_as_png()
Path("page.png").write_bytes(png_bytes)

For consistent dimensions, set a stable window size before the first capture and leave it unchanged through pagination. A driver screenshot should not be assumed to create an arbitrarily tall full-page image in every browser. Firefox’s Python API documents a full-document screenshot method; verify the current support and behavior of your chosen browser and driver if that scope is required.

4. Adapt the loop to the site’s pagination

Numbered pagination

If the site uses numbered links, locate the next page number from the current marker, click it, and wait for the marker or results identifier to change. Avoid deriving a final page solely from the number of visible links if the site uses truncated pagination such as “1, 2, 3, …, 20.”

Next button stays enabled on the last page

Some sites leave the next control enabled even when it no longer advances. Compare a page marker, URL, or result identifier before and after the click. If the expected value does not change before the timeout, record diagnostics and stop or fail the run; do not save the same state under a new page number.

Infinite scrolling

Infinite scroll is not ordinary pagination. Scroll the results area or page to trigger loading, wait for the result count or last result identifier to increase, and capture at the intended boundaries. Define those boundaries first: one screenshot per newly loaded batch and one screenshot of the completed list are different deliverables.

Results in an iframe

frame = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "iframe.results-frame")))
driver.switch_to.frame(frame)

# Locate and interact with results inside the frame here.

# Return to the top-level document when done.
driver.switch_to.default_content()

Selenium initially interacts with the top-level document. Switch to the relevant frame before locating its contents; see Selenium’s frame documentation.

5. cURL, Python, and Node.js alternatives

Selenium’s browser automation API is language-specific. The Python code above is the complete Selenium example. These cURL, Python, and Node.js examples are for ScreenshotNeo’s screenshot API, which captures a URL in one request; they do not click through a site’s pagination. Use browser automation when you must visit and capture each interactive state.

ScreenshotNeo accepts the parameter names used by other screenshot APIs, which can make switching easier. See the ScreenshotNeo documentation for request options and API details.

Or skip the browser setup

If you need a screenshot of a URL rather than a sequence of Selenium-driven pagination states, ScreenshotNeo is a website screenshot API and MCP server for developers. One GET request returns PNG, JPEG, WebP, or PDF. For a page that exposes a particular state directly by URL, start with this call:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
const image = Buffer.from(await res.arrayBuffer());
await import('node:fs/promises').then(fs => fs.writeFile('shot.webp', image));

Cookie banners are accepted like a visitor, and 60+ known consent platforms, newsletter popups, and chat widgets are removed before capture; each step can be turned off.
Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing. Responses identify the page verdict and billing status in headers.
An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
The free plan includes 1,000 screenshots a month with no card. Paid plans start at $5 for 3,000 screenshots. Every feature is available on every plan.

Create a free ScreenshotNeo account and get 1,000 screenshots a month with no card.

6. Reliability, performance, and cost

Wait for meaning, not elapsed time. Prefer a condition that confirms the new page is ready over an arbitrary sleep. Fixed sleeps waste time on fast pages and can still be too short on slow ones.
Keep captures deterministic. Use the same browser, viewport, zoom, and relevant page state for every iteration. If content changes over time, record the URL and page marker alongside each image.
Bound the run. Set a maximum page count, use a real final-page signal, and surface timeouts. This prevents accidental infinite loops and makes partial runs visible.
Account for browser work. Each page transition and image capture consumes browser time and memory. Save files incrementally rather than holding a large sequence of image byte arrays in memory.
Check site access rules. Automated requests may encounter consent, authentication, bot checks, or usage limits. Follow the target site’s terms and permitted access.
Plan for storage. Image size depends on dimensions and page content. Choose PNG for lossless output; use another supported browser-side workflow if the deliverable requires a different format.

No universal duration or cost figure applies: page weight, browser, network, wait condition, number of pages, and storage destination all affect a run. Selenium itself does not price screenshots per capture; your infrastructure and browser execution environment determine operational cost.

7. Troubleshooting

Symptom	Likely cause	Fix
The screenshot repeats the previous results.	The click returned before the app updated, or the wait watched the wrong state.	Wait for a changed page marker, URL, result identifier, or replaced container. Confirm the chosen value actually changes on this site.
`TimeoutException` after clicking next.	The selector is wrong, the page did not advance, the target state differs, or a challenge/sign-in state appeared.	Inspect the current URL, page source, visible page, and marker. Correct the locator or predicate; handle access states according to the site’s rules.
`StaleElementReferenceException`.	The page rebuilt a node referenced by an earlier lookup.	Re-find the results and next control after the transition. Use staleness as a wait only when the old container is expected to be replaced.
The click fails or hits an overlay.	The control is hidden, disabled, covered, or not yet interactable.	Wait for visibility and enabled state, inspect overlays and disabled attributes, and use the site’s actual interactive control.
`NoSuchElementException` for results or next.	Incorrect selector, content still loading, or the element lives in a frame or shadow root.	Verify the DOM selector, wait for presence, switch to the correct frame, and account for shadow DOM where applicable.
Output dimensions vary.	Window size, browser scaling, or responsive layout changed.	Set a fixed window size before navigation and keep it for all captures; check browser zoom and device scale settings.
The loop never ends.	The final-page test is missing or unreliable, or the next control remains enabled.	Use a site-confirmed terminal signal, detect an unchanged page marker after a click, and retain a maximum-page safety bound.
Waits take unexpectedly long.	Implicit and explicit waits are mixed, or the predicate repeatedly performs slow lookups.	Use one wait strategy, keep explicit predicates focused, and set a deliberate timeout based on the page behavior.

8. FAQ

Can Selenium capture a screenshot without saving it to disk?

Yes. Use a byte-returning screenshot method such as get_screenshot_as_png() for the browsing context or the corresponding element screenshot API, then pass the bytes to your storage or image-processing code.

Should I use one screenshot per page or one tall image?

For paginated results, one image per state usually preserves the relationship between each page and its page marker. A tall full-document image is a separate requirement; verify support for it in your specific browser and driver.

Why not use a fixed sleep after every click?

A fixed delay does not confirm that the results changed. It can make fast pages slower and slow pages flaky. Wait for a condition tied to the site’s actual transition.

Can the ScreenshotNeo call capture every pagination state automatically?

The one-call URL screenshot captures a page URL; it does not run this Selenium click loop. It is suitable when the desired state has its own directly accessible URL. Use browser automation for interactive pagination that must be traversed.