Selenium Screenshot of an Infinite Scroll Page Times Out: Troubleshooting Guide

If a Selenium screenshot workflow times out on an infinite-scroll page, first identify which command timed out: navigation, a scrolling script, an explicit wait, or screenshot saving. Then synchronize the next step with a real page signal, such as a new item appearing or the item count increasing. A completed navigation does not mean JavaScript-driven content has finished loading, and increasing every timeout can hide the actual failure.

1. Identify the operation that timed out

Record the exact command and exception before changing settings. Selenium has separate waits for different operations:

Failure point	What the timeout means	First check
driver.get(url)	Navigation did not satisfy the configured page-load strategy before the page-load timeout.	Does the site keep network activity open, or is the selected page-load strategy waiting longer than expected?
execute_async_script(...)	The asynchronous script did not finish before the script timeout.	Does every script path call its completion callback?
WebDriverWait(...).until(...)	The chosen condition did not become true before the explicit-wait deadline.	Is the condition observable and appropriate for this page?
save_screenshot(...) or screenshot retrieval	The failure occurred during capture or saving, not necessarily during page loading.	Check the exception, output path, browser state, and whether the page is still rendering.

Keep the exception text and a short log of the last successful command. A timeout at navigation should not be diagnosed as a scroll-loop failure, and a screenshot-save error should not be treated as a page-load timeout.

2. Why infinite scroll makes navigation waits misleading

Selenium waits for a configured document readyState during navigation. That state covers assets defined in the HTML; JavaScript can continue changing the page afterward, and elements needed for later interaction may not yet exist. Infinite-scroll pages commonly fetch and render more items only after scrolling, so there may be no single moment when the entire page is “loaded.”

The useful rule is to wait for the state required by the next action. Before scrolling, wait for the initial content you need. After scrolling, wait for evidence that the site responded, such as an increased result count, a newly visible item, or a loading indicator disappearing. Selenium’s waiting strategies documentation explains navigation waits, explicit waits, and the difference between document readiness and application state.

3. Runnable Python example: wait for new content while scrolling

This example captures the current browser window after the page stops producing new items, reaches a scroll limit, or reaches a maximum iteration count. Replace the URL and item selector with selectors for the target site. The example deliberately treats the item count as a site-specific signal; a site that recycles DOM nodes or uses virtual scrolling needs a different signal.

from pathlib import Path
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

URL = "https://example.com/infinite-feed"
ITEM_SELECTOR = "article.feed-item"  # Replace with the site's item selector
OUTPUT = Path("infinite-scroll.png")
MAX_SCROLLS = 30
STABLE_ROUNDS_TO_STOP = 3
WAIT_SECONDS = 10

options = webdriver.ChromeOptions()
# Uncomment to run without a visible browser window:
# options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)
driver.set_page_load_timeout(30)
driver.set_script_timeout(15)

try:
    try:
        driver.get(URL)
    except TimeoutException:
        # Navigation can time out while leaving a usable partial page.
        # Inspect and log this state for your site before deciding to continue.
        print("Navigation timed out; current URL:", driver.current_url)
        print("Document readyState:", driver.execute_script("return document.readyState"))

    wait = WebDriverWait(driver, WAIT_SECONDS)
    wait.until(lambda d: len(d.find_elements(By.CSS_SELECTOR, ITEM_SELECTOR)) > 0)

    previous_count = len(driver.find_elements(By.CSS_SELECTOR, ITEM_SELECTOR))
    stable_rounds = 0

    for step in range(MAX_SCROLLS):
        old_height = driver.execute_script("return document.documentElement.scrollHeight")
        old_y = driver.execute_script("return window.scrollY")
        print(f"step={step} y={old_y} height={old_height} items={previous_count}")

        driver.execute_script("window.scrollTo(0, document.documentElement.scrollHeight)")

        try:
            wait.until(
                lambda d: len(d.find_elements(By.CSS_SELECTOR, ITEM_SELECTOR)) > previous_count
                or d.execute_script("return document.documentElement.scrollHeight") > old_height
            )
        except TimeoutException:
            # No observable growth before the deadline. This can mean the feed
            # ended, the selector is wrong, or the site uses another signal.
            stable_rounds += 1
            print("No detected growth; stable round", stable_rounds)
        else:
            stable_rounds = 0

        current_count = len(driver.find_elements(By.CSS_SELECTOR, ITEM_SELECTOR))
        current_height = driver.execute_script("return document.documentElement.scrollHeight")
        current_y = driver.execute_script("return window.scrollY")
        print(f"step={step} y={current_y} height={current_height} items={current_count}")

        if current_count > previous_count:
            previous_count = current_count
        if stable_rounds >= STABLE_ROUNDS_TO_STOP:
            break
        if current_height == old_height and current_y == old_y:
            break

    # WebDriver's standard screenshot methods capture the current window.
    driver.save_screenshot(str(OUTPUT))
    print("Saved viewport screenshot:", OUTPUT.resolve())
finally:
    driver.quit()

The stop rules are safeguards, not proof that every possible feed item was loaded. Adjust MAX_SCROLLS, STABLE_ROUNDS_TO_STOP, and the wait condition for the page. A feed may require scrolling a particular container rather than the window, clicking a “load more” control, or waiting for a network-driven loading marker to disappear.

4. Choose a wait condition that matches the page

Wait for an item count to increase

When each fetched result adds a new matching node, record the current count and wait until it grows. This is the example’s primary condition. Use a selector scoped to the feed so unrelated page elements do not satisfy the wait.

Wait for a particular item

If the next expected item has a stable identifier or text, wait for that item to become present or visible. This is often clearer than waiting for the whole page height to change.

Wait for loading to finish

If the site exposes a loading indicator, wait for it to disappear after the scroll triggers a request. Confirm the indicator is actually inserted and removed on the target page; waiting for an element that never appears or is hidden in a different way will time out.

Wait for a meaningful height change

Document height can be a useful fallback, but ads, images, sticky elements, and layout shifts can change height without adding feed items. Virtualized feeds can add content while keeping height constant. Treat height as supporting evidence, not a universal completion signal.

Do not use a fixed sleep as the main synchronization method

A fixed delay can be useful briefly while diagnosing a page, but it is unreliable as the primary wait: it wastes time when content arrives quickly and still fails when the page is slower than the chosen delay. Prefer a condition tied to the application state your next step needs.

5. Configure Selenium timeouts for the operation

Set timeouts deliberately and keep them distinct. The Selenium Python API documents set_page_load_timeout, set_script_timeout, and screenshot methods such as save_screenshot on the current window: WebDriver API documentation.

driver.implicitly_wait(0)            # Do not mix implicit and explicit waits
 driver.set_page_load_timeout(30)     # Navigation deadline, in seconds
 driver.set_script_timeout(15)        # execute_async_script deadline, in seconds

Remove the leading space before driver.set_page_load_timeout if copying the snippet into a block that begins at column zero; alternatively use this correctly aligned version:

driver.implicitly_wait(0)
driver.set_page_load_timeout(30)
driver.set_script_timeout(15)

These values are examples, not recommended universal limits. Do not increase them all to the same large number. A longer deadline can be appropriate if a known operation legitimately takes longer, but it does not fix an incorrect selector, a script that never completes, or a wait condition the page cannot satisfy. Selenium cautions against mixing implicit and explicit waits because their combined timing can be unpredictable.

6. Viewport screenshot versus full-page capture

The standard Python WebDriver screenshot methods capture the current window. Scrolling the page and calling save_screenshot therefore produces a viewport image at the current scroll position; it does not automatically stitch every viewport into one full-document image. Decide which deliverable you need:

• Viewport: capture one visible region, possibly after scrolling to the desired section.
• Full-page image: verify the browser-specific capture technique and browser support you plan to use. Selenium’s generic screenshot method is not a cross-browser guarantee of full-page capture.
• Long-page record: capture multiple viewport screenshots with their scroll positions, or use a browser-specific full-page mechanism after verifying its behavior for the target browser.

Infinite-scroll pages may load more content only while scrolling, so a full-page attempt before triggering those loads can omit items. If content is virtualized, earlier items may be removed from the DOM as later ones appear, making one final full-page capture insufficient. Record the browser, capture method, and whether the page retains previously loaded items.

7. Troubleshooting common failures

Symptom	Likely cause	Fix
Timeout raised by driver.get	Navigation did not meet the configured load condition in time.	Log the exception and resulting page state. Review the page-load strategy and timeout; do not assume scrolling caused it.
Timeout from execute_async_script	The script did not invoke its completion callback, or its work exceeded the script deadline.	Check every success and error path in the script; increase the script timeout only if the operation legitimately needs more time.
Explicit wait for items never succeeds	The CSS selector is wrong, items are inside an iframe or shadow root, content has not been triggered, or the site uses a different loading signal.	Inspect the DOM after the scroll, confirm the selector and browsing context, and choose an observable condition the site actually satisfies.
Wait succeeds immediately without new content	The condition is already true before the scroll, or it matches unrelated content.	Compare against a baseline count or wait for a specific new item/state rather than mere presence.
Scroll loop runs until its maximum	The stop signal never stabilizes, new content keeps arriving, or the site appends unrelated nodes.	Log count, scroll position, document height, and loading state; set a task-appropriate cap and inspect the last iterations.
Page height stays constant while items change	The site may virtualize or recycle content, or append within a fixed-height scrolling container.	Observe the feed container and item identities, and scroll that container rather than relying on document height.
Only the last visible section appears in the image	The standard screenshot captured the current viewport.	Use a verified browser-specific full-page method or save multiple viewport captures with positions.
Screenshot is blank or incomplete	Capture ran before rendering settled, the browser is on an error/partial page, or the relevant content was outside the viewport.	Check current URL, ready state, item count, and visible page state immediately before capture; wait on the relevant signal.
Wait duration is unexpectedly long	Nonzero implicit waits are interacting with explicit waits.	Set implicit wait to zero and use explicit waits for the conditions that matter.

8. Reliability, runtime, and cost considerations

• Reliability: use explicit state checks and bounded loops. Log the URL, browser, scroll position, document height, item count, and wait condition on each iteration so an intermittent failure is diagnosable.
• Runtime: each scroll and explicit wait adds time. Stop when a meaningful end condition is reached; cap iterations and per-step waits so a never-ending feed cannot run indefinitely.
• Repeatability: pages can change due to personalization, geolocation, authentication, consent state, or dynamic content. Use the same browser context and inputs when comparing captures, and avoid assuming a universal selector or end condition.
• Cost: Selenium itself does not determine your hosting or browser-grid charges. Repeated retries, long waits, and browser sessions consume whatever compute or grid resources your setup uses; keep retries bounded and target the actual failed operation.

9. Or skip the browser setup

For a one-call screenshot, ScreenshotNeo accepts a URL and returns an image or PDF. Its capture flow removes cookie banners, newsletter popups, and chat widgets before the shot. Bot checks, blank pages, and failed loads are never billed; responses identify page verdict and billing status. An MCP server gives AI agents tools for screenshots and PDFs. The free plan includes 1,000 screenshots a month with no card, and paid plans start at $5 for 3,000. See the API documentation for the request options and behavior.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/infinite-feed -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://example.com/infinite-feed"},
    timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({
  access_key: 'YOUR_API_KEY',
  url: 'https://example.com/infinite-feed'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
const fs = await import('node:fs/promises');
await fs.writeFile('shot.webp', Buffer.from(await res.arrayBuffer()));

These calls capture the requested page URL; they do not guarantee that an infinite-scroll application has loaded every item that requires user scrolling. Use the DIY Selenium flow when the required result depends on interacting with the page and reaching a specific content state. Sign up for 1,000 free screenshots a month, with no card.

10. FAQ

Does document.readyState == "complete" mean the feed is ready?

No. It describes document resource loading, not whether later JavaScript-driven requests and rendering have finished.

Can I just raise the page-load timeout?

Only if navigation itself is the slow operation. First confirm the failing command; a script or explicit-wait timeout needs a fix at that step.

Will one Selenium screenshot call capture an entire infinite page?

Do not assume so. The standard Python API captures the current window, and full-page behavior depends on the browser-specific method.

Why does the feed never reach a stable height?

It may load continuously, change layout as media appears, or use a fixed-height/virtualized container. Use item identity or loading state as the signal when height is not meaningful.

Primary references

• Selenium: Waiting Strategies
• Selenium Python WebDriver API
• ScreenshotNeo API documentation