Capture lazy-loaded web content with Selenium WebDriverWait

Selenium’s navigation wait does not guarantee that JavaScript-driven or lazy-loaded content is ready. Use WebDriverWait after the action that triggers loading, and wait for a condition that matches what your next step needs: an element’s presence, its visibility, a loading indicator disappearing, or a page-specific signal. No generic selector or fixed delay proves that every item on every page has loaded.

This guide uses Python. It covers a basic explicit wait, page-specific conditions, infinite scrolling, errors, and operational tradeoffs. The selectors and timeout values are examples: replace them with signals from the page you are allowed to automate.

1. Why navigation completion is not content completion

Selenium navigation waits for a configured document readiness state, normally complete. That describes the browser’s document loading state; JavaScript can still change the page afterward. A lazy-loaded card, image, or other element needed by your next action may not exist yet. Selenium’s waiting strategies explain this distinction.

An explicit wait repeatedly checks a condition until it succeeds or the timeout expires. This makes the wait describe the state your task needs instead of assuming that a duration is enough. For example, waiting until one result card is present can be adequate if you need to read that card. It does not establish that a growing list or an entire infinite-scroll page is complete.

2. Install Selenium and prepare a browser

Install Selenium in the Python environment used by your script:

python -m pip install selenium

For a runnable browser example, install a supported browser such as Chrome. Selenium Manager can help Selenium obtain a compatible driver in common setups; see the official Selenium Manager documentation. Browser and driver provisioning may differ in containers, restricted networks, and managed environments.

3. Wait for the content your task needs

This example opens a page, waits for matching result cards, and reads their text. The selector is illustrative, not a selector verified for a particular website.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

URL = "https://example.com/results"
CARD_SELECTOR = ".result-card"  # Replace with a selector from the target page.
TIMEOUT_SECONDS = 15  # Illustrative; tune for the page and environment.

driver = webdriver.Chrome()
try:
    driver.get(URL)
    wait = WebDriverWait(driver, TIMEOUT_SECONDS)

    # Wait for the elements needed by the next operation.
    cards = wait.until(
        EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, CARD_SELECTOR)
        )
    )
    captured_text = [card.text for card in cards]
    print("Found", len(captured_text), "matching cards")
    for text in captured_text:
        print(text)
except TimeoutException as exc:
    print(f"Expected page state did not appear within {TIMEOUT_SECONDS}s")
    raise
finally:
    driver.quit()

presence_of_all_elements_located succeeds when matching elements are present at the time Selenium checks. It does not wait for a dynamically growing set to stop growing. If the task is to click or read displayed content, use a condition that checks visibility instead.

Choose the condition for the next action

Task needs	Condition pattern	What success tells you
Locate a matching element	presence_of_element_located(locator)	An element matching the locator exists in the DOM.
Read or interact with a displayed element	visibility_of_element_located(locator)	A matching element is present and visible.
Wait for a previous element to be replaced or removed	staleness_of(old_element)	The old reference is no longer attached to the DOM.
Wait for expected text	text_to_be_present_in_element(locator, text)	The specified text is present in the matching element.
Wait for a site-specific loading rule	A custom callable passed to until()	Whatever truthy result your page-specific rule returns.

Selenium documents these and other expected conditions. Presence, visibility, and expected text each describe a limited condition; pick the one that matches the next step.

4. Adapt the wait to the page’s loading behavior

Find a real signal in the target page rather than relying on a guessed class or an arbitrary pause. Inspect the page’s DOM and loading behavior in a browser, and decide what “ready” means for the task.

Wait for a loading indicator to disappear

If the page shows a loading indicator during the relevant request, disappearance can be useful. It only describes that indicator, so confirm that its lifecycle corresponds to the content you need.

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

spinner = (By.CSS_SELECTOR, ".results-loading")
results = (By.CSS_SELECTOR, ".result-card")

wait.until(EC.visibility_of_element_located(spinner))
wait.until(EC.invisibility_of_element_located(spinner))
visible_cards = wait.until(EC.visibility_of_all_elements_located(results))

If the spinner disappears before results render, wait for the results condition as well. If the spinner never appears for a fast response, the first wait may time out; use it only if that state is part of the page’s actual behavior.

Wait for text or a known count

If a result count is displayed and trustworthy for your task, wait until it reaches the expected value. The following custom condition is an example; adapt its selector and expected count.

from selenium.webdriver.common.by import By

count_locator = (By.CSS_SELECTOR, ".result-count")
expected_count = 20

def count_reached(driver):
    elements = driver.find_elements(*count_locator)
    if not elements:
        return False
    text = elements[0].text
    return text if text == str(expected_count) else False

count_text = wait.until(count_reached)
print("Page reports", count_text, "results")

A displayed count can describe total available results rather than results currently rendered. Confirm what it means before using it as a completion rule.

Use a custom condition for application state

until() accepts a callable. It keeps checking until the callable returns a value other than False (or another falsey value); then it returns that value. This is useful when the relevant condition combines page-specific checks.

from selenium.webdriver.common.by import By

cards_locator = (By.CSS_SELECTOR, ".result-card")
ready_locator = (By.CSS_SELECTOR, "[data-results-ready='true']")

def results_ready(driver):
    ready = driver.find_elements(*ready_locator)
    cards = driver.find_elements(*cards_locator)
    if ready and cards:
        return cards
    return False

cards = wait.until(results_ready)
print("Ready cards:", len(cards))

This example assumes the page exposes a meaningful readiness attribute. Do not invent such an attribute in your script: inspect the actual page or use another observable condition. Selenium’s Python API documents the WebDriverWait constructor and until() behavior.

5. Handle infinite scroll and incrementally loaded lists

For an infinite-scroll page, define a stopping rule before collecting data. A loop that scrolls once and waits for one card proves only that a card appeared. It does not prove that all content has arrived. Possible page-specific signals include an end marker, a known expected count, or no new items after a deliberate scroll-and-check cycle. These are implementation patterns, not universal guarantees. Some pages expose no observable end condition, so a script may not be able to prove completeness.

This example stops when an end marker appears or when the number of cards has not increased for a configured number of consecutive checks. The stable-count rule is a task-defined cutoff, not proof that the site has no more content. Adapt the selectors, scroll target, timeout, and cutoff to the page.

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

URL = "https://example.com/feed"
CARD_SELECTOR = ".feed-card"  # Replace with the target page's selector.
END_SELECTOR = ".feed-end"   # Replace, or set to None if there is no end marker.
TIMEOUT_SECONDS = 10
MAX_STABLE_ROUNDS = 3

 driver = webdriver.Chrome()
try:
    driver.get(URL)
    wait = WebDriverWait(driver, TIMEOUT_SECONDS)
    stable_rounds = 0
    previous_count = 0

    while True:
        cards = driver.find_elements(By.CSS_SELECTOR, CARD_SELECTOR)
        current_count = len(cards)
        if current_count > previous_count:
            previous_count = current_count
            stable_rounds = 0

        if END_SELECTOR and driver.find_elements(By.CSS_SELECTOR, END_SELECTOR):
            break
        if stable_rounds >= MAX_STABLE_ROUNDS:
            break

        if cards:
            driver.execute_script(
                "arguments[0].scrollIntoView({block: 'end'});", cards[-1]
            )
        else:
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        try:
            wait.until(
                lambda d: len(d.find_elements(By.CSS_SELECTOR, CARD_SELECTOR))
                > current_count
                or (END_SELECTOR and d.find_elements(By.CSS_SELECTOR, END_SELECTOR))
            )
        except TimeoutException:
            stable_rounds += 1
        else:
            updated_count = len(driver.find_elements(By.CSS_SELECTOR, CARD_SELECTOR))
            if updated_count == current_count:
                stable_rounds += 1

    cards = driver.find_elements(By.CSS_SELECTOR, CARD_SELECTOR)
    captured_text = [card.text for card in cards]
    print("Collected", len(captured_text), "cards before stopping")
finally:
    driver.quit()

Code correction: remove the leading space before driver = webdriver.Chrome() in the code block if copying it; Python does not allow an unexpected indent at top level.

For a clean copy, the setup line should be:

driver = webdriver.Chrome()

In production, consider a stronger stopping signal than stable counts, such as a confirmed end marker or a known expected count. Scroll the element or container that actually triggers loading; some pages use a nested scroll area rather than the window. Keep a maximum number of rounds or an overall deadline so a broken end condition cannot create an endless loop.

6. Implicit waits, explicit waits, and fixed sleeps

Approach	Scope	Use	Tradeoff
Explicit wait	One named condition at the point it is needed	Preferred for a specific lazy-content state	Requires choosing a meaningful condition.
Implicit wait	Global element-location calls	Applies a general search timeout	Does not express visibility, text, or page completion.
Fixed sleep	One unconditional pause	Rare debugging or a known external timing requirement	Can waste time when the page is fast and still be too short when it is slow.

Selenium warns against combining implicit and explicit waits because their durations can interact unpredictably. Prefer explicit waits for dynamic page states and leave the implicit wait at its default of zero. See the official waiting guidance.

# Avoid combining this global wait with explicit waits:
driver.implicitly_wait(10)

# Prefer a condition at the point it is needed:
wait = WebDriverWait(driver, 15)
card = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".result-card")))

7. Timeout, polling, and reliability choices

The Python WebDriverWait API documents a default polling interval of 0.5 seconds and ignores NoSuchElementException by default while polling. A timeout is not a recommendation: choose one that fits the application, network, browser, and environment. The API also documents that until() raises TimeoutException when its condition never succeeds within the limit.

Keep timeouts bounded and handle them as an explicit failure or expected branch. Log the URL, condition, elapsed time, and relevant page state when diagnosing failures. Avoid catching a timeout and proceeding as if the requested content were complete. If the page replaces an element during rendering, a saved reference can become stale; use locator-based expected conditions that find the current element, or wait for staleness_of(old_element) before locating its replacement.

Shorter polling intervals can check a condition more often, but they also cause more repeated WebDriver commands. The default 0.5-second interval is an API setting, not a performance benchmark. Tune only when the page and remote browser setup justify it.

8. Troubleshooting

Symptom	Likely cause	Fix
TimeoutException waiting for a card	Wrong selector, content never triggered, slow response, or a condition the page does not reach.	Inspect the live DOM, trigger the relevant scroll or interaction, and wait for a real page-specific signal. Set a bounded timeout appropriate to the environment.
Wait succeeds, but only some list items were captured	The condition proved that matching elements existed, not that a growing list was finished.	Use an end marker, known expected count, or documented stopping rule; state the limit if no reliable end signal exists.
Element is present but text is empty or click fails	Presence does not mean visible, interactable, or populated.	Wait for visibility or expected text, and check the page’s own readiness state for the relevant content.
StaleElementReferenceException	Rendering replaced or detached the element after it was located.	Wait for staleness if appropriate, then locate the element again with a locator-based condition.
Wait lasts much longer than the configured timeout	Implicit and explicit waits may be combined, or nested waits add time.	Remove the implicit wait, avoid nested long waits, and keep one clear deadline for the operation.
Infinite-scroll loop never ends	The page has no detected end marker, the wrong scroll container is used, or the chosen condition never changes.	Verify the scroll target and selectors. Add a maximum-round or overall-time limit and report that the stopping rule was reached.
Driver cannot start or browser version mismatch	Browser or driver is missing, incompatible, or unavailable in the runtime.	Check installed browser versions, Selenium Manager access, and container provisioning; use the official Selenium setup documentation.

9. Performance, reliability, and cost

Browser automation carries the overhead of starting and controlling a real browser. Reuse a driver for a related sequence of page actions, close it reliably in a finally block, and avoid waiting for broader conditions than the task requires. For a list, collect only the fields needed and use a clear end rule to avoid unnecessary scrolling.

Reliability depends on stable page signals and a bounded failure path. A selector tied to changing presentation markup can break; a page-owned readiness marker or stable content condition is generally easier to reason about when available. Record when the script times out or reaches its stopping cutoff so partial capture is not mistaken for complete capture.

There is no universal runtime or cost figure for this method. Costs depend on browser infrastructure, execution time, concurrency, and the environment used to run the automation. Selenium itself does not define a per-screenshot API charge in the cited waiting documentation.

10. Or skip the browser setup

For a screenshot rather than browser automation logic, ScreenshotNeo is a website screenshot API and MCP server. One GET request returns an image or PDF; its API documentation describes the available parameters. Adapt the target URL as needed:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

With ScreenshotNeo, cookie banners are accepted and removed along with known consent platforms, newsletter popups, and chat widgets before the shot; those cleanup steps can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers identify the page verdict and billing status. An MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. Sign up for free and get 1,000 screenshots a month with no card.

11. FAQ

Does presence_of_all_elements_located wait until a list is complete?

No. It returns matching elements present when checked. Define a separate completion or stopping rule for content that arrives incrementally.

What timeout should I use?

There is no universal ideal. Choose a bounded timeout based on the page and runtime, then handle timeout as a meaningful outcome.

Can document.readyState == "complete" replace an explicit wait?

No. It describes document readiness, not whether later JavaScript has finished rendering the particular content your task needs.

Can I use a fixed sleep?

You can, but a sleep does not inspect page state. It may be longer than needed or too short. An explicit wait ties progress to an observable condition.

What if the page has no end marker?

Use a task-defined cutoff such as an expected count or a bounded number of scroll checks, and report that cutoff. If the page exposes no reliable completion signal, completeness may not be provable.