Capture Dynamic Content Inside a Scroll Container with Selenium in Java

To capture dynamically loaded items inside a nested panel, identify the element that actually scrolls, find items within that element, and scroll from it with Selenium 4.2 or later wheel Actions. After each scroll, wait for a page-specific change, collect the visible data, and stop at an end marker or after a bounded number of passes with no progress. This avoids scrolling the document accidentally and handles JavaScript content that arrives after navigation.

1. Identify the scroll container and the content

Inspect the page in browser developer tools. Find the element with a constrained height and scrollable overflow, commonly a div with overflow-y: auto or scroll. Confirm that this element’s scroll position changes when you use the mouse wheel over the panel. The structure and selectors vary by site.

Choose a stable locator for the container and a locator for its items. Scope item lookup to the container with container.findElements(...); a document-wide locator may pick up unrelated elements. Prefer stable identifiers, attributes, or semantic selectors over generated class names.

WebElement container = driver.findElement(By.cssSelector(".scrollable-list"));
By itemLocator = By.cssSelector(".list-item");

2. Scroll the container with Selenium Java

Selenium’s wheel Actions API supports an element-based scroll origin. Wheel input was introduced in Selenium 4.2, so use a compatible Selenium version and check the Java API for your installed release if an overload differs. The example below collects item text after each pass, waits for item-count growth, and bounds both the total work and consecutive no-progress passes.

import java.time.Duration;
import java.util.LinkedHashSet;
import java.util.Set;

import org.openqa.selenium.By;
import org.openqa.selenium.TimeoutException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.interactions.Actions;
import org.openqa.selenium.interactions.WheelInput;
import org.openqa.selenium.support.ui.WebDriverWait;

public class CaptureScrollableItems {
    public static Set<String> capture(WebDriver driver) {
        WebElement container = driver.findElement(
            By.cssSelector(".scrollable-list")
        );
        By itemLocator = By.cssSelector(".list-item");
        WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
        Set<String> captured = new LinkedHashSet<>();
        int stalled = 0;

        for (int pass = 0; pass < 100 && stalled < 3; pass++) {
            // Re-query on every pass; renders may replace existing elements.
            var items = container.findElements(itemLocator);
            int before = items.size();
            for (WebElement item : items) {
                captured.add(item.getText().trim());
            }

            WheelInput.ScrollOrigin origin = WheelInput.ScrollOrigin
                .fromElement(container);
            new Actions(driver)
                .scrollFromOrigin(origin, 0, 500)
                .perform();

            try {
                wait.until(d -> container.findElements(itemLocator).size() > before);
                stalled = 0;
            } catch (TimeoutException noGrowth) {
                stalled++;
            }
        }
        return captured;
    }
}

This is a template: replace both selectors, the scroll amount, and the wait condition to match the page. Put normal driver setup and navigation in your test or application, then call capture(driver). Selenium’s wheel actions guide describes scrolling from an element, including offset origins. The Actions API overview notes wheel input is available from Selenium 4.2.

3. Wait for the right condition

A successful navigation with readyState=complete does not mean a JavaScript-driven list has finished loading. Use an explicit wait for the change that matters to your capture. Selenium’s waiting strategies guide explains explicit waits and warns that mixing implicit and explicit waits can produce unpredictable timing.

Appending list: wait for item count to increase.
Replacing rows or virtualized list: wait for the last item’s stable key or text to change; the count can stay constant.
Loading indicator: wait for it to disappear, then re-query items.
Known end: wait for an end marker or for a “Load more” control to become absent or disabled.

Do not keep old WebElement references across renders. A replaced node can make a reference stale; locate the elements again after updates. Selenium documents that element references are not automatically relocated in its common errors guide.

4. Handle virtualized lists and other page patterns

Virtualized rows

Virtualized interfaces render only a window of rows and recycle or replace them as you scroll. Extract each pass’s values immediately into a set or other durable collection. Waiting for count growth will time out in this pattern, so compare a stable identifier from the last visible row before and after scrolling, or observe another application signal.

Scroll origin offsets

If the panel is small or has nested controls, an origin at the element center may not trigger its scroll handler as expected. Use WheelInput.ScrollOrigin.fromElement(container, xOffset, yOffset) with an offset inside the visible scrollable area. Check the Java API for the Selenium version in the project.

Load-more buttons and sentinels

Some pages load on a button click or when an intersection-observed sentinel enters view rather than on any scroll event. Follow the page’s actual behavior: click the control, or scroll until the sentinel is visible, then wait for the content condition. Do not assume every infinite list uses the same trigger.

5. Alternative: set scrollTop with JavaScript

For a page that reacts to scroll position but does not respond reliably to wheel input, setting the element’s scrollTop can be a useful diagnostic or implementation choice. This changes the scroll position directly; it does not reproduce all user wheel events. Compare behavior in the browser and driver combination you use.

JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript(
    "arguments[0].scrollTop = arguments[0].scrollTop + arguments[1];",
    container,
    500
);

Continue to use an explicit wait after setting the position. Selenium’s element interactions guide describes automatic scrolling for element interactions; that behavior is distinct from controlling a nested panel’s scroll position.

6. cURL, Python, and Node.js options

The browser automation method above is Selenium Java. For HTTP-oriented workflows, cURL, Python, or Node.js can call ScreenshotNeo’s screenshot API directly; these calls capture a page image rather than extracting a list of DOM items. See the ScreenshotNeo API documentation for request options.

cURL

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

Python

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)

Node.js

const q = new URLSearchParams({
  access_key: 'YOUR_API_KEY',
  url: 'https://stripe.com'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
const bytes = Buffer.from(await res.arrayBuffer());
await import('node:fs/promises').then(fs => fs.writeFile('shot.webp', bytes));

Or skip the browser setup

ScreenshotNeo is a website screenshot API and MCP server from ScreenshotNeo. One GET request returns a PNG, JPEG, WebP, or PDF. For a screenshot of a page, use:

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

Cookie and consent banners, newsletter popups, and chat widgets are removed before the shot, and each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits cost nothing; response headers report the page verdict and whether the shot was billed. Its MCP server lets Claude, Cursor, and other MCP clients use take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. These are page screenshots, not a replacement for Selenium when you need to extract DOM rows. Read the API documentation, then sign up for 1,000 free screenshots a month with no card.

Troubleshooting

Symptom	Likely cause	Fix
The whole page scrolls, not the panel	The wheel origin is outside the actual scrollable area, or the selected element is a wrapper.	Inspect which element’s scroll position changes. Use that element as the origin; try an offset inside its visible area.
No new items appear	Wrong container or item selector, insufficient scroll, or the page uses a different loading trigger.	Verify selectors in developer tools, increase the delta if appropriate, and inspect for a load-more button or sentinel.
The wait times out although rows changed	A virtualized list keeps a constant row count.	Wait for last-row identity, a loading state, or another observable change instead of count growth.
`StaleElementReferenceException`	A render replaced a previously located node.	Re-find the container and items after the update; keep extracted strings or identifiers rather than element references.
The loop never ends	There is no real end condition or the page keeps reporting small changes.	Use an end marker when available and always enforce a maximum pass count and no-progress limit.
Timing is flaky	A fixed sleep or navigation completion is being treated as proof that asynchronous content is ready.	Wait explicitly for the content state needed by the next step. Avoid mixing implicit and explicit waits.
Wheel API does not compile	The project uses a Selenium version before wheel input support or a different Java overload.	Use Selenium 4.2 or later and consult the API for the installed version.

Performance, reliability, and cost

Keep the scroll delta large enough to make progress but small enough not to skip trigger regions or lose virtualized rows before extraction. Capturing strings or stable attributes on each pass is usually cheaper and more reliable than retaining many live element handles. Choose explicit wait timeouts from the page’s expected response time, and cap passes so a feed that never signals completion cannot run indefinitely.

Browser-based Selenium work consumes the browser and driver resources for the duration of navigation, rendering, scrolling, and waits. Reuse a driver where the surrounding test or job allows, and keep selectors and end conditions specific to the page. ScreenshotNeo bills only clean shots; bot checks, blank pages, timeouts, failed loads, and cache hits are not billed. Its plans are Free: 1,000 monthly; Starter: $5 for 3,000; Growth: $15 for 15,000; Pro: $39 for 60,000; Scale: $99 for 250,000; Business: $249 for 1,000,000. Yearly billing gives two months free, and every feature is available on every plan.

FAQ

Can Selenium scroll inside a div?

Yes. Use wheel Actions with a scroll origin anchored to the element that actually scrolls, then verify that the panel responds in your target browser.

Why does scrolling stop loading content?

The site may load only near a sentinel, require a button click, or virtualize rows. Inspect the page behavior and wait for its actual signal.

Does ScreenshotNeo return the text from each row?

No. ScreenshotNeo returns screenshots or PDFs. Use Selenium when you need DOM text or structured row data.

Can I use this with older Selenium?

Element-based wheel scrolling requires the wheel Actions support available from Selenium 4.2. For older setups, check whether upgrading is possible or use a page-appropriate alternative such as setting the container’s scroll position with JavaScript.