How to Scroll Down to Load Web Content with RSelenium
Use RSelenium JavaScript, waits, and scroll loops to load lazy content reliably, handle iframes, and capture fully rendered pages.

Direct answer: With an active RSelenium remoteDriver, execute JavaScript in the page. To move directly to the current bottom, use remDr$executeScript("window.scrollTo(0, document.body.scrollHeight);"). For lazy-loaded or infinite-scroll pages, scroll in measured increments, wait for a page-specific condition, and repeat until the content you need exists.
A scroll changes the viewport. It does not guarantee that JavaScript has finished adding new nodes. Selenium documents that navigation readiness can occur while scripts continue changing the page, so a reliable RSelenium workflow combines scrolling with an explicit wait for a result element, a changed result count, or another meaningful application state.
1. Set up RSelenium and open the page
Install RSelenium and start a compatible Selenium server or browser container. The example below assumes that a WebDriver endpoint is available on localhost:4444.
install.packages("RSelenium")
library(RSelenium)
remDr <- remoteDriver(
remoteServerAddr = "localhost",
port = 4444L,
browserName = "chrome"
)
remDr$open()
remDr$navigate("https://example.com/articles")
remoteDriver$executeScript(script, args = list(...)) injects JavaScript into the currently selected browsing context and returns the script result. The selected frame matters: JavaScript runs in the current frame, not automatically in every iframe on the page. See the RSelenium remoteDriver reference and Selenium’s frame documentation.
2. Scroll to the current document bottom
Use window.scrollTo when one movement is enough, such as a conventional page whose content is already present and only needs to be brought into view.
remDr$executeScript(
"window.scrollTo(0, document.body.scrollHeight);"
)
You can verify the resulting position by returning values from JavaScript:
position <- remDr$executeScript(
"return { y: window.scrollY, viewport: window.innerHeight, height: document.body.scrollHeight };"
)
position[[1]]
A single jump uses the page height known at that moment. If reaching the bottom causes another request and increases the document height, the first command is no longer sufficient. Use an incremental loop for that case.
3. Scroll by measured increments
window.scrollBy moves relative to the current position. An 800-pixel increment is a practical starting point, but the right amount depends on viewport size, animation, and how much content the site loads per trigger.

remDr$executeScript("window.scrollBy(0, 800);")
For a basic finite page, repeat the movement until the viewport reaches the bottom:
repeat {
state <- remDr$executeScript(
"return { y: window.scrollY, viewport: window.innerHeight, height: document.documentElement.scrollHeight };"
)[[1]]
if (state$y + state$viewport >= state$height - 5) break
remDr$executeScript("window.scrollBy(0, 800);")
Sys.sleep(0.5)
}
The sleep gives the browser a chance to paint and run page JavaScript. It is only a pacing delay; it is not proof that the desired content has loaded. Prefer an explicit condition whenever the page exposes one.
4. Load infinite-scroll content with a stop condition
Infinite-scroll pages typically append distinct results as the reader approaches the bottom. Google Search Central describes this pattern as loading more pages of content during scrolling and recommends that sites expose equivalent pagination for crawlers. For automation, observe the result list itself rather than assuming that reaching the current bottom means completion.
The following helper counts result nodes and scrolls until the count stops increasing for several checks or a maximum number of rounds is reached. Replace .result-card with a selector from the target site.
count_results <- function() {
remDr$executeScript(
"return document.querySelectorAll(arguments[0]).length;",
list(".result-card")
)[[1]]
}
last_count <- count_results()
stalled_rounds <- 0L
max_rounds <- 40L
for (round in seq_len(max_rounds)) {
remDr$executeScript("window.scrollBy(0, 800);")
deadline <- Sys.time() + 10
repeat {
Sys.sleep(0.25)
current_count <- count_results()
if (current_count > last_count || Sys.time() >= deadline) break
}
if (current_count > last_count) {
stalled_rounds <- 0L
last_count <- current_count
} else {
stalled_rounds <- stalled_rounds + 1L
}
if (stalled_rounds >= 3L) break
}
message("Loaded ", last_count, " results")
This is a deliberate stopping heuristic, not a universal detector. A site may temporarily return zero new results, replace nodes instead of appending them, or require a “Load more” click. If the page has a spinner, end marker, next-page token, or network state exposed in the DOM, use that signal instead.
5. Wait for the application state you need
Selenium’s page-load strategy governs document navigation, while single-page applications can continue fetching data and changing visibility afterward. Treat navigate returning as the start of your page-specific work.
For a known element, poll until it is present and visible:
wait_for_visible <- function(selector, timeout = 15) {
end_time <- Sys.time() + timeout
repeat {
found <- tryCatch(
remDr$findElement(using = "css selector", value = selector),
error = function(e) NULL
)
if (!is.null(found)) {
visible <- tryCatch(found$isElementDisplayed()[[1]], error = function(e) FALSE)
if (isTRUE(visible)) return(found)
}
if (Sys.time() >= end_time) {
stop("Timed out waiting for ", selector)
}
Sys.sleep(0.25)
}
}
card <- wait_for_visible(".result-card")
For an attribute or text state, return the value from JavaScript and wait until it changes:
wait_for_count_increase <- function(selector, old_count, timeout = 10) {
end_time <- Sys.time() + timeout
repeat {
new_count <- remDr$executeScript(
"return document.querySelectorAll(arguments[0]).length;",
list(selector)
)[[1]]
if (new_count > old_count) return(new_count)
if (Sys.time() >= end_time) return(new_count)
Sys.sleep(0.25)
}
}
6. Scroll a known element into view
If the target already exists in the DOM, locate it and scroll it into view. This is preferable to guessing a pixel offset because the browser accounts for the element’s actual position.
target <- remDr$findElement(
using = "css selector",
value = "#pricing"
)
remDr$executeScript(
"arguments[0].scrollIntoView({block: 'center', inline: 'nearest'});",
list(target)
)
Scrolling an element into view cannot create an element that has not been inserted yet. On a lazy page, first scroll the relevant container to trigger loading, wait for the selector, then call scrollIntoView.
7. Handle nested scroll containers
Many feeds do not use the window as their scroll owner. A modal, sidebar, or feed may have overflow: auto and its own scroll height. Inspect the container and scroll that node directly.
container <- remDr$findElement(
using = "css selector",
value = ".feed"
)
remDr$executeScript(
"arguments[0].scrollTop = arguments[0].scrollHeight;",
list(container)
)
For incremental movement:
remDr$executeScript(
"arguments[0].scrollBy(0, arguments[1]);",
list(container, 600)
)
Compare the container’s scrollTop + clientHeight with scrollHeight to detect its bottom. If the top-level window appears stationary, check for this nested region before changing selectors or timing.
8. Work inside an iframe
When the content is inside an iframe, switch into that frame before finding elements or executing page JavaScript. Selenium’s frame model requires the correct browsing context.
frame <- remDr$findElement(
using = "css selector",
value = "iframe.content-frame"
)
remDr$switchToFrame(frame)
remDr$executeScript("window.scrollBy(0, 800);")
# Return to the top-level document when finished.
remDr$switchToFrame(NULL)
An iframe from another origin can still be automated through WebDriver, but JavaScript executed in the parent document cannot directly inspect the iframe’s DOM. Always select the frame first.
9. Use asynchronous JavaScript when the page exposes a callback
RSelenium also provides executeAsyncScript. The injected script receives a callback as its final argument and must call it. This is useful when you want the browser-side code to wait for an event or timer, but it cannot span a page navigation.
loaded <- remDr$executeAsyncScript(
"var done = arguments[arguments.length - 1];
var selector = arguments[0];
var deadline = Date.now() + 10000;
(function check() {
if (document.querySelector(selector)) return done(true);
if (Date.now() > deadline) return done(false);
setTimeout(check, 100);
})();",
list(".result-card")
)
if (!isTRUE(loaded[[1]])) stop("Result did not appear")
10. Common failures and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Nothing new appears | The page paginates, needs a button click, or uses a nested scroll region. | Inspect the DOM and scroll the actual container or automate the next-page control. |
| Results are missing after navigation | JavaScript data loading continued after document readiness. | Wait for the result selector, count increase, spinner disappearance, or another application signal. |
| Element cannot be found | It has not been inserted yet, or the selector is evaluated in the wrong frame. | Trigger the loading step, switch to the iframe, then locate the element again. |
| The window does not move | A modal or feed owns scrolling. | Find the element with overflow scrolling and update its scrollTop. |
| Scroll reaches a visual bottom too early | New content extends the document after the measurement. | Re-read scrollHeight after each increment and use a stalled-count or end-marker rule. |
| Async script times out | The callback is never called or the condition never occurs. | Call the callback on success and timeout, and keep the script within one page lifecycle. |
11. Performance and reliability checklist
- Use the smallest scroll increment that reliably triggers the site’s loader; very large jumps can skip an intersection-observer threshold.
- Poll a meaningful DOM condition instead of adding long fixed sleeps.
- Set a maximum number of rounds and an overall deadline so a broken feed cannot run forever.
- Record result counts, scroll positions, and the final stopping reason for debugging.
- Close the driver with
remDr$close()when the job completes. - Expect animations, rate limits, network failures, and bot defenses to make repeated runs variable; retry bounded operations and preserve partial progress where possible.
- For reproducible captures, set the viewport and browser timezone consistently and wait for fonts, images, and application data that affect layout.
remDr$setWindowSize(1280, 900)
# ... scrolling and extraction ...
remDr$close()
12. Or skip the browser setup
If your goal is a rendered screenshot rather than browser automation itself, ScreenshotNeo provides a single HTTP request for a PNG, JPEG, WebP, or PDF. Its capture process accepts cookie and consent banners, removes more than 60 known consent platforms plus newsletter popups and chat widgets, and lets you turn each step off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed; the response reports the result through X-Page-Verdict and X-Billed headers.

Read the complete parameter reference in the ScreenshotNeo documentation.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" \
-d access_key=YOUR_API_KEY \
--data-urlencode url=https://stripe.com \
-o shot.webp
Python
import requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)
Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.arrayBuffer();
await Bun.write('shot.webp', data);
ScreenshotNeo also supports full-page capture with lazy images loaded, CSS-selector element capture, dark mode, device presets, arbitrary viewports, retina scale, custom CSS and JavaScript, clicks, selector waits, delays, network-idle waits, blocked requests and resource types, headers, cookies, user agents, Authorization, timezone, geolocation, transparent backgrounds, resizing, chosen-TTL caching, signed links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, a usage API, and an OpenAPI specification. Its MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients.
The Free plan includes 1,000 screenshots each month with no card. Paid plans start at $5 for 3,000 screenshots; yearly billing gives two months free, and every feature is available on every plan. Create a free ScreenshotNeo account.
13. FAQ
Does scrollTo trigger lazy loading?
It can trigger loaders that watch viewport position, but it is not guaranteed. Incremental scrolling and a page-specific wait are safer for infinite feeds.
Should I use Selenium wheel actions instead?
Wheel actions provide a user-like scroll-by-amount or element-origin action. Selenium notes that they do not automatically bring the target element into view, and the documented action API is Chromium-only; verify support in your browser and binding before depending on it.
How do I know when an infinite scroll is finished?
Use the site’s end marker, a disabled loader, a next-page state, or a stable result count across repeated waits. A fixed number of scrolls is only a fallback.
Why does a screenshot miss content that RSelenium can see?
A screenshot taken before the application settles may capture an earlier state. Wait for the same selector or result count you use in RSelenium, or use ScreenshotNeo’s selector and network-idle wait options.
Can I scroll a page without loading every result?
Yes. Stop as soon as the target element or count is available. This reduces requests and runtime compared with scrolling to the end.


