How to Bulk Screenshot Web Pages with Selenium Grid

Short answer: Selenium Grid provides remote browser capacity and routes WebDriver sessions across machines. It does not accept a list of URLs and create an image archive by itself. Your client application must define the URL list, start a bounded number of remote browser sessions, assign pages to them, wait for the page state you need, capture screenshot bytes, save each image and its metadata, handle failures, and close every session.

This guide builds that workflow with Python and Selenium. It also covers Grid setup, concurrency, screenshot behavior, reliability, security, and common errors. The same design works with other WebDriver language bindings.

1. How bulk screenshots with Grid work

Grid routes WebDriver commands to remote browser instances and supports parallel execution across machines, browser versions, and platforms. The client owns the batch: URL enumeration, task scheduling, retries, artifact names, persistence, and per-URL status records are application responsibilities inferred from the client/server WebDriver model. Selenium Grid documentation · Remote WebDriver documentation

Start a reachable Selenium Grid and decide which browser and viewport configuration to use.
Put URLs or page variants in a worklist with stable IDs.
Run a limited number of workers. Each worker creates a RemoteWebDriver session against the Grid endpoint.
For each assigned URL, navigate, wait for the readiness condition your project needs, and take a screenshot.
Save the bytes locally or to your chosen storage with a deterministic filename and a record of the result.
Close the session in a finally block, including when navigation or saving fails.

One task can represent one URL or a page variant such as a different viewport, browser, or locale. Use separate tasks when the resulting images must be independently identified.

2. Start Selenium Grid

The official getting-started path requires Java 11 or higher, a browser, browser drivers (or Selenium Manager), and the Selenium Server JAR. A Standalone Grid is suitable for a single-machine setup and uses port 4444 in the basic example. Hub-and-Node and Distributed arrangements support deployments with multiple machines or changing capacity. Choose based on required browsers and operating systems, desired parallel sessions, and available machines. Getting started with Selenium Grid

Start the server using the Selenium Server JAR you downloaded:

java -jar selenium-server-<version>.jar standalone

For a multi-machine arrangement, configure the documented Hub-and-Node or Distributed roles for your environment and point clients at the Grid’s client-facing endpoint. Keep the endpoint within its intended trust boundary. Selenium warns that exposed Grid access can reveal internal applications or files and allow third parties to run binaries; use private networking and firewall and access controls appropriate to the deployment.

3. Runnable Python batch example

Install Selenium, make sure the Grid endpoint is reachable from the machine running this script, then save the following as bulk_screenshots.py. Set GRID_URL if your endpoint differs from the local Standalone address. The script keeps a fixed worker limit, names screenshots using stable URL IDs, writes a JSONL outcome record for every URL, and always quits each remote session.

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from urllib.parse import urlparse
import json
import re
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait

GRID_URL = "http://localhost:4444"
OUTPUT_DIR = Path("screenshots")
MAX_WORKERS = 3
PAGE_TIMEOUT_SECONDS = 60
READY_TIMEOUT_SECONDS = 20

# Give each page a stable ID. Avoid deriving filenames from raw URLs alone:
# URLs can contain unsafe characters or collide after sanitization.
URLS = [
    {"id": "home", "url": "https://example.com/"},
    {"id": "docs", "url": "https://www.selenium.dev/documentation/"},
]

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)


def safe_id(value: str) -> str:
    result = re.sub(r"[^A-Za-z0-9._-]+", "_", value).strip("._-")
    return result or "page"


def capture(item: dict) -> dict:
    driver = None
    started = time.time()
    image_path = OUTPUT_DIR / f"{safe_id(item['id'])}__chrome-1440x1000.png"
    record = {
        "id": item["id"],
        "url": item["url"],
        "started_at_unix": started,
        "browser": "chrome",
        "viewport": {"width": 1440, "height": 1000},
        "screenshot": str(image_path),
    }

    try:
        options = Options()
        options.add_argument("--headless=new")
        options.add_argument("--window-size=1440,1000")
        driver = webdriver.Remote(command_executor=GRID_URL, options=options)
        driver.set_page_load_timeout(PAGE_TIMEOUT_SECONDS)
        driver.set_window_size(1440, 1000)
        driver.get(item["url"])

        # This generic condition is only an example. Replace it with an
        # application-specific condition when the page has a known ready marker.
        WebDriverWait(driver, READY_TIMEOUT_SECONDS).until(
            lambda d: d.execute_script("return document.readyState") == "complete"
        )

        if not driver.save_screenshot(str(image_path)):
            raise RuntimeError("WebDriver reported that the screenshot was not saved")

        record["status"] = "ok"
        record["title"] = driver.title
    except Exception as exc:
        record["status"] = "error"
        record["error"] = f"{type(exc).__name__}: {exc}"
        # Remove a possibly partial artifact so downstream consumers do not
        # mistake it for a successful capture.
        image_path.unlink(missing_ok=True)
    finally:
        if driver is not None:
            try:
                driver.quit()
            except Exception as exc:
                record["quit_error"] = f"{type(exc).__name__}: {exc}"

    record["finished_at_unix"] = time.time()
    return record


def main() -> None:
    results = []
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
        futures = [pool.submit(capture, item) for item in URLS]
        for future in as_completed(futures):
            results.append(future.result())

    # Write in deterministic ID order, regardless of task completion order.
    results.sort(key=lambda row: row["id"])
    with (OUTPUT_DIR / "manifest.jsonl").open("w", encoding="utf-8") as output:
        for result in results:
            output.write(json.dumps(result, ensure_ascii=False) + "\n")

    failures = [row for row in results if row["status"] != "ok"]
    print(f"Captured {len(results) - len(failures)}/{len(results)} pages; manifest: {OUTPUT_DIR / 'manifest.jsonl'}")
    if failures:
        raise SystemExit(1)


if __name__ == "__main__":
    main()

Install the dependency with python -m pip install selenium. The script uses one session per URL and a thread pool to cap concurrent sessions. For large worklists, stream tasks from a queue or database rather than keeping the entire list in memory. Ensure IDs are unique across the full batch and across any browser or viewport variants; otherwise output files can overwrite one another.

Waiting for the right page state

document.readyState == 'complete' does not mean that a single-page application has finished fetching data, images have decoded, fonts have loaded, or animations have stopped. It is only a generic baseline. If your site exposes a stable marker, wait for it:

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 20).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, "main article"))
)

For a fixed delay, use time.sleep(seconds) sparingly; it may waste time on fast pages and still be too short on slow ones. If visual comparability matters, define the readiness condition and viewport consistently for all captures. Selenium’s interaction documentation notes that screen resolution can affect rendering. Working with windows and tabs

4. Screenshot output, viewport, and full-page behavior

WebDriver screenshot behavior is not a guarantee of full-page output. Selenium’s JavaScript API describes takeScreenshot() as making a best effort, in order, to return the entire page, the current window, the visible part of the current frame, or the browser display. The actual result depends on browser and driver support; confirm the behavior for the Grid node and browser you use. Selenium JavaScript WebDriver API

The Python example saves a PNG of the current browser screenshot. Use set_window_size(width, height) and matching browser options to set a stable viewport. A tall viewport is not the same as a supported full-page capture. If the requirement is a full-page image, verify its output dimensions and content for your exact browser/driver combination before relying on it.

For repeatable comparisons, record browser name and version, viewport dimensions, relevant page variant, and capture time in the manifest. A page’s responsive layout, dynamic content, fonts, and animation can still change the resulting image.

5. cURL and Node.js examples

WebDriver’s protocol is a session-based HTTP interface, but raw cURL requires you to manage session creation, command paths, response values, and session deletion yourself. For a batch workflow, the Selenium language bindings handle that protocol and are usually the simpler client. These minimal examples show the shape of one remote session; repeat the capture step for assigned work and always delete the session.

cURL: one remote Chrome session

With a Grid endpoint on localhost, create a session, navigate, request a screenshot, decode the returned Base64 value, and delete the session. This shell example uses jq and base64:

set -eu
GRID="http://localhost:4444"
SESSION_JSON=$(curl -fsS -X POST "$GRID/session" \
  -H 'Content-Type: application/json' \
  -d '{"capabilities":{"alwaysMatch":{"browserName":"chrome","goog:chromeOptions":{"args":["--headless=new","--window-size=1440,1000"]}}}}')
SESSION_ID=$(printf '%s' "$SESSION_JSON" | jq -r '.value.sessionId // .sessionId')

cleanup() {
  curl -sS -X DELETE "$GRID/session/$SESSION_ID" >/dev/null || true
}
trap cleanup EXIT

curl -fsS -X POST "$GRID/session/$SESSION_ID/url" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/"}' >/dev/null

SHOT_JSON=$(curl -fsS "$GRID/session/$SESSION_ID/screenshot")
printf '%s' "$SHOT_JSON" | jq -r '.value' | base64 --decode > example.png

Run that sequence once per page or build a session pool in a real client. The exact capability acceptance and full-page behavior depend on your Grid and browser configuration. Do not expose an unprotected Grid endpoint to untrusted networks.

Node.js: one remote Chrome session

Install the Selenium WebDriver package with npm install selenium-webdriver. This example uses the JavaScript bindings to create a remote session, navigate, save a screenshot, and quit in a finally block:

const { Builder, Browser } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');
const fs = require('node:fs/promises');

async function capture(url, outputPath) {
  const options = new chrome.Options()
    .addArguments('--headless=new', '--window-size=1440,1000');
  const driver = await new Builder()
    .usingServer('http://localhost:4444')
    .forBrowser(Browser.CHROME)
    .setChromeOptions(options)
    .build();

  try {
    await driver.manage().setTimeouts({ pageLoad: 60000 });
    await driver.manage().window().setRect({ width: 1440, height: 1000 });
    await driver.get(url);
    const pngBase64 = await driver.takeScreenshot();
    await fs.writeFile(outputPath, Buffer.from(pngBase64, 'base64'));
  } finally {
    await driver.quit();
  }
}

capture('https://example.com/', 'example.png').catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

To process a batch, place URL records on a queue and let a fixed number of workers call capture. Do not start one session per URL without a limit: that can overwhelm Grid nodes and the target site.

6. Set concurrency and capacity deliberately

Concurrency is a capacity decision, not a number that Grid can choose correctly for every environment. Selenium’s getting-started guide gives a rough reference of around one CPU and one GB of RAM per browser session, while warning these are not universal rules. It also notes that default Node concurrency is CPU-limited and Safari is limited to one session. Measure your nodes under the pages and browser versions you actually capture. Selenium Grid capacity guidance

Selenium’s Grid applicability page illustrates planning with the formula “number of tests × average test time ÷ number of nodes = total execution time.” Its arithmetic examples are explanatory, not throughput guarantees. Real runs also include browser startup, queueing, page load variability, target-site limits, screenshot transfer, and storage. When to Use Grid

Start small: choose a worker limit your node capacity can handle, then increase it while observing queue time, CPU, memory, failures, and output throughput.
Respect the target: parallel requests can affect the sites being captured. Set an appropriate request rate and honor access restrictions.
Separate bottlenecks: browser capacity, Grid queue, target response time, and artifact storage can each limit throughput.
Keep work bounded: use a queue and backpressure for very large batches rather than creating an unbounded number of futures or sessions.
Retry selectively: retry transient session or navigation failures with a small limit and delay. Do not endlessly retry deterministic errors such as invalid URLs or unsupported capabilities.

7. Reliability, filenames, and remote files

Use one outcome record per requested capture, including failures. A useful record includes stable page ID, requested URL, browser and version, viewport, start and finish times, image path, status, and error text. Write artifacts atomically where practical: save to a temporary path and rename after a successful write. This prevents downstream jobs from treating partial files as finished captures.

Choose a deterministic filename from a stable page ID plus the browser and viewport variant. Do not use only a sanitized URL as a key: distinct URLs can collapse to the same filename, and URL paths may contain characters unsuitable for a filesystem. Include a unique ID or collision-resistant digest when the source data does not provide one.

Screenshot bytes returned by the WebDriver client are transferred to the client process, so the script can write them to client-side storage. Browser downloads are a separate concern: Selenium documents managed downloads, which must be enabled on Grid and opted into by the client; the listed files are an immediate snapshot, and Selenium does not wait for downloads to finish. Remote WebDriver and managed downloads

8. Troubleshooting

Symptom	Likely cause	Fix
Connection refused or client cannot reach Grid	Grid is stopped, the endpoint or port is wrong, or the client cannot route to it.	Confirm the Grid process and client-facing URL, check network routing and firewall rules, and test the endpoint from the client machine.
New session fails or stays queued	No matching browser node is available, capabilities do not match, or capacity is occupied.	Check registered nodes and requested capabilities, reduce worker count, or add suitable node capacity.
Session creation reports browser/driver problems	Browser availability, version compatibility, or driver setup differs on the node.	Install or configure the required browser and driver on nodes, or use Selenium Manager as appropriate for the setup.
Screenshot is blank or shows a loading state	The page has not reached the state your capture requires, or navigation failed.	Wait for an application-specific element or data-ready condition, inspect the page and navigation result, and record per-URL failures.
Screenshot is clipped instead of full-page	The chosen browser/driver path only captures the current window or visible area.	Verify screenshot semantics for that configuration; do not assume the WebDriver call means full-page capture.
Images differ between runs	Viewport, browser version, responsive layout, dynamic content, fonts, or timing changed.	Pin the intended browser configuration, set a consistent viewport, wait for a stable page marker, and store capture metadata.
Output file is missing or corrupt	Capture failed, a write was interrupted, or concurrent tasks used the same filename.	Use unique deterministic IDs, validate the save result, write to a temporary file then rename, and retain an error record.
Batch becomes slower as workers increase	Grid nodes, targets, transfer, or storage have saturated; more sessions are waiting or contending.	Measure queue time and resource use, lower concurrency, and identify the limiting system before adding workers.
Grid is reachable by unintended clients	The endpoint is exposed outside its intended trust boundary.	Restrict network access and apply firewall and access controls appropriate to the deployment; do not run an exposed Grid as a public service.

9. Cost and approach tradeoffs

Self-managed Grid gives control over browser versions, operating systems, network access, and where artifacts run. Its costs include node compute and memory, browser startup and maintenance, storage and transfer, and engineering time for orchestration and recovery. A hosted Grid can reduce infrastructure operations, but compare its browser coverage, network access model, concurrency limits, artifact handling, and terms before choosing a provider. The research sources establish Grid deployment patterns and capacity considerations, not a specific provider’s pricing or features.

For a one-off or modest batch, a local Standalone Grid may be enough. For repeated batches that need multiple nodes or browser configurations, a managed Hub-and-Node or Distributed setup can organize capacity, while your client still controls the list and capture workflow. Estimate cost from actual run duration and resource use in your environment; Selenium’s rough sizing guidance is a starting reference, not a billing estimate.

10. Or skip the browser setup

If you need clean website screenshots without managing remote browser sessions and batch infrastructure, ScreenshotNeo is a website screenshot API and MCP server. Your application can still loop over a URL list, while each URL is captured with one request. The ScreenshotNeo API documentation covers request options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie banners are accepted and removed before the shot, along with supported newsletter popups and chat widgets; each cleanup step can be turned off.
Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed. Response headers identify the page verdict and billing status.
An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents and MCP clients.
1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000. Every feature is on every plan.

Create a free ScreenshotNeo account to start with 1,000 screenshots a month and no card.

11. FAQ

Does Selenium Grid accept a CSV of URLs?

No. Grid routes WebDriver sessions to browser instances. Your client must read the CSV or other worklist and schedule captures.

Can one remote session capture several pages?

Yes. A client can navigate one session through several URLs, but each navigation changes the page state. Separate sessions can isolate tasks; reuse can reduce session startup work. Choose based on isolation needs and measured capacity, and always quit sessions.

Can I take screenshots from more than one browser?

Yes, if matching browser nodes are registered and your client requests their capabilities. Record the browser and version because rendering can differ.

Does saving a remote screenshot require downloading a file from the node?

No. WebDriver returns screenshot data through the client API. Browser-managed downloads are a separate feature with separate Grid and client configuration.

How should I choose a worker count?

Begin below the available session capacity, then measure queueing, node resources, page completion, and artifact writing with representative pages. Increase only while the end-to-end batch benefits.