Headless Website Testing with Selenium

Headless Selenium runs a real Chrome, Firefox, or Edge browser without opening a visible window. WebDriver drives that browser through the vendor’s automation API, so your test exercises the same application you deploy instead of a mocked HTTP client. Add the browser’s headless option, use explicit waits for observable conditions, assert with your test framework, and always call quit() in teardown.

What headless Selenium is

Headless mode changes how the browser is displayed; it does not turn Selenium into an HTTP checker. The browser still parses HTML, runs JavaScript, applies layout and security rules, stores cookies, and makes network requests. Selenium WebDriver is a W3C Recommendation and uses browser automation APIs supplied by browser vendors.

Headless: suited to CI and servers without a desktop session.
Headed: useful while developing locators or inspecting visual failures.
WebDriver is not a test framework: use pytest, unittest, JUnit, NUnit, Cucumber, Robot Framework, or an equivalent framework for assertions and reports.

Read the Selenium WebDriver overview and W3C WebDriver specification.

Install Selenium and a browser

Install Chrome, Edge, or Firefox on the runner.
Install the language binding: python -m pip install -U selenium pytest.
Use Selenium 4.6 or newer so Selenium Manager can discover the browser and resolve a matching driver when a WebDriver is instantiated.

python -m pip install -U selenium pytest
python -c "from selenium import webdriver; d=webdriver.Chrome(); d.quit(); print('ok')"

See Selenium Manager documentation for driver discovery and management.

Complete Python headless test

from pathlib import Path
import pytest
from selenium import webdriver
from selenium.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

@pytest.fixture
def driver():
    options = Options()
    options.add_argument('--headless=new')
    options.add_argument('--window-size=1440,1000')
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-dev-shm-usage')
    browser = webdriver.Chrome(options=options)
    browser.set_page_load_timeout(45)
    yield browser
    browser.quit()

def test_homepage_has_heading(driver):
    driver.get('https://example.com')
    heading = WebDriverWait(driver, 15).until(
        EC.visibility_of_element_located((By.TAG_NAME, 'h1'))
    )
    assert heading.text == 'Example Domain'

def test_capture_diagnostics_on_failure(driver, request):
    try:
        driver.get('https://example.com')
        WebDriverWait(driver, 15).until(EC.title_contains('Example'))
        assert 'Example' in driver.title
    except Exception:
        out = Path('artifacts')
        out.mkdir(exist_ok=True)
        driver.save_screenshot(str(out / f'{request.node.name}.png'))
        (out / f'{request.node.name}.html').write_text(driver.page_source, encoding='utf-8')
        raise

Replace the URL and assertion with your application. Prefer IDs, names, and stable attributes such as data-test; avoid generated class names and absolute XPath.

Browser-specific headless options

Chrome

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless=new')
options.add_argument('--window-size=1440,900')
driver = webdriver.Chrome(options=options)
try:
    driver.get('https://example.com')
finally:
    driver.quit()

Edge

from selenium import webdriver
from selenium.webdriver.edge.options import Options
options = Options()
options.add_argument('--headless=new')
driver = webdriver.Edge(options=options)
try:
    driver.get('https://example.com')
finally:
    driver.quit()

Firefox

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument('-headless')
driver = webdriver.Firefox(options=options)
try:
    driver.get('https://example.com')
finally:
    driver.quit()

The Selenium project documents headless runs for these browsers. Pin browser images in CI when reproducibility matters.

Wait for conditions, not arbitrary sleeps

Wait for the exact condition required by the next action: presence, visibility, clickability, a URL, a title, or a custom JavaScript condition.

wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '[data-test="results"]')))
wait.until(EC.element_to_be_clickable((By.ID, 'submit'))).click()
wait.until(EC.url_contains('/complete'))

Do not mix implicit and explicit waits. Increasing a timeout without identifying the unmet condition usually hides the real race, such as a delayed API response, redirect, animation, or changed locator.

Interactions and browser state

wait = WebDriverWait(driver, 15)
email = wait.until(EC.visibility_of_element_located((By.NAME, 'email')))
email.clear()
email.send_keys('ci@example.test')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '[data-test="save"]'))).click()
wait.until(EC.text_to_be_present_in_element((By.ID, 'status'), 'Saved'))

driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')

Give each test a new session. A fresh profile prevents cookies, local storage, service workers, and permissions from leaking between tests. Use quit(), not only close(), so the complete session and browser process end.

Run headless Selenium in CI

A Linux runner needs a browser and its shared libraries. Containerized jobs often need a larger shared-memory mount. --disable-dev-shm-usage can move Chrome’s shared-memory use to disk, while --no-sandbox should be used only when the image requires it.

name: browser-tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: python -m pip install -U selenium pytest
      - run: pytest -q
      - if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: browser-artifacts
          path: artifacts/

Keep browser and driver versions aligned, record versions in CI logs, and upload screenshots and HTML on failure. Parallelize only when each test owns its data and browser session.

Node.js example

npm install selenium-webdriver

const { Builder, By, until } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

(async function () {
  const options = new chrome.Options();
  options.addArguments('--headless=new', '--window-size=1440,1000');
  const driver = await new Builder().forBrowser('chrome').setChromeOptions(options).build();
  try {
    await driver.get('https://example.com');
    const heading = await driver.wait(until.elementLocated(By.css('h1')), 15000);
    await driver.wait(until.elementIsVisible(heading), 5000);
    if ((await heading.getText()) !== 'Example Domain') throw new Error('unexpected heading');
  } finally {
    await driver.quit();
  }
})();

RemoteWebDriver and Selenium Grid

Use Grid when you need browsers on other machines, multiple operating systems and browser versions, or parallel sessions. Local headless runs have less setup and network overhead; Grid adds capacity and central scheduling but requires browser images, routing, logs, and data isolation.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless=new')
driver = webdriver.Remote(command_executor='http://grid-host:4444', options=options)
try:
    driver.get('https://example.com')
finally:
    driver.quit()

curl -fsS http://grid-host:4444/status

Grid executes tests across machines. Start with one worker per browser capacity unit, then measure queue time and resource use before increasing concurrency.

Diagnostics with WebDriver BiDi

Selenium’s WebDriver BiDi work provides a bidirectional channel that can stream network requests, console messages, and JavaScript errors. Use supported BiDi features alongside screenshots, HTML, and assertions when a DOM failure does not explain the cause.

Common errors and fixes

Symptom	Cause	Fix
`SessionNotCreatedException`	Browser and driver mismatch	Install a supported browser, upgrade Selenium, or pin matching binaries.
Chrome cannot start in a container	Sandbox, shared memory, permissions, or missing libraries	Use a maintained image, run as non-root, increase `/dev/shm`, and add container flags only when required.
`NoSuchElementException`	Element is late, inside an iframe, or locator changed	Wait for the condition, switch to the frame, and use a stable locator.
`ElementClickInterceptedException`	Overlay, animation, or layout shift	Wait for the overlay to disappear and the element to become clickable.
Page timeout	Slow API, failed request, redirect, or wrong readiness condition	Capture URL, HTML, console, and network evidence; wait on the specific state.
Passes locally but fails in CI	Different browser, viewport, timezone, fonts, data, or CPU	Pin the environment, set inputs explicitly, and save artifacts.
Tests affect one another	Shared session or backend state	Create a session per test, call `quit()`, and isolate accounts and data.

Performance, reliability, and cost

Startup: a browser session is heavier than an HTTP request. Reuse sessions only when a test intentionally shares state.
Waits: condition waits reduce idle time and make failures diagnosable.
Coverage: local headless is simple for one browser; Grid costs machines or a hosted service but provides browser and OS coverage.
Reproducibility: pin browser versions, viewport, locale, timezone, test data, and fonts.
Security: keep credentials and cookies in CI secrets and do not print them.

Or skip the browser setup

For a screenshot or PDF rather than an interactive assertion, ScreenshotNeo provides a single GET request. Before capture it accepts cookie and consent banners and removes 60+ known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the verdict and billing with X-Page-Verdict and X-Billed headers. Its MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients.

See the ScreenshotNeo API docs for all options.

curl -G 'https://api.screenshotneo.com/v1/shot' \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

import requests
r = requests.get('https://api.screenshotneo.com/v1/shot', params={'access_key': 'YOUR_API_KEY', 'url': 'https://stripe.com'}, timeout=90)
r.raise_for_status()
open('shot.webp', 'wb').write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const image = Buffer.from(await res.arrayBuffer());
require('fs').writeFileSync('shot.webp', image);

It supports full-page and element capture, custom waits, CSS and JavaScript, device and viewport controls, headers and cookies, request blocking, caching, signed links, async jobs, bulk capture, PDF options, and usage reporting. There are 1,000 free screenshots each month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

FAQ

Do I still need ChromeDriver?

Usually Selenium Manager handles browser and driver discovery when a WebDriver is created in current Selenium releases. Pin and provide binaries yourself when your CI image requires strict version control.

Does headless render differently?

It uses the same browser engine, but viewport, GPU, fonts, flags, and timing can differ. Set those inputs explicitly and compare a headed run for visual discrepancies.

When should I use Grid?

Use Grid for multiple browser and OS combinations or parallel capacity across machines. Keep a local headless job for fast feedback.

Why does Selenium not report pass or failure?

WebDriver controls the browser; your test framework supplies assertions, setup, retries, and reports.