Python Automation Scripts for Browser Tasks

To automate browser tasks with Python, install Playwright and its browser binaries, open a browser page, locate elements, perform the needed actions, and wait for the specific result your task depends on. For a straightforward script, Playwright’s synchronous API is easy to follow; use its asynchronous API when integrating with an asyncio application.

This guide covers browser setup, navigation, forms, waits, downloads, screenshots, troubleshooting, and practical reliability and cost considerations. The examples use public demo pages; replace them with pages and accounts you are authorized to use.

1. What browser automation does

Browser automation controls a real browser engine to perform actions such as opening a page, filling a form, clicking a button, and checking the resulting page. That differs from downloading HTML and parsing it: browser-driven pages can execute JavaScript, respond to user interactions, and load content dynamically.

Playwright is a general-purpose browser automation library with Python sync and async APIs. Its documentation covers Chromium, Firefox, and WebKit. It can also use branded Chrome and Edge channels when available, subject to browser installation and, in managed environments, enterprise policy constraints. Selenium also has a Python client for browser interaction automation; choose based on your existing project, browser requirements, and environment rather than assuming a universal winner. Playwright installation and introduction · Selenium Python API.

2. Install Playwright and its browsers

Installing the Python package and installing browser binaries are separate steps. Playwright’s CLI installs browser versions compatible with the installed library.

python -m venv .venv
# macOS or Linux:
source .venv/bin/activate
# Windows PowerShell:
# .venv\Scripts\Activate.ps1

python -m pip install playwright
python -m playwright install

The browser installation command can install the supported browsers. To install only Chromium, use python -m playwright install chromium. If you upgrade Playwright or see an error that a compatible executable is missing, rerun the install command for the browser you use. See the official Python library setup and browser guide for platform-specific details and browser channels.

3. A complete synchronous script

This runnable example opens a page, fills and submits a form, waits for the result heading, prints it, saves a screenshot, and closes the browser even if an error occurs. The demo form is hosted for practice by Playwright.

from pathlib import Path
from playwright.sync_api import sync_playwright


def main():
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        page = browser.new_page()
        try:
            page.goto("https://playwright.dev/python/docs/library", wait_until="domcontentloaded")
            print("Title:", page.title())

            # A dedicated practice page can be substituted for this documentation URL.
            page.goto("https://www.saucedemo.com/", wait_until="domcontentloaded")
            page.get_by_placeholder("Username").fill("standard_user")
            page.get_by_placeholder("Password").fill("secret_sauce")
            page.get_by_role("button", name="Login").click()
            page.get_by_text("Products", exact=True).wait_for(state="visible")

            Path("artifacts").mkdir(exist_ok=True)
            page.screenshot(path="artifacts/products.png", full_page=True)
            print("Result:", page.get_by_text("Products", exact=True).inner_text())
        finally:
            browser.close()


if __name__ == "__main__":
    main()

The demo account and page are examples, not credentials for a private service. For your own site, use a test environment and credentials you control. The core API pattern is also shown in the official Pages guide.

4. Navigate, locate, and interact

Navigation

page.goto(url) navigates to a URL. By default it waits for the page’s load event. You can select a different milestone such as domcontentloaded or commit when appropriate, but those milestones do not prove that a dynamic application is ready for the next action. Prefer a wait for the content or state your task actually needs. Playwright navigation guide.

Locators

Use locators that express how a person identifies the control: role and accessible name, label, placeholder, or visible text. For example:

page.get_by_role("button", name="Continue").click()
page.get_by_label("Email address").fill("person@example.com")
page.get_by_placeholder("Search").fill("invoice")
page.get_by_text("Settings", exact=True).click()

When a page has repeated matching elements, scope a locator to a meaningful container or use a CSS selector that identifies the intended element. Avoid brittle selectors tied to incidental layout or generated class names where a semantic locator is available.

Common interactions

# Select an option
page.get_by_label("Country").select_option(label="Canada")

# Check a checkbox
page.get_by_label("I agree").check()

# Hover and keyboard input
page.get_by_role("menuitem", name="Account").hover()
page.get_by_label("Search").press("Enter")

# Read text or an attribute
status = page.get_by_role("status").inner_text()
link = page.get_by_role("link", name="Download").get_attribute("href")

Most locator actions wait for the element to become actionable. If multiple elements match, make the locator more specific rather than clicking whichever happens to match first.

5. Wait for the task’s actual result

A page can continue fetching data and populating the interface after its load event. A fixed sleep is usually a poor readiness check: it can waste time on fast runs and still fail on slow ones. Wait for a result element, URL, or other task-specific condition.

# Wait until a result appears
page.get_by_role("heading", name="Order complete").wait_for(state="visible")

# Or wait for a URL transition
page.wait_for_url("**/account/**")

# For a page that updates a known element
page.locator("[data-state='saved']").wait_for(state="attached")

Locator waits and navigation waits have timeouts. Set a reasonable default for the site and override it for an unusually slow operation:

page.set_default_timeout(10_000)
page.set_default_navigation_timeout(30_000)
page.get_by_role("button", name="Generate report").click(timeout=20_000)

Playwright’s navigation documentation explains why load alone is not a signal that all dynamic content is ready. Tie each wait to the state needed by the next step.

6. Use the async API in asyncio applications

For an asyncio application, use Playwright’s async API throughout the browser workflow and await each operation. Do not call blocking sync Playwright methods from an event loop.

import asyncio
from playwright.async_api import async_playwright


async def main():
    async with async_playwright() as playwright:
        browser = await playwright.chromium.launch(headless=True)
        page = await browser.new_page()
        try:
            await page.goto("https://playwright.dev/python/docs/library", wait_until="domcontentloaded")
            await page.get_by_role("heading", name="Library").wait_for(state="visible")
            print(await page.title())
            await page.screenshot(path="library.png", full_page=True)
        finally:
            await browser.close()


if __name__ == "__main__":
    asyncio.run(main())

Keep the browser lifecycle inside the async context, and await page operations. If your application already owns an event loop, call await main() from its async entry point instead of starting another loop with asyncio.run().

7. Handle downloads and popups

Register for a download before clicking the control that starts it, then save the resulting file. This avoids racing the click against the download event.

from pathlib import Path

Path("downloads").mkdir(exist_ok=True)
with page.expect_download() as download_info:
    page.get_by_role("link", name="Download CSV").click()
download = download_info.value
download.save_as(Path("downloads") / download.suggested_filename)

For a link or button that opens a new tab, wait for the popup around the action:

with page.expect_popup() as popup_info:
    page.get_by_role("link", name="Open report").click()
report_page = popup_info.value
report_page.wait_for_load_state("domcontentloaded")

For async code, use the corresponding async context managers and await the click. If an action sometimes opens a popup and sometimes navigates the existing tab, handle both outcomes explicitly for the site’s documented behavior.

8. Browser, context, and capture options

Choose the browser engine and context settings to match the task. Playwright documents Chromium, Firefox, WebKit, and branded Chrome or Edge channels when available. A browser context isolates cookies and storage; reuse one context when a workflow needs a session, and create a fresh one for independent sessions.

browser = playwright.chromium.launch(headless=True)
context = browser.new_context(
    viewport={"width": 1440, "height": 900},
    device_scale_factor=1,
    locale="en-US",
    timezone_id="UTC",
)
page = context.new_page()

# Keep login state only when appropriate for this trusted workflow.
context.storage_state(path="storage-state.json")

Storage state can contain authentication material. Protect it like a credential, keep it out of source control, and use a restricted test account. Other useful options include viewport size, locale, timezone, color scheme, and device scale factor. Browser and launch options vary by engine; check the browser documentation for supported settings and branded-browser constraints.

For debugging, run headed mode with headless=False, or enable a Playwright trace in a controlled environment. Do not expose traces or screenshots if they contain private page content, tokens, or personal data.

9. Reliability, performance, and cost

• Reliability: Use semantic locators and condition-based waits. Make scripts safe to retry where possible, and distinguish a genuine task failure from a selector or timing problem. Always close browser resources using a context manager or finally.
• Performance: Reuse a browser process for a batch of independent tasks when isolation requirements allow, while using separate contexts for separate sessions. Avoid unnecessary fixed sleeps and full-page screenshots when a smaller result is enough. Parallel work consumes more CPU and memory; keep concurrency within the capacity of your machine or CI runner.
• Cost: Playwright is a software library; a local run has no per-screenshot API charge, though compute, CI minutes, browser downloads, and maintenance have costs. Hosted browser infrastructure, if used, can have its own pricing. This guide makes no claims about relative runtime speed.
• Responsible use: Automate only sites and accounts you are authorized to use. Protect credentials and session files, and follow the site’s terms and applicable rules. Do not treat browser automation as a way to bypass access controls or bot checks.

10. Troubleshooting

Symptom	Likely cause	Fix
Executable or browser binary is missing	The package is installed but its compatible browser binary is not, or the library was upgraded.	Run python -m playwright install chromium (or install the browser you use) in the same environment.
Browser launch is blocked in managed Chrome or Edge	Enterprise policy can restrict control of branded browsers.	Check the organization’s browser policy and use an approved engine or setup; consult the official browser guide.
Timeout waiting for a button or heading	The page is still loading, the locator does not match, or the expected state never occurs.	Inspect the page and locator, wait for the exact prerequisite state, and confirm the workflow succeeded before waiting for its result.
Click has no effect	The target may be covered, disabled, ambiguous, or not the intended control.	Use a specific role/name locator, wait for the correct state, and inspect the rendered page in headed mode.
Script works locally but fails in CI	Different installed browser binaries, missing system dependencies, timing, or environment configuration.	Install the documented browsers in the CI image, keep library and browser setup aligned, and wait for observable page state rather than relying on local timing.
Async runtime error about an existing event loop	asyncio.run() was called inside an already running loop.	Await the async workflow from the existing loop; reserve asyncio.run() for a standalone entry point.
Download file is missing	The script clicked before registering the download listener or saved to an unexpected path.	Wrap the click in expect_download() and explicitly save the suggested filename to a known directory.

11. Or skip the browser setup

If your task is to capture a page as an image or PDF rather than interact with its controls, ScreenshotNeo offers a website screenshot API and MCP server. The DIY Playwright flow above remains the right fit when the task needs browser interaction. For a screenshot, one GET request returns an image or PDF; see the ScreenshotNeo API documentation.

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)

ScreenshotNeo removes cookie and consent banners, newsletter popups, and chat widgets before the shot; each cleanup step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server includes take_screenshot, get_page_info, and capture_pdf for AI agents. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. Create a free ScreenshotNeo account.

12. FAQ

Can I use Playwright without opening a visible browser window?

Yes. The examples launch Chromium with headless=True. Set headless=False when you need to watch the workflow while debugging.

Should I use Playwright sync or async?

Use sync for a simple linear script. Use async when the surrounding Python application already uses asyncio or when you need to coordinate asynchronous work.

Does a screenshot require browser automation?

Not necessarily. Use Playwright when you need to manipulate the page before capture. For a direct website screenshot or PDF, ScreenshotNeo’s API can return the artifact without you managing a browser runtime.

Can I automate any website?

Technical capability does not establish permission. Use sites and accounts you are authorized to access, safeguard credentials, and follow applicable site rules.

Primary references

• Playwright Python library: installation and sync/async examples
• Playwright Python browsers: supported browsers, binaries, and channels
• Playwright Python pages: navigation, locators, fill, and click
• Playwright Python navigations: lifecycle and dynamic content
• Selenium Python API documentation