How to use Urlwatch with a website that needs JavaScript rendering

If a page’s meaningful content appears only after JavaScript runs, configure urlwatch to use a Browser job. Set navigate to the page URL; urlwatch uses Playwright to render it. Install urlwatch’s optional Playwright dependency and the browser binaries, then choose a readiness condition such as wait_for when the desired content appears after navigation. Browser jobs use substantially more resources than ordinary URL jobs, so first check whether the data is available from an API that a regular URL job can monitor.

1. Decide whether you need a browser

A regular urlwatch url job reads the server response. If that response already contains the text you want to track, use it. If the content appears only after client-side JavaScript runs, use a Browser job with navigate. urlwatch describes Browser jobs as resource-intensive and recommends using them only when a URL job does not return the correct results. urlwatch Jobs documentation

Before launching a browser, inspect the page’s network activity. If the page fetches the required data from an API endpoint, and that endpoint is accessible and stable enough for your use, monitor it with a URL job instead. This can avoid browser startup and rendering work. Confirm that the endpoint returns the exact data you need before depending on it.

2. Install urlwatch and its browser runtime

The Browser job needs urlwatch’s optional Playwright package and installed Playwright browser binaries. Follow the current installation instructions for your urlwatch version, install the optional Playwright dependency, then run playwright install. Install the browser you intend to select in the job configuration. The urlwatch documentation lists browser selection as an option; the selected browser must be installed in the environment where urlwatch runs. See the official job documentation for current package and browser details.

Keep the runtime and browser installation in the same environment used for scheduled checks. Installing urlwatch in one virtual environment and Playwright’s browser binaries in another is a common cause of jobs that work interactively but fail under a scheduler.

3. Add a Browser job in urls.yaml

Use urlwatch --edit to edit the job list, or edit the YAML file directly. Jobs are separated by ---. A minimal JavaScript-rendered page job is:

name: "JavaScript-rendered product page"
navigate: "https://site.example/product"

The navigate key selects the Browser job. Replace the example URL with the page you need to monitor. Give the job a meaningful name so notifications identify the page clearly. If a regular URL job is still present for the same page, remove or disable the redundant job once you have confirmed the browser result.

4. Wait for the content you actually monitor

A page navigation event does not necessarily mean that a JavaScript application has populated the content you care about. Choose a wait condition based on how the site loads.

Setting	What it waits for	When to use it
`wait_until: load`	The load event	When the page’s relevant work completes by the normal load event.
`wait_until: domcontentloaded`	The DOMContentLoaded event	When the initial document is enough to begin, and the target content is already present by then.
`wait_until: commit`	Until a response is received and document loading begins	When you need an early navigation milestone and will wait for the content separately.
`wait_until: networkidle`	A period with no network connections	Use only when it is a suitable signal for the page; urlwatch’s advanced guide discourages relying on it by default.
`wait_for`	A CSS or XPath selector to appear	Usually the clearest choice when a known element signals that the monitored content is ready.

The documented default timeout for wait_for is 30 seconds. Choose a selector for the actual content, such as a result title or price element, rather than a page wrapper that exists before asynchronous data arrives. A generic wrapper can appear too early and leave urlwatch comparing incomplete output. See the Jobs documentation and advanced guide for the supported settings and behavior.

For example, add a selector that exists only when the target content is ready:

name: "JavaScript-rendered product page"
navigate: "https://site.example/product"
wait_for: "main .product-details"

Use the exact option shape supported by your installed urlwatch release; consult its job documentation if you need to combine or adjust navigation and selector waits. Do not assume that networkidle means an application is finished: analytics, polling, and other persistent requests can make network activity a poor readiness signal.

5. Filter the rendered result

Start by checking what the Browser job returns. Once the rendered content is correct, add urlwatch filters to extract or normalize the part that matters. Filters can select HTML elements, convert content, and reduce the monitored output. This keeps unrelated page changes from generating notifications. Add filters after confirming that the browser has loaded the desired content, or a correct filter may appear empty simply because the page was captured too early. See the urlwatch introduction and documentation for filter chains.

6. Schedule recurring checks

urlwatch compares each run’s filtered output with the previous result and reports differences through configured reporters. Run it from cron or another scheduler to check repeatedly. The official quick start recommends not running checks more often than every 30 minutes and shows this cron example:

*/30 * * * * urlwatch

Make sure the scheduled command uses the same configuration, Python environment, Playwright installation, and browser binaries as your manual run. The actual interval is determined by the scheduler. See the urlwatch quick start.

Browser defaults can be configured globally under job_defaults.browser in urlwatch.yaml. Keep those defaults in the configuration file, separate from individual job definitions in urls.yaml. Use per-job settings when a page needs a different readiness condition from the rest.

7. Troubleshoot common failures

Symptom	Likely cause	Fix
Content is missing, although it appears in a normal browser.	The job is a regular `url` job, or the browser capture happens before client-side rendering completes.	Use `navigate` to select Browser mode. Add a meaningful `wait_for` selector or choose a suitable `wait_until` condition.
The job reports a missing Playwright module or cannot launch a browser.	The optional Playwright dependency or browser binaries are not installed in the runtime used by urlwatch.	Install the optional package and run `playwright install` in the relevant environment. Confirm that the browser selected by the job is installed there.
The job times out waiting for an element.	The selector is wrong, the element never appears, or the page is slower than the configured timeout.	Inspect the rendered DOM and correct the CSS or XPath selector. Choose an element that reliably indicates the desired content. The documented default selector wait timeout is 30 seconds; adjust the supported timeout setting when a slower page warrants it.
Network-idle waiting takes too long or behaves inconsistently.	The site keeps background connections active, or network quiet does not correspond to application readiness.	Prefer a selector tied to the target content, or another navigation condition that matches the page. The advanced guide discourages treating `networkidle` as a universal signal.
The first manual run works, but scheduled runs fail.	The scheduler uses another working directory, configuration, Python environment, or user account.	Use explicit paths and the same runtime as the successful manual run. Verify that the scheduled account can access the configuration and browser installation.
Notifications include unrelated changes or are always noisy.	The job compares too much rendered page content, including dynamic or irrelevant regions.	After confirming the rendered output, use filters to select and normalize the specific content being monitored.
Monitoring is slow or resource-heavy.	A full browser is being launched when the raw response or an API already contains the data.	Check whether a regular URL job can monitor the server response or a suitable API endpoint. Reserve Browser jobs for content that requires rendering.
Old instructions recommend hooks for this task.	They may describe urlwatch 1.x.	For current 2.x usage, follow the current Browser job documentation. The migration guide says hooks were replaced in 2.0 by support for extending job kinds, filters, and reporters.

8. Performance, reliability, and operating cost

A Browser job starts and runs a headless browser, so it consumes substantially more resources than a URL job. If an API response or server-rendered page supplies the target data, monitoring that with a URL job is generally the lighter design. If rendering is necessary, make each check do only the work needed: wait for a specific content signal and filter the result to the relevant information.

For reliability, monitor a selector tied to the content rather than relying only on a generic navigation event. Keep the selector stable, check it when the target site changes, and use a timeout that allows the page enough time to render without leaving a broken job waiting indefinitely. Treat a site’s API endpoint as a site-specific dependency: verify access and returned fields before switching to it.

Schedule checks at an interval that fits the page’s update cadence and urlwatch’s guidance. The quick start recommends no more frequently than every 30 minutes. The dossier provides no browser runtime benchmark or fixed hosting price, so size the machine and estimate recurring hosting costs from your own deployment rather than assuming a universal figure. A small always-on host may be useful for unattended cron runs, but that is an operational inference, not a urlwatch product requirement.

Or skip the browser setup

If you need a screenshot of the rendered page instead of a change-monitoring workflow, ScreenshotNeo is a website screenshot API and MCP server. One GET request returns an image or PDF; see the ScreenshotNeo API documentation for options and response details.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before the shot. Bot checks, blank pages, and failed loads are never billed. Its MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. Sign up for 1,000 free screenshots a month, with no card.

FAQ

Does urlwatch execute JavaScript for every job?

No. Use a Browser job only for pages whose needed content is absent from the ordinary URL response. Browser jobs are more resource-intensive.

Can urlwatch monitor changes, or does it only take screenshots?

urlwatch compares job output between runs and reports differences. ScreenshotNeo returns screenshots or PDFs; it is a screenshot API and MCP server rather than the urlwatch change-comparison workflow described here.

The supplied urlwatch research does not specify an authentication configuration for this scenario. Check the current urlwatch job documentation for supported browser settings and the site’s access rules before relying on an authenticated page.

Do I need to write a custom urlwatch hook?

For current urlwatch 2.x usage, the documented Browser job is the relevant built-in approach. The migration guide notes that the old hooks mechanism was replaced in 2.0 by extension support for job kinds, filters, and reporters.