How to Monitor Pages That Require Login with Urlwatch

Start with a Urlwatch URL job and a valid session cookie if the server returns the page content when that cookie is sent. If the page content is created by JavaScript, use a Browser job. If the page gets its data from a suitable API, monitoring that response may be faster and lighter. There is no universal login recipe: the right method depends on the site’s authentication and rendering, and you should verify that Urlwatch receives the authenticated content before trusting alerts.

Use this only for pages you are authorized to access and monitor. Do not put live credentials or session values in a public repository or shared configuration.

1. Choose the retrieval method

Method	Use it when	Tradeoff
URL job with cookies	The server returns the needed content for a request carrying a valid session cookie.	Cookies expire or rotate; a later request may return a login page.
URL job for an API	A permitted, suitable endpoint provides the data you want to monitor.	Authentication, access, and endpoint stability depend on the site.
Browser job	The content appears only after JavaScript runs in a browser.	Requires Playwright and installed browsers, and uses substantially more resources than a URL job.
Custom hook or job	The site’s login flow needs site-specific handling beyond the standard job configuration.	You must implement and maintain the custom code. The example login hook is illustrative, not a built-in general-purpose login feature.

Urlwatch’s documentation recommends considering an API response when it supplies the required information, since it can often be faster than browser rendering. Use an API only when its use is permitted and the interface is appropriate for your monitor. See the Urlwatch Jobs documentation.

2. Try a URL job with a session cookie

A normal URL job retrieves a document from the server. Urlwatch supports cookies, HTTP methods, and request data. If you can obtain a valid session cookie through an authorized login, configure the required cookie name and value in the job.

name: "Authenticated page"
url: "https://example.invalid/private-page"
cookies:
  SESSION_COOKIE_NAME: "<session-cookie-value>"
filter:
  - css: "main"

This is a configuration-shape example, not a tested configuration for a real site. Replace the placeholder URL, cookie name, and value with the site’s actual values. Cookie formats, expiry, and whether one cookie is sufficient are site-dependent. The CSS filter is optional; choose a selector that isolates the content whose changes matter.

Find and validate the cookie

1. Sign in to the site through its normal, authorized process.
2. Use the browser’s developer tools or another approved method to identify the cookie associated with the authenticated session. Do not copy unrelated cookies unless the site requires them.
3. Put the cookie into a private local Urlwatch configuration. Protect that file from other users and exclude it from version control.
4. Run the job and inspect the retrieved output. Confirm it contains the private page’s content rather than a login form, redirect response, or access-denied page.
5. Apply a filter to reduce irrelevant page noise, then inspect the filtered result and its diff before enabling notifications.

A cookie is a bearer credential: anyone who obtains a valid session value may be able to use it. Keep it out of logs, screenshots, shared examples, issue reports, and source control. If the site supports a dedicated, limited-purpose account or token, follow its documented security guidance.

3. Monitor an API response when it fits

If the page obtains the desired information from an API endpoint, a URL job can be a simpler target than rendering the whole page. The Urlwatch documentation supports HTTP methods and request data for URL jobs, but it does not establish a universal method for authenticating to every API.

As a diagnostic, you can inspect the page’s network requests in browser developer tools to understand where its data comes from, provided doing so is allowed by the site’s terms and your authorization. Determine whether that endpoint is intended and stable enough for your use. Then configure the request details and any required authentication according to the site’s own documentation. Do not assume that a browser cookie, endpoint, or request format will remain valid indefinitely.

Validate the response just as you would a page: check that it contains the expected data, filter to the fields that matter, and detect authentication failures so an error response does not look like a meaningful content change.

4. Use a Browser job for JavaScript-rendered content

Use a Browser job only when the content you need is unavailable in the server response or a suitable API response and JavaScript must run to expose it. In Urlwatch 2.29 documentation, Browser jobs use Playwright, require the optional Playwright package and installed browsers, and are more resource-intensive than URL jobs. The documented options include navigation, load waiting, selectors to wait for, and browser choice.

# Shape of a Browser job; adapt options to the installed Urlwatch version.
kind: browser
navigate: "https://example.invalid/private-page"
wait_until: "networkidle"
wait_for:
  - "main"
filter:
  - css: "main"

Install the optional Playwright dependency and browser binaries using the instructions for your Urlwatch installation and version. Consult the official job documentation for the exact supported keys and syntax. Browser-job configuration can vary by release; do not treat this illustrative shape as a guarantee that every listed key is accepted by every version.

The browser job must still be authenticated. A JavaScript-rendering job alone does not log in for you. Where the documented browser-job options do not cover the site’s authentication flow, you may need a suitable cookie-based setup or custom integration. Complex flows involving CSRF state, OAuth, multifactor authentication, CAPTCHA, or rotating sessions are site-specific and may not be automatable by a generic configuration.

5. Treat the custom-login example as custom code

The Urlwatch example configuration contains a commented-out custom-login example. It says the job kind is defined in hooks.py and must be enabled; its username and password fields illustrate parameters for that custom job. It is not a turnkey built-in form-login system. Review the official example configuration and the corresponding hook code for the version you use before adapting it.

A custom integration may be appropriate when standard URL or Browser jobs cannot handle a site’s workflow and you can maintain the code. Do not assume that submitting a username and password handles CSRF tokens, multifactor prompts, anti-bot checks, or session renewal. Follow the site’s permitted access method and protect any secrets the custom job uses.

6. Filter output and guard against false change alerts

Urlwatch compares retrieved output with prior output and can notify through configured reporters, including email, terminal, Slack, Matrix, Discord, Telegram, and ntfy. Reporters deliver the result; they do not solve authentication. See the Urlwatch handbook for job, filter, and reporter configuration.

1. Run the job and inspect its raw output for redirects, login prompts, access errors, or unexpected content.
2. Filter the output to the relevant section or fields, reducing changes from navigation, timestamps, advertisements, and other unrelated page elements.
3. Use the example configuration’s --test-filter workflow to inspect what your filter retains.
4. Review several runs, including a fresh authenticated session, before relying on alerts.
5. Check that a session-expiry page or failed response is recognizable as an error and does not silently become the new expected content.

Or skip the browser setup

ScreenshotNeo takes a screenshot or PDF with one GET request. Its capture flow accepts cookie and consent banners like a visitor and removes 60+ known consent platforms, newsletter popups, and chat widgets before the shot; each cleanup step can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers report the page verdict and billing status. An MCP server provides take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.

This is a screenshot API, not a replacement for Urlwatch’s change-detection and notification workflow. For pages your authorized session can access, use its cookie options as documented and validate the resulting capture. See the ScreenshotNeo website and API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webp

Python:

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://example.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://example.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo includes 1,000 screenshots per month free with no card; paid plans start at $5 for 3,000 screenshots. Sign up for 1,000 free screenshots a month, with no card.

Troubleshooting

Symptom	Likely cause	What to check
Urlwatch retrieves a login page	The cookie is missing, invalid, expired, or insufficient; the site may redirect unauthenticated requests.	Sign in again, confirm the required cookie names and values, and inspect the final retrieved output for a redirect or login form.
The page loads but the target content is absent	The server response may not include content rendered later by JavaScript.	Check whether an appropriate API provides the data. If not, use a Browser job with the required Playwright installation and wait condition.
A Browser job fails to start	The optional Playwright package or browser binaries may be missing, or the job configuration may not match the installed version.	Install the documented dependency and browsers; compare the job keys with the documentation for your Urlwatch version.
The Browser job returns before the content appears	Navigation completion does not necessarily mean the target content has rendered.	Use the documented selector-wait or load-wait options appropriate to the page, then inspect output again.
Every run reports changes	Dynamic or irrelevant page content is included, or the monitor alternates between authenticated and expired-session output.	Filter to the meaningful content and verify the session remains valid on each run.
A custom login job does not work	The sample is commented and depends on a hook; the site’s flow may need more than username and password.	Enable and maintain the corresponding custom code only if appropriate. Account for site-specific state and authentication requirements.
API output changes or authentication fails	The endpoint or its authorization may be unstable, restricted, or changed by the site.	Use the site’s documented interface, verify permission, and validate both successful and error responses.

Performance, reliability, and cost

• Prefer the lightest method that returns the right content. URL jobs avoid browser rendering overhead; Urlwatch’s documentation notes that an API can often be faster than a Browser job.
• Browser jobs use more resources. Use them when rendering is necessary, and avoid adding browser setup when a direct response is sufficient.
• Sessions need maintenance. Cookie expiry or rotation can interrupt monitoring. Revalidate the authenticated response periodically and after changes to the site’s login flow.
• Filters improve signal. Monitoring a stable, relevant fragment reduces alerts caused by unrelated page changes.
• Plan for failure states. Login pages, access errors, and missing content should be noticed as retrieval failures rather than accepted as ordinary updates.
• Cost depends on your runtime and configuration. Browser jobs consume substantially more resources than URL jobs; no specific runtime cost or benchmark is established here.

FAQ

Can Urlwatch log in to any website automatically?

No universal login flow is documented. Whether a cookie, API request, Browser job, or custom integration works depends on the site’s authentication and access rules.

Will a cookie keep working forever?

No. Session cookies can expire or rotate. Verify the response regularly so an expired session does not produce misleading alerts.

Do email or chat reporters authenticate the page?

No. Reporters send change notifications after retrieval; authentication must be handled by the job or an integration.

Is the custom-login example a built-in feature?

No. The example describes a hook-based job that must be enabled and adapted; it is not a general-purpose login mechanism.

Where can I check Urlwatch’s supported job options?

Use the Urlwatch Jobs documentation for the current documented URL and Browser job configuration.