How to Add Custom Headers and Cookies to PageCrawl.io Requests

To add a custom header to a monitored PageCrawl page, open that page’s settings and enable Power User mode. This reveals the Custom Headers setting for the target request. To set a cookie in the monitored page’s browser context, add a Custom JavaScript action; PageCrawl’s documented example is document.cookie = 'lang=en; path=/; max-age=86400'. These are separate from the Authorization: Bearer ... header used to authenticate requests to PageCrawl’s management API.

PageCrawl’s available help documentation does not specify the Custom Headers entry syntax or confirm API payload keys for target headers or cookies. Follow the UI’s current instructions, and do not assume that a Cookie header belongs in the Custom Headers field. If you need to configure these settings through the API, check the current PageCrawl OpenAPI specification before relying on field names.

1. Add a custom header to a monitored page

1. Open the monitored page’s settings in PageCrawl.
2. Enable Power User mode to reveal advanced settings.
3. Find Custom Headers and configure the header there for requests to the monitored target.
4. Save the settings and check the monitor’s next run to confirm the target accepts the request.

The official advanced-configuration guide identifies Custom Headers as a Power User setting, but the documentation available for this article does not show a concrete key/value syntax or an API example. Use the format shown in your PageCrawl interface rather than guessing at a JSON structure. PageCrawl: Advanced Configuration Options for Power Users.

When a header is for authentication

Use the target site’s required header name and value as specified by that site. For example, a site may require an authorization token, but the exact header and credential format depend on the site. Do not put a PageCrawl management API token into a target-site header unless the target itself explicitly requires that credential.

2. Set a cookie in the monitored browser

To set a cookie in the page’s browser context, create a Custom JavaScript action and use a document.cookie assignment. PageCrawl documents this example:

document.cookie = 'lang=en; path=/; max-age=86400'

This sets a cookie named lang to en, with the path / and a lifetime of 86,400 seconds (one day). Adapt the cookie name, value, path, and lifetime to the target site’s needs. PageCrawl says custom JavaScript actions run in the browser context before tracked elements are extracted. PageCrawl: JavaScript Tracked Elements and Custom JavaScript Actions.

Cookie scope and limitations

• Path: The cookie is sent for matching paths. Use path=/ when it should apply across the site.
• Lifetime: max-age=86400 makes it expire after one day. Omit or change the lifetime only if that matches the intended session behavior.
• Domain: A cookie applies according to browser cookie domain rules. Do not add a domain attribute unless the target’s domain setup requires it.
• Secure and HttpOnly: JavaScript cannot create an HttpOnly cookie. A Secure cookie is intended for secure connections. Check the target site’s requirements before choosing attributes.
• Timing: The action must run in the browser context before the tracked elements are extracted. If the target reads a cookie earlier during navigation, a JavaScript action may be too late; use a documented cookie mechanism if PageCrawl provides one in the current interface, or consult its current API documentation.

3. Authenticate requests to PageCrawl’s API

The PageCrawl management API uses a Bearer token in the HTTP Authorization header. This authenticates your client to PageCrawl; it does not authenticate the browser to the monitored website.

cURL

curl -H "Authorization: Bearer YOUR_API_TOKEN" \
  "https://pagecrawl.io/"

Use the documented API endpoint and request method for the operation you need; the URL above is illustrative of the header form, not a PageCrawl API operation. Consult the API and Webhooks guide for the actual endpoint and parameters.

Python

import requests

api_token = "YOUR_API_TOKEN"
url = "PAGECRAWL_API_ENDPOINT"

response = requests.get(
    url,
    headers={"Authorization": f"Bearer {api_token}"},
    timeout=30,
)
response.raise_for_status()
print(response.text)

Replace PAGECRAWL_API_ENDPOINT with the endpoint and method documented for your operation. Add any operation-specific parameters according to the API reference.

JavaScript (Node.js)

const apiToken = process.env.PAGECRAWL_API_TOKEN;
const url = 'PAGECRAWL_API_ENDPOINT';

const response = await fetch(url, {
  headers: { Authorization: `Bearer ${apiToken}` },
});

if (!response.ok) {
  throw new Error(`PageCrawl API returned ${response.status}`);
}

console.log(await response.text());

Set PAGECRAWL_API_TOKEN in your environment and replace the placeholder URL with the documented endpoint. Keep API tokens out of source control and client-side code.

4. Keep the three contexts straight

What you are configuring	Where it applies	Documented approach
Custom target-site HTTP header	Request to the monitored website	Enable Power User mode and use the Custom Headers setting.
Cookie for the monitored page	Browser context for the monitored page	Use a Custom JavaScript action with document.cookie.
PageCrawl API credential	Your request to PageCrawl’s management API	Send Authorization: Bearer YOUR_API_TOKEN.

5. API-only configuration: verify before sending

The readable PageCrawl API guide documents Bearer authentication for API requests and links an OpenAPI specification. It does not confirm the exact request fields for setting custom headers or cookies on a monitored target. Before automating those settings:

1. Open the current OpenAPI specification.
2. Find the operation for updating a monitored page or its settings.
3. Confirm the exact property names, value types, and whether headers or cookies are supported for that operation.
4. Use a non-sensitive test value first, then verify behavior with a monitor run.

Do not copy guessed fields such as headers or cookies into a payload and assume they are supported. The available documentation does not establish those names or whether a Cookie header should be entered in the Custom Headers field.

6. Troubleshooting

Symptom	Likely cause	What to check
Custom Headers is not visible	Power User mode is off.	Enable Power User mode in the monitored page’s settings, then check the advanced settings again.
The target still redirects to login	The target header or cookie is missing, malformed, scoped incorrectly, or applied too late.	Check the target’s authentication requirements, the configured header in the UI, cookie path and lifetime, and whether the cookie exists before extraction.
A cookie assignment has no effect	The action may run on the wrong page context, after the site already checked the cookie, or with an invalid attribute/value.	Confirm the action runs before tracked-element extraction, check cookie scope, and verify the target’s expected cookie name and value.
PageCrawl API returns an authentication error	The Bearer token may be missing, invalid, expired, or sent to the target site instead of the PageCrawl API.	Send Authorization: Bearer YOUR_API_TOKEN to the PageCrawl API endpoint and verify the token and endpoint.
API accepts the request but monitor behavior does not change	The payload fields may not be the documented fields for monitor configuration, or the operation may not support that setting.	Check the current OpenAPI spec and API guide; do not infer target header/cookie keys from the management API’s Authorization example.
Header works in a manual request but not in the monitor	The value, destination, redirects, or browser/request context differs.	Confirm the header is configured on the correct monitored page and that the target accepts it on the final request as well.

7. Reliability, performance, and cost considerations

• Use the narrowest credential: A target-site credential belongs only on requests that need it. A PageCrawl API token belongs only on your API client’s requests to PageCrawl.
• Account for expiration: A short-lived token or cookie can make a monitor fail later. Set an appropriate lifetime and arrange a renewal process when the target uses expiring credentials.
• Validate after changes: Check a subsequent monitor run after changing headers or JavaScript. A successful API update does not by itself prove that the target accepted the credential.
• Keep secrets controlled: Avoid placing credentials in public repositories, logs, or JavaScript that runs in an untrusted client environment.
• Performance: The cited setup documentation does not publish a performance impact for custom headers or cookie actions. Keep custom actions limited to what the page requires; additional page-side work can affect when the page reaches the state your monitor reads.
• Cost: The cited documentation does not provide pricing information for these configuration steps. Check PageCrawl’s current plan and usage terms for any account-specific cost implications.

Or skip the browser setup

If your task is to capture a page rather than monitor tracked changes, ScreenshotNeo provides a website screenshot API and MCP server. Its API supports custom headers, cookies, and Authorization; see the ScreenshotNeo API documentation. For example, this cURL request captures a page as WebP:

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

Equivalent Python request:

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)

Equivalent Node.js request:

const q = new URLSearchParams({
  access_key: 'YOUR_API_KEY',
  url: 'https://stripe.com'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`ScreenshotNeo returned ${res.status}`);
await Bun.write('shot.webp', res);

• Cookie banners are accepted and removed, along with known newsletter popups and chat widgets, before the screenshot; each step can be turned off.
• Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed; response headers report the page verdict and billing status.
• An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents.
• The Free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 screenshots. Every feature is on every plan.

Sign up for ScreenshotNeo’s free plan to get 1,000 screenshots a month with no card.

FAQ

Is the PageCrawl API Authorization header sent to the monitored website?

No. It authenticates your client to PageCrawl’s management API. Target-site headers are configured on the monitored page; browser cookies are set in the monitored page’s context.

Can I set a cookie by adding a Cookie header under Custom Headers?

The cited PageCrawl documentation does not establish that. It documents setting a browser-context cookie through a Custom JavaScript action, so use that method unless current PageCrawl documentation confirms another supported mechanism.

Can I create an HttpOnly cookie with document.cookie?

No. Browser JavaScript cannot set the HttpOnly attribute. Use a server-side or other supported cookie mechanism if the target requires an HttpOnly cookie.

Does ScreenshotNeo replace PageCrawl monitoring?

No. ScreenshotNeo is a screenshot API and MCP server. Use PageCrawl for the monitored-page configuration described here; use ScreenshotNeo when you need a screenshot or PDF capture.