How to Monitor Websites with an API

Direct answer: monitor a website with an API by creating a recurring HTTP or HTTPS check, defining the request and authentication, validating both the response status and expected content, measuring latency and TLS state, and routing failures to alerts. Use a browser-capable synthetic monitor when the check must execute JavaScript or verify a real user journey.

What API website monitoring does

An API-based monitor is a scheduled request plus explicit success criteria and alert handling. A check normally records:

Target URL, host, path, port, and HTTP method.
Request headers, body, cookies, and authentication.
Expected status code and optional response-content assertions.
Latency, timeout, redirect behavior, SSL and certificate signals.
Probe location, interval, retry policy, and maintenance windows.
Alert destinations such as email, chat, incident management, or webhooks.

Google Cloud public uptime checks issue requests from multiple locations and support HTTP, HTTPS, or TCP checks with paths, ports, headers, authentication, and required response content. Redirects are followed and the final response is evaluated. See the Google Cloud uptime-check documentation.

Choose the endpoint to monitor

Start with a deterministic endpoint. A public homepage is easy to understand, but a small /health or /ready endpoint is usually cheaper and less affected by content changes.
Add business-critical routes. Monitor login, checkout, search, or API routes separately when their failure would affect users.
Keep checks independent. One check should answer one question, such as “does the API return a valid customer list?”
Protect private endpoints. Store tokens and passwords in the monitoring provider’s protected configuration or secret store. Never commit them to source code or put them in a public URL.

Define request and success criteria

Status code alone is weak evidence. A server can return 200 OK while serving an error page, an empty response, or a degraded fallback. For important endpoints, combine several assertions:

Signal	Example rule	Why it matters
Status	Require `200` for a read request or `201` for creation	Catches HTTP-level failures
Content	Require a known phrase or JSON field such as `status: ok`	Detects application errors returned with a successful status
Latency	Alert when response time exceeds your user-facing threshold	Finds slow degradation before total failure
TLS	Track certificate validity and expiry	Prevents certificate outages
Location	Require success from the locations that represent customers	Separates regional failures from global failures

Run a basic HTTP check with cURL

This command checks an endpoint, follows redirects, fails on HTTP errors, and prints the status and total time. Replace the URL and adjust the expected status for your service.

curl --fail-with-body --silent --show-error --location \
  --max-time 20 \
  --write-out '\nstatus=%{http_code} time_seconds=%{time_total}\n' \
  https://example.com/health

To assert response content as well, capture the body and search it:

body="$(curl --fail-with-body --silent --show-error --max-time 20 https://example.com/health)" || exit 1
printf '%s' "$body" | grep -F '"status":"ok"' >/dev/null || {
  echo "health assertion failed" >&2
  exit 1
}

Run a check in Python

import sys
import time
import requests

URL = "https://example.com/health"
EXPECTED_STATUS = 200

started = time.perf_counter()
try:
    response = requests.get(URL, timeout=(5, 15), allow_redirects=True)
except requests.RequestException as exc:
    print(f"request failed: {exc}", file=sys.stderr)
    sys.exit(1)

elapsed = time.perf_counter() - started
print(f"status={response.status_code} latency_seconds={elapsed:.3f}")

if response.status_code != EXPECTED_STATUS:
    print("unexpected status", file=sys.stderr)
    sys.exit(1)

try:
    payload = response.json()
except ValueError:
    print("response was not JSON", file=sys.stderr)
    sys.exit(1)

if payload.get("status") != "ok":
    print("content assertion failed", file=sys.stderr)
    sys.exit(1)

Schedule this script from your monitoring service or a protected worker. Keep credentials in environment variables or the provider’s secret store:

import os
import requests

response = requests.get(
    "https://api.example.com/private/health",
    headers={"Authorization": f"Bearer {os.environ['HEALTH_TOKEN']}"},
    timeout=20,
)
response.raise_for_status()

Run a check in Node.js

const url = 'https://example.com/health';
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 20_000);
const started = performance.now();

try {
  const response = await fetch(url, {
    redirect: 'follow',
    signal: controller.signal,
  });
  const elapsed = (performance.now() - started) / 1000;
  console.log(`status=${response.status} latency_seconds=${elapsed.toFixed(3)}`);

  if (response.status !== 200) process.exitCode = 1;
  const payload = await response.json();
  if (payload.status !== 'ok') process.exitCode = 1;
} catch (error) {
  console.error(`request failed: ${error.message}`);
  process.exitCode = 1;
} finally {
  clearTimeout(timeout);
}

Use a managed monitoring API

Managed services provide the recurring schedule, distributed probe locations, result history, alert routing, and API lifecycle operations for creating, editing, deleting, listing, and reading check state. DigitalOcean documents URL and IP checks, latency, uptime, SSL certificates, and API-managed alerts in its Uptime documentation. Uptime.com documents scheduled HTTP(S) checks for websites and API endpoints, response-code validation, probe locations, custom headers, and access tokens.

When comparing providers, check these capabilities:

Area	Questions to answer
API lifecycle	Can you create, update, pause, delete, list, and read check state through an API?
Protocols	Are HTTP, HTTPS, TCP, and browser or scripted tests supported?
Network reach	Which probe regions are available? Can private networks be reached?
Request control	Can you set methods, headers, bodies, cookies, tokens, and authentication?
Assertions	Can checks validate status, text, JSON fields, redirects, latency, and TLS?
Alerting	Are webhooks, chat, incident integrations, escalation, and recovery alerts available?
Operations	What are the retention period, audit trail, dashboard, rate limits, and cost?

Probe locations, intervals, retries, and alerts

Locations

Select regions where customers and critical dependencies live. A failure from one location may indicate a regional routing or provider problem; failures from several independent locations are stronger evidence of an outage.

Intervals

Short intervals reduce detection time but increase request volume and cost. Use a fast check for critical paths and a slower check for low-risk pages. Keep the interval long enough that your endpoint can handle the resulting traffic.

Retries and confirmation

A single failed request can be caused by packet loss, a transient DNS issue, rate limiting, or planned maintenance. Use a retry or confirmation policy that fits the incident cost. Avoid hiding real failures with unlimited retries; alert after a bounded number of attempts and include every failed location.

Alert payloads

Make alerts actionable. Include the check name, URL, failing location, timestamp, status code, latency, assertion result, redirect target, and a short response context that does not expose secrets. Send recovery notifications so incidents can be closed automatically.

Browser checks for JavaScript-heavy websites

Basic uptime checks do not load page assets or run JavaScript. Google Cloud states: “Uptime checks don’t load page assets or run JavaScript.” An HTTP check therefore cannot prove that a client-side route, login flow, checkout, or rendered dashboard works.

Use a browser-capable or scripted synthetic test when the requirement is a user journey:

Open the page in a real browser context.
Wait for a selector, a network-idle condition, or a bounded delay.
Perform the required clicks, form input, or navigation.
Assert that the final URL, visible selector, or page data is correct.
Capture a screenshot or trace when the test fails.

Keep a separate cheap HTTP check for basic availability. The browser test should cover behavior that the HTTP check cannot observe.

Or skip the browser setup

ScreenshotNeo provides a website screenshot API and MCP server for browser-rendered checks. A GET request returns a PNG, JPEG, WebP, or PDF, and the capture can use full-page mode, a CSS selector, device presets or custom viewports, dark mode, retina scale, custom CSS and JavaScript, waits, blocked resources, headers, cookies, authentication, timezone, geolocation, and more. See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie and consent banners, newsletter popups, and chat widgets are removed before the shot. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the result with X-Page-Verdict and X-Billed headers. Its MCP server gives Claude, Cursor, and other MCP clients the take_screenshot, get_page_info, and capture_pdf tools. The free plan includes 1,000 screenshots each month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

Authentication, headers, and request bodies

Use a dedicated read-only token for monitoring.
Rotate tokens before expiry and monitor authentication failures separately from application failures.
Send required Accept, Content-Type, correlation, and version headers.
For POST or PUT checks, use a safe test account and an idempotent operation. Never trigger a real purchase or destructive mutation from a recurring monitor.
Mask authorization headers, cookies, and response fields in logs and alert payloads.

Redirects, DNS, TLS, and rate limits

Redirects: decide whether the final destination is the assertion target. A redirect loop or unexpected host should fail the check.
DNS: record resolution failures separately from HTTP failures so the owning team can act quickly.
TLS: validate hostname and certificate chain, and alert before expiry.
Rate limits: identify the monitor with a user agent, use a suitable interval, and allow-list probe IPs only when you understand the security impact.
Maintenance: pause or suppress alerts during planned changes, then verify that checks resume.

Performance, reliability, and cost

Performance

Measure DNS, connection, TLS, time to first byte, and total response time when the provider exposes those phases. Keep health responses small and deterministic. Avoid making a monitor download a large page when a dedicated endpoint can answer the same availability question.

Reliability

Use independent locations, bounded retries, explicit timeouts, and content assertions. Review false positives caused by redirects, expired credentials, transient network faults, maintenance, and rate limits. Test the configuration before routing alerts to an on-call team.

Cost

Request volume grows with the number of URLs, locations, and frequency. Browser tests and large responses generally consume more resources than a small health endpoint. Estimate monthly requests as:

monthly_requests = checks × locations × (minutes_per_month / interval_minutes)

Start with critical endpoints and representative regions, then add coverage based on incident history. Compare retention, alert integrations, browser-test usage, and API rate limits in addition to the advertised check price.

Troubleshooting common failures

Symptom	Likely cause	Fix
Unexpected 200 response	Error page or empty payload returned with success status	Add text or JSON-field assertions
Fails only from one region	Regional DNS, routing, firewall, or dependency issue	Compare probe locations and inspect regional access rules
Timeouts during traffic spikes	Slow dependency or threshold too low	Separate connection and response timeouts, inspect traces, and set a user-relevant threshold
401 or 403	Expired token, missing header, clock policy, or allow-list block	Use a dedicated token, verify headers, and update access rules securely
Certificate failure	Expired certificate, hostname mismatch, or incomplete chain	Renew the certificate and install the complete chain for the monitored hostname
Redirect loop	HTTP-to-HTTPS or canonical-host configuration loop	Inspect every `Location` response and assert the intended final URL
False alerts during deploys	Short-lived startup or migration window	Use maintenance suppression and bounded confirmation retries
Browser test passes but API check fails	Different headers, cookies, location, or authentication context	Align request context or keep separate checks for separate user paths
Browser test fails on a consent dialog	Overlay blocks the expected selector	Accept or remove the banner before interacting, then assert the page state

Deployment checklist

The target is public or reachable from the selected probe network.
The method, path, port, headers, body, and authentication are documented.
Status and content assertions match a known-good response.
Timeouts and latency thresholds reflect user impact.
At least two relevant probe locations are configured for critical services.
Secrets are stored outside source code and alert messages.
Failure and recovery notifications reach the responsible team.
Maintenance suppression and escalation rules are tested.
A browser or scripted test covers every JavaScript-dependent journey.

FAQ

Can an API monitor check a private website?

Only if the monitoring provider can reach the private network through an approved agent, tunnel, or private probe. A public uptime check cannot reach an internal host.

Should I monitor the homepage or a health endpoint?

Use both when they answer different questions: a small health endpoint for service availability and a homepage or browser journey for user-visible behavior.

How often should checks run?

Choose the shortest interval that fits your detection objective and request budget. Critical paths need faster detection; low-risk pages can run less often.

Why did a check pass while users reported an outage?

The check may use a different region, credentials, cache state, or protocol, or it may not execute JavaScript. Compare the failing user path with the monitor’s exact request context.

What should an alert contain?

Include the check, URL, location, timestamp, status, latency, assertion result, and safe response context so the responder can begin diagnosis without opening another system.