How to Monitor a Website with a Curl Script

A reliable curl monitor makes one bounded HTTP request, checks an explicit status rule, records the result, and exits nonzero when the check fails. That exit code lets cron, systemd, or an alerting wrapper detect a problem. Use GET when you need to validate content; use HEAD only if the endpoint supports it.

1. Create a curl monitoring script

This POSIX shell script follows redirects, limits connection and total time, saves the response body to a temporary file, records the final HTTP status and elapsed time, and optionally checks for a stable body marker. It accepts 2xx and 3xx responses by default; change the rule if your endpoint needs a narrower range.

#!/usr/bin/env sh
set -eu

url=${1:-https://example.com/health}
connect_timeout=5
max_time=15
body_file=$(mktemp "${TMPDIR:-/tmp}/curl-monitor.XXXXXX")
trap 'rm -f "$body_file"' EXIT HUP INT TERM

if result=$(curl --silent --show-error --location \
  --connect-timeout "$connect_timeout" --max-time "$max_time" \
  --output "$body_file" \
  --write-out '%{http_code} %{time_total}' \
  "$url"); then
  :
else
  rc=$?
  printf '%s TRANSPORT_ERROR curl_exit=%s url=%s\n' "$(date -u +%FT%TZ)" "$rc" "$url" >&2
  exit 1
fi

http_code=${result% *}
time_total=${result#* }

case "$http_code" in
  [0-9][0-9][0-9]) ;;
  *) printf '%s INVALID_STATUS url=%s\n' "$(date -u +%FT%TZ)" "$url" >&2; exit 1 ;;
esac

# Accept 200-399. Tighten this to 200-299 if redirects should fail.
if [ "$http_code" -lt 200 ] || [ "$http_code" -ge 400 ]; then
  printf '%s HTTP_FAILURE status=%s elapsed=%ss url=%s\n' \
    "$(date -u +%FT%TZ)" "$http_code" "$time_total" "$url" >&2
  exit 1
fi

# Optional content assertion: uncomment and set a stable, non-secret marker.
# if ! grep -Fq 'healthy' "$body_file"; then
#   printf '%s BODY_FAILURE status=%s elapsed=%ss url=%s\n' \
#     "$(date -u +%FT%TZ)" "$http_code" "$time_total" "$url" >&2
#   exit 1
# fi

printf '%s OK status=%s elapsed=%ss url=%s\n' \
  "$(date -u +%FT%TZ)" "$http_code" "$time_total" "$url"

Save it as check-site.sh, then make it executable with chmod 755 check-site.sh. Run ./check-site.sh https://example.com/health. The script writes one concise log line and returns status 0 for success or 1 for a failed check.

2. Choose what counts as healthy

Check	Use it when	Trade-off
HTTP status	You need to know that the endpoint responds with an accepted status.	A 200 page can still contain an application error.
Body marker	A health endpoint returns stable, deterministic content.	Markers on personalized pages can vary; avoid checking an exact string on a home page.
Elapsed time	You want to log latency or enforce a response-time limit.	A slow response can be a warning rather than an outage; choose thresholds for your service.
TLS validity	Certificate validity is part of the service’s health.	Do not disable certificate verification to make a failed check pass.

The sample accepts 200–399 after redirects. If redirects are unexpected, remove --location and treat 3xx as failure, or adjust the status rule to accept only 200–299. With redirects enabled, curl’s reported HTTP code is the final response code. Follow redirects only when the destination is part of the intended health check.

3. HEAD or GET?

HEAD requests headers without downloading the body, which can reduce transfer for a simple availability check. The curl project documents that --head (-I) makes a HEAD request, and notes that some servers reject HEAD even when other requests work. Use GET when you need a content assertion or when HEAD is unsupported. With GET, use --output to keep the body out of the terminal.

# HEAD: status and timing, no response body
curl --silent --show-error --head --location \
  --connect-timeout 5 --max-time 15 \
  --output /dev/null --write-out 'HTTP %{http_code} in %{time_total}s\n' \
  https://example.com/health

# GET: useful when checking content
curl --silent --show-error --location \
  --connect-timeout 5 --max-time 15 \
  --output /tmp/health-body \
  --write-out 'HTTP %{http_code} in %{time_total}s\n' \
  https://example.com/health

For a production monitor, keep the script’s explicit status evaluation even if you add curl’s --fail. That option makes HTTP error responses fail the command and suppresses the body, but recording the code helps distinguish an HTTP 404 or 500 from DNS, TLS, or timeout failures.

4. Configure timeouts, redirects, and diagnostics

--connect-timeout 5 bounds the time spent establishing a connection.
--max-time 15 bounds the complete request, including response transfer. Set it long enough for normal responses but short enough that scheduled runs cannot pile up.
--location follows redirects. Omit it when a redirect itself should be reported as unexpected.
--speed-limit BYTES with --speed-time SECONDS aborts a transfer that stays below a chosen rate for the specified period. This is useful for stalled downloads, but is usually unnecessary for a small health endpoint.
--verbose shows curl’s request and connection details for diagnosis. Avoid leaving verbose logs enabled if requests include sensitive headers.

For stricter TLS checks, curl verifies certificates by default. A certificate problem should be investigated at the server or trust-store level; do not use --insecure as a monitoring fix. The monitoring-plugins check_curl reference is useful when you need established warning and critical time thresholds, redirect controls, timeout, TLS settings, or certificate-validity checks.

5. Add authentication safely

Prefer a small, deterministic health endpoint that does not require credentials. If authentication is necessary, keep secrets in a protected environment or secret store, pass only the required headers, and ensure logs never print them. For example, curl can read a token from an environment variable:

curl --silent --show-error --location \
  --connect-timeout 5 --max-time 15 \
  --header "Authorization: Bearer ${HEALTH_TOKEN:?set HEALTH_TOKEN}" \
  --output /dev/null --write-out '%{http_code} %{time_total}\n' \
  https://example.com/private-health

Environment variables are more convenient than hard-coding secrets in a script, but access to the process environment and scheduler configuration still needs to be protected. Avoid putting tokens in command-line arguments where local process inspection or logs could expose them.

6. Schedule checks and make failures visible

A cron job can run the script every five minutes and append both normal output and errors to a log:

*/5 * * * * /usr/local/bin/check-site.sh https://example.com/health >>/var/log/check-site.log 2>&1

Install the script at the referenced path, ensure the cron user can execute it and write the log, and use absolute paths because cron often has a limited environment. The script’s nonzero exit code is the signal for cron email, a wrapper that pages on failure, or a heartbeat service. If you need alert routing or retained history, make sure the alerting layer records a failed run rather than only missing output.

Systemd timers or another scheduler are also suitable. Whichever scheduler you use, avoid overlapping runs: the interval should be longer than the maximum request plus expected execution time, or add a lock if overlap would cause trouble.

7. Troubleshooting

Symptom	Likely cause	What to do
curl exits nonzero before an HTTP status is recorded	DNS resolution, connection, TLS negotiation, or timeout failed.	Read the curl error on stderr; check DNS, routing, certificate chain, and firewall rules. Keep certificate verification enabled.
HEAD returns 405 or 501 while a browser works	The server or proxy does not allow HEAD.	Use GET and write the body to a temporary file or `/dev/null`.
The check reports 3xx as healthy unexpectedly	The sample accepts 200–399 and follows redirects.	Remove `--location` to detect redirects, or narrow the accepted status range to 200–299.
A 404 or 500 is logged as a transport failure	The script may be relying on `--fail` without separately recording the HTTP status.	Capture `%{http_code}` with `--write-out` and classify HTTP status separately from curl’s transport exit.
It works interactively but not in cron	Different PATH, user permissions, environment, working directory, or log access.	Use absolute paths, set needed environment variables in a protected way, and check permissions as the cron user.
Checks overlap or accumulate	The request can run longer than the schedule interval.	Bound the total request with `--max-time`; lengthen the interval or use a lock.
Content check fails intermittently	The marker is personalized, localized, dynamic, or absent on an error template.	Use a dedicated health endpoint with stable content and avoid secrets in the marker.
Logs contain credentials or sensitive response data	Verbose mode, headers, URL query parameters, or body output are being recorded.	Remove sensitive data from URLs, avoid verbose production logs, do not print the body, and restrict log permissions.

8. Performance, reliability, and cost

A local curl script is lightweight and inexpensive: it uses the machine and network where it runs, and needs only the schedule and log storage you provide. It sees the site from one vantage point, so a healthy result from your server does not prove the site is reachable from other networks or regions. It also depends on the host, scheduler, DNS, and outbound network remaining available.

Use a small health endpoint to reduce transfer and keep checks predictable. Avoid an overly frequent interval that creates needless traffic or masks scheduling delays. Record timestamp, target, status, duration, and failure class; rotate logs and set permissions so they do not grow without limit or expose secrets. A hosted monitor is useful when you need independent external checks, retained history, or notification workflows. Compare its method, status/content rules, timeout and latency thresholds, redirect/TLS behavior, authentication, alert channels, retention, and check locations.

9. Or skip the browser setup

A curl health check tells you whether an HTTP endpoint responds. If the job is capturing a rendered page, ScreenshotNeo is a website screenshot API and MCP server: one GET request returns PNG, JPEG, WebP, or PDF, and its API documentation covers the request options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Cookie and consent banners, newsletter popups, and chat widgets are removed before capture; each cleanup step can be turned off.
Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed. Responses include page-verdict and billing headers.
An MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf.
The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. All features are on every plan.

Create a free ScreenshotNeo account for 1,000 screenshots a month with no card.

10. FAQ

Does a successful curl command prove a website is fully healthy?

No. It confirms only the conditions your script checks from the machine running it. A separate application health endpoint and external vantage points may be needed for broader coverage.

Should I check the home page or a health endpoint?

Prefer a small health endpoint with predictable status and content. A home page can be personalized, cache-dependent, or slower and less deterministic.

When should I replace a script with hosted monitoring?

Move when you need checks outside your own network, durable history, multiple notification routes, or managed scheduling and alerting.

Sources

curl: HTTP scripting — HEAD requests, response headers, and verbose diagnostics.
curl FAQ — HTTP failure behavior.
curl man page — timeout and transfer-speed options.
Monitoring Plugins: check_curl — timing, timeout, redirects, TLS, and certificate checks.