How to Detect Anti-Bot Protection on Websites

To detect anti-bot protection, compare what you requested with what you received. Look for a verification interstitial, a provider-specific response marker, or challenge HTML where an API or asset should have been returned. Treat a 403, 429, timeout, or blank page as an access failure until you have corroborating evidence.

A strong Cloudflare-specific signal is the cf-mitigated: challenge response header. Cloudflare documents that its challenge response uses text/html, even when the original request expected another resource type. That identifies a Cloudflare Challenge Page for that request; it does not prove that every route uses Cloudflare or that no other controls are present. See Cloudflare’s response detection guidance.

1. What anti-bot protection looks like

Anti-bot systems can act at several layers:

Layer	Evidence to inspect	What it can tell you
HTTP response	Status, headers, content type, body	Whether this request was intercepted or challenged
Rendered page	Interstitial, browser-check message, unexpected HTML	Whether navigation was gated before the destination
Session and browser behavior	Cookies, injected scripts, delayed navigation	Whether browser-side signals may be part of the decision
Owner telemetry	WAF, bot-management, or security logs	Which rule and provider made the decision, when you control the site

Cloudflare describes challenges as checks that verify whether a visitor is human or automated. Some challenges run JavaScript automatically, so the absence of a CAPTCHA or visible prompt does not prove that protection is disabled. Most human visitors may be verified without an explicit interaction, depending on the challenge type and request signals. Read the Cloudflare challenge overview.

2. The most useful signals

Verification interstitial

A provider-branded page that says the browser is being verified is direct evidence that the request reached a challenge mechanism. Record the URL, status code, response headers, timestamp, and a copy of the body when you are authorized to diagnose the site. Branding should be treated as evidence only when it is consistent with the response and domain you contacted.

Cloudflare’s `cf-mitigated` header

For a Cloudflare response, inspect:

cf-mitigated: challenge

Cloudflare documents this value as the marker for a Challenge Page. It is provider-specific, so do not expect another vendor to emit the same header.

Unexpected HTML for a non-HTML request

If an API call, image request, or XHR receives an HTML document, compare the Content-Type and body with the resource you expected. A Cloudflare challenge response is documented as text/html, even when the original request targeted JSON or another type. The mismatch supports the conclusion that the response was intercepted.

JavaScript detection and cookies

A page can include a browser-side detection script or set a session cookie used by a protection system. Cloudflare says JavaScript Detections is one input among several. A missing signal may have legitimate causes such as disabled JavaScript, an extension, a failed script, or a network problem. A script or cookie alone does not prove that the site blocked you.

Status codes without fingerprints

A 403, 429, timeout, empty body, or connection reset shows that access failed or was limited. None identifies a particular anti-bot vendor by itself. Check headers, content type, body, and repeated observations before drawing a conclusion.

3. A responsible inspection workflow

Load the page normally. Note whether the expected content appears, a verification page appears, or navigation pauses while browser checks run.
Capture one response. Record status, headers, content type, final URL, and a small body sample. Do this only for pages and systems you are authorized to inspect.
Compare expected and actual content. An HTML challenge document returned from an API or asset endpoint is a useful interception clue.
Check session behavior. Look for cookies, injected scripts, or a second request that completes navigation after browser-side processing.
Repeat carefully. Compare a normal browser navigation with your authorized diagnostic request. Keep the URL, method, headers, and timestamp consistent.
Stop at observation. Document the evidence instead of attempting to bypass the site’s controls.

4. Inspect headers with cURL

Use -D - to print response headers and -o to save the body:

curl -sS -D - -o response.html https://example.com/

To display only the fields most useful for detection:

curl -sS -D - -o response.html https://example.com/ \
  | tr -d '\r' \
  | grep -Ei '^(HTTP/|content-type:|content-length:|location:|server:|cf-mitigated:|set-cookie:)'

Open response.html and compare it with the page you expected. A Cloudflare challenge is strongly indicated when the headers include cf-mitigated: challenge and the body is an interstitial HTML document. Do not classify every HTML response as a challenge.

5. Inspect a response with Python

This script prints the final URL, status, content type, relevant headers, and a body preview:

import requests

url = "https://example.com/"
r = requests.get(url, timeout=30, allow_redirects=True)

print("status:", r.status_code)
print("final_url:", r.url)
print("content_type:", r.headers.get("content-type"))

for name in ("cf-mitigated", "server", "location", "set-cookie"):
    value = r.headers.get(name)
    if value:
        print(f"{name}: {value}")

print("body_preview:")
print(r.text[:500])

Interpret the result as evidence about that request. A cf-mitigated value of challenge is a documented Cloudflare marker. An HTML body by itself is only a clue until its content and headers support the conclusion.

6. Inspect a response with Node.js

Node.js 18 or newer includes fetch:

const url = 'https://example.com/';
const res = await fetch(url, { redirect: 'follow' });

console.log('status:', res.status);
console.log('final_url:', res.url);
console.log('content_type:', res.headers.get('content-type'));

for (const name of ['cf-mitigated', 'server', 'location', 'set-cookie']) {
  const value = res.headers.get(name);
  if (value) console.log(`${name}: ${value}`);
}

const body = await res.text();
console.log('body_preview:\n', body.slice(0, 500));

If the request expects JSON, explicitly check the content type before parsing:

const type = res.headers.get('content-type') || '';
if (!type.includes('application/json')) {
  throw new Error(`Expected JSON, received ${type || 'unknown content type'}`);
}
const data = JSON.parse(await res.text());

7. How to interpret the evidence

Observation	Supports	Does not establish
Provider-branded verification interstitial	The request is being challenged by that provider’s mechanism	That every route or request is protected
`cf-mitigated: challenge`	Cloudflare Challenge Page for that response	That other vendors use the header or are absent
Non-HTML request returns challenge HTML	The original response may have been intercepted	That every HTML response is a challenge
JavaScript detection script or session cookie	A browser-side signal may be deployed	That a block occurred or a bot decision was made
403, 429, timeout, or empty page alone	Access failed or was limited	Which system caused it
No visible challenge	Nothing conclusive	That protection is absent

Cloudflare’s Bot Score is a product-specific value from 1 to 99. Its documented groupings are 1 for automated, 2–29 for likely automated, and 30–99 for likely human, with a separate verified-bot category. These labels apply to Cloudflare’s products and are not a universal probability scale. See Cloudflare’s Bot Score documentation.

8. If you own the website

Public responses cannot reveal your complete protection configuration. Check the security provider’s event logs, bot analytics, WAF rules, and custom rules. Cloudflare documents bot-related fields for custom rules and notes that score availability depends on the plan; granular scores require Enterprise Bot Management. Logs can show the matched rule, action, request path, and reason far more reliably than a visitor’s browser can.

Keep an observation record containing:

UTC timestamp and request URL
HTTP method and status
Response headers and content type
Redirect chain and final URL
Whether JavaScript and cookies were enabled
Body hash or a short sanitized excerpt
Relevant provider event or WAF log ID

9. Troubleshooting common misdiagnoses

Symptom	Likely cause	What to do
403 with no provider header	WAF rule, authorization failure, IP policy, or anti-bot control	Inspect the body and owner logs; do not name a vendor from status alone.
429 responses	Rate limit, quota, or automated-traffic rule	Check `Retry-After`, request frequency, and service documentation.
JSON parser fails on a 200 response	An HTML challenge or error page was returned	Check `Content-Type` and save the body before parsing.
Browser works but script fails	Different headers, cookies, JavaScript execution, or browser signals	Compare authorized requests and inspect redirects and cookies.
First request lacks a JavaScript signal	The signal is injected or completed after the initial HTML response	Observe the follow-up browser requests; treat the first response as incomplete evidence.
Blank page or timeout	Network issue, origin failure, rendering problem, or protection	Check DNS, TLS, server timing, and corroborating headers before concluding it is anti-bot.
No CAPTCHA appears	Non-interactive or automatically solved challenge	Inspect headers, scripts, cookies, and navigation timing.

10. Performance, reliability, and cost notes

Performance: A challenge can add a browser round trip and JavaScript execution before the destination loads. Measure time to the final URL, not only time to first byte.
Reliability: Record the exact request context. Decisions can vary with session cookies, browser capabilities, headers, IP reputation, and route.
Detection quality: Combine response-layer evidence with rendered-page and session evidence. One ambiguous symptom is not enough.
Cost: Simple header inspection with cURL, Python, or Node.js has no service fee beyond your own compute and network usage. Browser automation may consume more CPU, memory, and bandwidth.
Authorization: Limit tests to systems you own or are permitted to diagnose. Recognition and documentation are different from bypassing controls.

11. Or skip the browser setup

If your goal is a dependable screenshot while documenting how a page behaves, ScreenshotNeo provides a single request to capture a URL as PNG, JPEG, WebP, or PDF. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and responses identify the result with X-Page-Verdict and X-Billed headers.

cURL:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webp

Python:

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://example.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://example.com' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

See the ScreenshotNeo API documentation for request options. The service also offers an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. It includes 1,000 screenshots per month free with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

12. FAQ

Can I prove a site has anti-bot protection from a 403?

No. A 403 can come from authorization, IP policy, a WAF rule, or anti-bot logic. Look for a challenge body, provider marker, or owner-side event log.

Does a CAPTCHA have to appear?

No. Challenges can run JavaScript automatically or use request signals without asking for visible interaction.

Is `cf-mitigated: challenge` a universal header?

No. It is Cloudflare’s documented marker for a Cloudflare Challenge Page.

Can a normal browser page still be protected?

Yes. A challenge may verify a likely human automatically, so a normal-looking page does not show that protection is absent.

What is the safest next step when I am blocked?

Document the response and contact the site owner or administrator. Do not attempt to evade controls without explicit authorization.