How to Detect Anti-Bot Blocking in Browser Automation

Detect anti-bot blocking by comparing an automated session with a known-good interactive browser while recording the redirect chain, final URL, status code, response headers and body markers, cookies, JavaScript execution, console errors, timing, and a screenshot or saved HTML. A challenge page, CAPTCHA or Turnstile widget, bot-specific cookie, missing application content, injected detection script, or a difference that consistently follows an automation variable is strong evidence. No single status code proves that a site detected your bot.

What anti-bot blocking looks like

Anti-bot systems combine network, browser, session, and behavior signals. Cloudflare documents heuristics, headers, session characteristics, browser signals, JavaScript detections, machine learning, and behavioral analysis rather than one universal test. Enforcement can produce several outcomes:

Outcome	Typical evidence	What it means
Hard block	403, 401, 429, or a provider-specific error page	The request was rejected at a network or policy layer. The code alone is not proof of bot detection.
Challenge or interstitial	Redirect to a challenge path, “verify you are human” text, CAPTCHA, Turnstile, or a browser-check page	The site wants JavaScript, interaction, or a valid session before serving the application.
Altered content	HTTP 200 but no expected data, an HTML shell, consent wall, or security message	The request succeeded technically while the application response was replaced or degraded.
Redirect loop	Repeated redirects between the target and a challenge endpoint	Cookies, JavaScript, IP reputation, or session state may not satisfy the challenge.
Soft degradation	Missing API data, slower responses, empty lists, or different markup	The site may be scoring or throttling automation without showing a visible block.

A successful load does not prove human treatment. Cloudflare’s Browser Run documentation states that requests from Browser Run are always identified as bot traffic. Treat the page content and the comparison evidence as more informative than a 200 response.

Build a reliable baseline

1. Open the same URL in a normal interactive browser.
2. Use the same account state, geography, approximate time window, and network when possible.
3. Save the final URL, redirect chain, status, headers, cookies, title, HTML, screenshot, console output, and network failures.
4. Run the automated session with the same URL and collect the same fields.
5. Repeat both runs. A one-off timeout is weak evidence; a stable difference tied to an automation variable is stronger.

Change one variable at a time: User-Agent, headless mode, IP or proxy, geography, session freshness, request rate, navigation order, and JavaScript availability. Do not conclude that a selector is wrong until you have inspected the response that created the page.

Capture evidence with Playwright (Node.js)

The following script records the signals needed to distinguish a selector failure from a challenge response. Save it as diagnose.js and run node diagnose.js https://example.com.

const { chromium } = require('playwright');
const fs = require('fs');

(async () => {
  const target = process.argv[2];
  if (!target) throw new Error('Usage: node diagnose.js https://example.com');

  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage({
    serviceWorkers: 'block',
    viewport: { width: 1440, height: 900 }
  });

  const responses = [];
  const failures = [];
  const consoleMessages = [];
  page.on('response', response => {
    if (responses.length < 300) {
      responses.push({
        url: response.url(),
        status: response.status(),
        headers: response.headers()
      });
    }
  });
  page.on('requestfailed', request => failures.push({
    url: request.url(),
    method: request.method(),
    error: request.failure()?.errorText
  }));
  page.on('console', message => consoleMessages.push({
    type: message.type(), text: message.text()
  }));

  const started = Date.now();
  const response = await page.goto(target, { waitUntil: 'domcontentloaded', timeout: 45000 });
  await page.waitForTimeout(3000);

  const result = {
    requestedUrl: target,
    finalUrl: page.url(),
    status: response?.status() ?? null,
    responseHeaders: response ? await response.allHeaders() : {},
    title: await page.title(),
    elapsedMs: Date.now() - started,
    cookies: await page.context().cookies(),
    userAgent: await page.evaluate(() => navigator.userAgent),
    webdriver: await page.evaluate(() => navigator.webdriver),
    bodyText: (await page.locator('body').innerText().catch(() => '')).slice(0, 12000),
    challengeMarkers: await page.evaluate(() => {
      const text = document.body?.innerText?.toLowerCase() || '';
      const selectors = [
        'iframe[src*="challenges.cloudflare.com"]',
        '[name="cf-turnstile-response"]',
        '[data-sitekey]',
        'input[name="cf-turnstile-response"]',
        '#challenge-running',
        '#challenge-stage'
      ];
      return {
        text: ['verify you are human', 'checking your browser', 'captcha', 'turnstile', 'attention required', 'access denied']
          .filter(marker => text.includes(marker)),
        selectors: selectors.filter(selector => document.querySelector(selector))
      };
    }),
    responses,
    failures,
    consoleMessages
  };

  fs.writeFileSync('automation-evidence.json', JSON.stringify(result, null, 2));
  fs.writeFileSync('automation-page.html', await page.content());
  await page.screenshot({ path: 'automation-page.png', fullPage: true });
  console.log(JSON.stringify({
    finalUrl: result.finalUrl,
    status: result.status,
    title: result.title,
    elapsedMs: result.elapsedMs,
    challengeMarkers: result.challengeMarkers,
    failedRequests: failures.length
  }, null, 2));
  await browser.close();
})();

Inspect automation-page.html and automation-page.png before changing selectors. If the expected application markup is absent and challenge markers are present, the failure is upstream of your locator.

Capture the same evidence with Selenium (Python)

import json
import sys
import time
from pathlib import Path
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

url = sys.argv[1] if len(sys.argv) > 1 else "https://example.com"
options = Options()
options.add_argument("--headless=new")
options.add_argument("--window-size=1440,900")
driver = webdriver.Chrome(options=options)
started = time.time()
try:
    driver.get(url)
    time.sleep(3)
    body = driver.find_element("tag name", "body").text
    html = driver.page_source
    lower = body.lower()
    markers = [m for m in ["verify you are human", "checking your browser", "captcha", "turnstile", "attention required", "access denied"] if m in lower]
    evidence = {
        "requestedUrl": url,
        "finalUrl": driver.current_url,
        "title": driver.title,
        "elapsedMs": round((time.time() - started) * 1000),
        "userAgent": driver.execute_script("return navigator.userAgent"),
        "webdriver": driver.execute_script("return navigator.webdriver"),
        "challengeMarkers": markers,
        "bodyText": body[:12000],
        "cookies": driver.get_cookies(),
    }
    Path("selenium-evidence.json").write_text(json.dumps(evidence, indent=2))
    Path("selenium-page.html").write_text(html)
    driver.save_screenshot("selenium-page.png")
    print(json.dumps(evidence, indent=2))
finally:
    driver.quit()

Interpret the strongest signals

Challenge pages and widgets

Search the final DOM and saved HTML for interstitial text, CAPTCHA, Turnstile, challenge endpoints, or an iframe hosted by a challenge provider. A widget alone proves that a challenge was presented, not that it was caused by headless mode; repeat with the same account, network, and browser variables.

Redirects and cookies

Record every Location header and the final URL. Look for provider-specific cookies such as cf_*, but treat cookie names as clues rather than proof. A redirect loop usually means the challenge state was not accepted, JavaScript did not run, cookies were blocked, or the IP/session remains disfavored.

JavaScript and runtime differences

Check navigator.webdriver, JavaScript errors, blocked scripts, missing Web APIs, and whether the application’s XHR or fetch calls complete. Cloudflare injects an invisible JavaScript Detection snippet into HTML page responses, not AJAX responses, and refreshes that detection within a 15-minute lifespan. A page that works interactively but fails before its application requests run points to runtime or challenge handling rather than a selector.

User-Agent and request fingerprints

Compare User-Agent, client hints, headers, TLS or proxy context, IP, and geography. Cloudflare documents User-Agent blocking and states that a missing or empty User-Agent receives a bot score of 1 from its heuristics engine. A changed User-Agent can alter policy, but matching it does not make a session human.

Bot scores

Cloudflare’s bot score ranges from 1 to 99: 1 is classified as automated, 2–29 as likely automated, and 30–99 as likely human. Granular scores require Enterprise Bot Management. Your browser script cannot infer this score from a normal response; request it from the site owner’s logs or security dashboard.

Separate anti-bot blocking from ordinary automation bugs

Symptom	Likely selector or application bug	Likely anti-bot behavior
Expected element is missing	Wrong frame, timing, route, or changed markup; page still contains normal content	HTML replaced by challenge/interstitial or expected API calls never run
Timeout	Slow resource, incorrect wait condition, or blocked dependency	Repeatable timeout only in automation, with challenge requests or redirect loops
HTTP 403/429	Authentication or application authorization issue	Response body contains provider challenge or access-denied markers and varies by automation variables
Works headed, fails headless	Viewport, rendering, or timing difference	Stable challenge page or different cookies/content in headless mode

Save the response body before attempting retries. Retrying a selector against a challenge document only obscures the original cause.

Find which layer is responsible

1. Network and HTTP: compare status, redirects, headers, DNS, proxy, IP, TLS context, and response size.
2. Browser runtime: compare JavaScript execution, Web APIs, automation signals, console errors, and blocked resources.
3. Session state: compare cookies, account, CSRF tokens, IP reputation, geography, and session age.
4. Behavior: compare request rate, navigation sequence, input timing, concurrency, and repeated identical actions.

Attribute the result when possible to a WAF or rate-limit challenge, JavaScript Detection, Turnstile, a User-Agent rule, Bot Fight Mode, Under Attack Mode, DDoS protection, or an upstream network failure. If the provider is unknown or the behavior cannot be reproduced, report the result as uncertain.

What status code proves a bot block?

None. A 403 can mean authorization, a WAF rule, or a normal application policy. A 429 can mean rate limiting without bot detection. A 200 can contain a challenge page or empty shell. The proof is the combination of response metadata, body markers, session differences, and reproducibility. Ask the site owner for the security event or bot score when you control the protected site.

Troubleshooting checklist

Problem	Cause to check	Fix or next diagnostic
Only the automated run sees “verify you are human”	Automation fingerprint, IP reputation, or behavior score	Run the baseline comparison; change one variable and record the resulting HTML, cookies, and redirects.
Browser stays on a challenge URL	JavaScript disabled, cookies blocked, or challenge cannot complete	Check console errors, blocked scripts, cookie storage, and frame access. Do not loop retries blindly.
Normal page returns with no data	Application API calls failed or content was replaced	Inspect network failures and response bodies for XHR/fetch requests, then compare the interactive session.
Headless mode fails while headed mode works	Viewport, timing, runtime, or a policy keyed to automation	Compare screenshots, headers, JavaScript values, and challenge markers; test a headed run with identical timing.
Intermittent 429 responses	Rate limit or concurrency threshold	Reduce request rate and concurrency, add bounded backoff, and check whether the body identifies a provider challenge.
CAPTCHA appears after several pages	Behavioral or session reputation threshold	Record the navigation count and timing at the transition; ask the site owner for the applicable policy.
Screenshot is blank	Failed load, bot block, transparent page, or capture before rendering	Save HTML, final URL, console errors, and network failures; verify the document has expected content before capture.

Performance, reliability, and cost

• Use one diagnostic run with full evidence, then reduce logging for production. Saving every response body and screenshot increases disk and storage cost.
• Set explicit navigation and total-operation timeouts. A timeout without the final URL and partial HTML is difficult to classify.
• Limit concurrency and use bounded exponential backoff for confirmed rate limits. Unbounded retries can strengthen the signal that triggers blocking.
• Cache a known-good baseline and compare hashes or key markers before expensive downstream parsing.
• Keep geography, proxy, User-Agent, account, and browser version stable while diagnosing. Changing all of them at once destroys your control group.
• Do not treat a successful screenshot as proof that the content is complete. Check title, expected selectors, API responses, and challenge markers.

Or skip the browser setup

For a diagnostic screenshot without maintaining Playwright or Selenium infrastructure, ScreenshotNeo provides a GET endpoint that returns PNG, JPEG, WebP, or PDF. See the ScreenshotNeo API documentation for all options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and each response reports the result in X-Page-Verdict and X-Billed headers. Its MCP server lets Claude, Cursor, and other MCP clients call take_screenshot, get_page_info, and capture_pdf. The Free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

FAQ

Can a 200 response still be a bot block?

Yes. Providers can return a challenge, empty shell, or altered page with 200. Inspect the body, DOM, redirects, cookies, and expected application data.

Does navigator.webdriver prove blocking?

No. It is one browser signal. Its presence can correlate with different treatment, but only a controlled comparison shows whether the site enforced a policy.

Should I rotate proxies to confirm detection?

Use a stable baseline first. Changing proxy, geography, and session together makes attribution impossible. If you control the site, inspect its security events instead.

How long does Cloudflare JavaScript Detection remain valid?

The documented detection refreshes within a 15-minute lifespan. Treat that as a provider-specific behavior, not a universal browser rule.

What should I report to the site owner?

Include timestamps, URL, account and geography, final URL, redirect chain, status and headers, response body markers, cookies, User-Agent, console and network failures, screenshots, and a working interactive comparison.