How to Debug Headless Chrome Access Denied Errors with Selenium Python
Find whether Selenium, Chrome, your network, or a WAF caused Access Denied, then fix it with evidence-driven debugging.
When Selenium Python shows an “Access Denied” page, Chrome may have started normally. The denial can come from the target application, a WAF or CDN, an authentication gateway, a corporate proxy, or an egress policy. Treat it as a response-layer problem until captured evidence proves that Chrome failed to start.
This guide gives you a repeatable workflow: prove the browser started, capture the denial, compare headed and headless sessions, check network identity, and handle intentional blocking through supported access paths.
1. Start with a minimal, observable Selenium script
Use Selenium 4’s current Chrome options API. The old options.headless = True property was removed. Chrome and ChromeDriver should have matching major versions.
from pathlib import Path
import json
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
URL = "https://example.com"
options = Options()
options.add_argument("--headless=new")
options.add_argument("--window-size=1365,900")
# Keep the browser output so startup failures are distinguishable from HTTP denials.
options.add_argument("--enable-logging")
options.add_argument("--v=1")
driver = webdriver.Chrome(options=options)
try:
driver.get(URL)
time.sleep(2)
print("capabilities:")
print(json.dumps(driver.capabilities, indent=2, default=str))
print("current_url:", driver.current_url)
print("title:", driver.title)
print("user_agent:", driver.execute_script("return navigator.userAgent"))
print("language:", driver.execute_script("return navigator.language"))
print("viewport:", driver.execute_script("return [innerWidth, innerHeight, devicePixelRatio]"))
Path("denial.html").write_text(driver.page_source, encoding="utf-8")
driver.save_screenshot("denial.png")
print("saved denial.html and denial.png")
finally:
driver.quit()A SessionNotCreatedException, missing binary error, or driver startup failure is a browser setup problem. An HTML document titled “Access Denied” means Chrome navigated and received content from some response layer.
2. Verify Chrome and ChromeDriver before changing flags
- Print the browser and driver versions from
driver.capabilities. - Confirm the Chrome and ChromeDriver major versions match. Selenium documents this as a requirement.
- Use Selenium Manager if you want Selenium to resolve a missing driver automatically.
- Pin both browser and driver versions in CI when reproducibility matters.
- Confirm the executable is available to the same user and container that runs the script.
Do not diagnose a WAF while the browser cannot create a session. Fix startup first, then capture the actual page returned by the site.
3. Capture the denial itself
Save enough evidence to identify who produced the page and what changed between runs.
- Final URL and every redirect location.
- Page title, complete page source, and a screenshot.
- Cookies before and after navigation.
- User-agent and client-hint values.
- Viewport size, device pixel ratio, locale, and timezone.
- Proxy configuration, DNS resolver, TLS interception status, and outbound IP.
- Navigation start time, redirect timing, and time to the denial.
- Browser console messages and JavaScript exceptions.
Selenium navigation does not guarantee a direct HTTP status API. If you need status codes, response headers, or a complete redirect chain, add a network-level capture layer such as your proxy, browser protocol logging, or an external request capture. Search the saved body for CDN challenge markers, login redirects, rate-limit messages, and corporate gateway banners.
Record cookies and browser-visible signals
cookies = driver.get_cookies()
print("cookies:", cookies)
signals = driver.execute_script("""
return {
userAgent: navigator.userAgent,
language: navigator.language,
languages: navigator.languages,
platform: navigator.platform,
webdriver: navigator.webdriver,
viewport: [innerWidth, innerHeight],
pixelRatio: devicePixelRatio,
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone
}
""")
print("signals:", json.dumps(signals, indent=2))These values are evidence, not a promise that changing them will bypass a control.
4. Compare headed and unified headless Chrome
Chrome now uses unified Headless and headful modes. Since Chrome 132, the old headless implementation is available only as the separate chrome-headless-shell binary. Use --headless=new for current Chrome.
Run two sessions with the same:
- Chrome build and ChromeDriver.
- URL, account, cookies, and authentication state.
- Proxy and outbound network.
- Locale, timezone, viewport, and timing.
- Navigation sequence and wait conditions.
Change only headless versus headed mode. Then compare:
| Signal | What to inspect | Why it matters |
|---|---|---|
| Headers | User-agent and client hints | A 2026 study reported that header-level signals accounted for 75% of Chromium-headless-only blocks in its experiment. |
| JavaScript | Visible properties, navigator.webdriver, language, and platform |
Application or WAF rules may branch on browser-visible values. |
| Rendering | Viewport, device pixel ratio, WebGL and GPU behavior | Responsive policies and browser checks can produce different responses. |
| Timing | Startup, redirects, challenge completion, and request intervals | Different timing can trigger rate limits or challenge flows. |
5. Check network identity and policy
A local headed run does not prove that a container, CI runner, or remote Selenium node has the same identity.
- Confirm the effective outbound IP from the failing host.
- Check whether an HTTP or SOCKS proxy is configured for Chrome and for the surrounding environment.
- Verify DNS answers from the failing machine.
- Look for corporate TLS interception or an authentication gateway.
- Check allowlists, account permissions, rate limits, and regional policy.
- Compare local and CI runs using the same account and URL.
Selenium’s documentation discusses remote sessions in complex network topologies and under strict corporate restrictions. A remote node can have different proxy settings, egress IP reputation, DNS, and policy even when your Python code is identical.
6. Authentication and intentional blocking
If the page requires login, complete the supported authentication flow and preserve the resulting session state. Capture the redirect to the login or consent page instead of assuming it is a bot block.
If a WAF or provider intentionally blocks automation, request an allowlist or use the provider’s official API. Respect the site’s terms, robots directives, rate limits, and access policy. Disabling navigator.webdriver, spoofing headers, rotating proxies, or solving CAPTCHAs is not a reliable or automatically permitted fix.
7. Common errors and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
SessionNotCreatedException |
Chrome and driver mismatch or incompatible startup configuration | Match major versions, inspect capabilities, and let Selenium Manager resolve the driver or pin compatible versions. |
| Binary location error | Chrome is absent or unavailable to the process user | Install Chrome in the runtime image and verify its executable path and permissions. |
| HTML says “Access Denied” | Target, WAF, gateway, proxy, or egress policy returned a denial document | Save source, screenshot, final URL, cookies, headers, and network evidence before changing options. |
| Headed succeeds, headless fails | Different headers, client hints, JavaScript signals, viewport, timing, or network identity | Run controlled comparisons and inspect each signal. Start with headers because the 2026 study found them responsible for 75% of blocks in its experiment. |
| Local succeeds, CI fails | Different outbound IP, proxy, DNS, TLS interception, or allowlist | Compare egress and policy from both hosts; ask the site owner to allowlist the CI identity when appropriate. |
| Blank page or timeout | Resource failure, navigation timing, blocked dependency, or site instability | Capture console output and timings, wait for a meaningful selector, and inspect network-level logs. |
| Repeated 403 or rate-limit page | Account or IP rate limit | Reduce request rate, follow the provider’s limits, authenticate correctly, or use an official API. |
8. A controlled diagnostic matrix
Change one variable at a time and keep a record of the result.
- Baseline: headed Chrome, local host, known account.
- Headless: same host and account, only
--headless=newchanges. - CI: same browser versions and script on the failing runner.
- Network: same session through the intended proxy or remote node.
- Timing: same waits and navigation sequence.
For each row, record whether Chrome started, the final URL, title, body markers, cookies, screenshot, and any available status and headers. This separates browser startup, rendering differences, response-layer denials, and network identity problems.
9. Performance, reliability, and cost considerations
- Use a deterministic viewport so responsive rules do not obscure comparisons.
- Reuse a browser session only when the site’s authentication and isolation requirements allow it; otherwise create clean sessions to avoid stale cookies.
- Keep browser and driver versions pinned in CI, and update them deliberately.
- Use explicit waits for a selector or documented state instead of arbitrary long sleeps, while retaining enough delay to observe challenge redirects.
- Separate browser logs from network captures so a failed navigation is not mistaken for a server response.
- Record outbound identity and proxy settings with every run; these often explain environment-specific failures.
There is no universal Chrome flag that defeats a WAF. The reliable path is evidence, controlled comparison, and a supported access policy.
10. Or skip the browser setup
ScreenshotNeo provides a website screenshot API and MCP server. One GET request returns a PNG, JPEG, WebP, or PDF. The API accepts the URL and access key directly:
Read the ScreenshotNeo API documentation for the complete option list.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);Cookie banners, popups, and chat widgets are removed before the shot. Bot checks, blank pages, and failed loads are never billed. Response headers identify the page verdict and whether the request was billed. An MCP server lets AI agents take screenshots with take_screenshot, inspect pages with get_page_info, and create PDFs with capture_pdf. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000.
Create a free ScreenshotNeo account.
FAQ
Does headless mode itself cause every 403?
No. A 403 can come from the target, WAF, authentication gateway, proxy, or egress policy. Compare evidence from headed and headless sessions.
Should I use the old options.headless = True setting?
No. Use webdriver.ChromeOptions(), add --headless=new, and pass the options to webdriver.Chrome.
Can Selenium tell me the HTTP status directly?
Not reliably through page navigation alone. Use browser protocol logging, a proxy, or another network capture layer when status codes and headers are required.
What should I compare first when only headless is blocked?
Capture user-agent and client-hint headers first, then compare JavaScript-visible properties, viewport, locale, timezone, rendering signals, timing, and network identity.
Is changing navigator.webdriver a supported fix?
No. It does not guarantee access and may violate the site’s policy. Use an allowlist or official API when automation is intentionally restricted.
