How to Fix Pyppeteer Click and Navigation Wait Issues

Most Pyppeteer click and navigation failures come from waiting for the wrong event or starting the wait too late. For a click that causes a document navigation, register waitForNavigation() and click() together with asyncio.gather(). For a click that only changes the current document, wait for the resulting selector or application state instead.

The key pattern is:

await asyncio.gather(
    page.waitForNavigation({'waitUntil': 'domcontentloaded'}),
    page.click('a.my-link'),
)

Pyppeteer documents this arrangement because a separate navigation wait can race with a navigation-triggering click. The API reference warns that the race can produce unexpected results (Pyppeteer API reference).

1. Identify what the click actually does

Before changing a timeout, classify the transition:

What happens after the click?	What to wait for
A new document loads or the page reloads	waitForNavigation() started before click()
The URL changes through the History API	waitForNavigation(), then a page-specific readiness check
A hash changes within the same document	Usually a URL or DOM condition; navigation may return None
A panel, table or modal changes without navigation	waitForSelector() or waitForFunction()
Content appears after an asynchronous request	Wait for the content or a meaningful application condition, not generic network idle

A click alone does not prove that navigation will happen. A single-page application may update the DOM, use pushState, or change a hash without loading a new document.

2. Use the race-free pattern for real navigation

Start the navigation wait in the same awaitable operation as the click. The wait is created before the click can trigger the transition:

import asyncio
from pyppeteer import launch

async def main():
    browser = await launch(headless=True)
    page = await browser.newPage()
    try:
        await page.goto('https://example.com', {'waitUntil': 'domcontentloaded'})

        await asyncio.gather(
            page.waitForNavigation({'waitUntil': 'domcontentloaded'}),
            page.click('a.my-link'),
        )

        print('Destination:', page.url)
        print('Title:', await page.title())
    finally:
        await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Use the selector that the page actually renders. If the click opens a new tab or window, this pattern is not sufficient; listen for the target and operate on the new page separately.

Choose waitUntil deliberately

Value	Meaning	Use it when
domcontentloaded	The HTML has been parsed.	You need the DOM quickly and will wait for a specific element afterward.
load	The page load event has fired. This is the documented default.	Resources required by the page’s load event should be complete.
networkidle0	No more than zero active connections for 500 ms.	The page is expected to become completely quiet.
networkidle2	No more than two active connections for 500 ms.	A small amount of background traffic is normal.

The network-idle choices are not universal “page ready” signals. Analytics, polling, streaming, advertisements and other background requests can keep a page from reaching an idle threshold. If your next step needs a visible component, wait for that component.

await asyncio.gather(
    page.waitForNavigation({
        'waitUntil': ['domcontentloaded', 'networkidle2'],
        'timeout': 45000,
    }),
    page.click('a.checkout'),
)
await page.waitForSelector('[data-testid="checkout-form"]', {
    'visible': True,
    'timeout': 15000,
})

3. Wait for the result when there is no navigation

For tabs, filters, accordions, modal dialogs and client-side searches, do not call waitForNavigation(). Wait for the state your code needs:

await page.click('button.show-results')
await page.waitForSelector('.results', {
    'visible': True,
    'timeout': 10000,
})

You can wait for an application-specific condition with waitForFunction():

await page.click('button.load-more')
await page.waitForFunction(
    "document.querySelectorAll('.result').length >= 20",
    {'timeout': 15000},
)

Prefer a condition that represents usable output: a result count, a status value, an enabled button or a visible component. Avoid waiting for an arbitrary delay unless the site has no observable readiness signal.

Waiting for a URL or History API update

History API URL changes count as navigation in Pyppeteer. You can combine the navigation wait with a URL assertion:

await asyncio.gather(
    page.waitForNavigation({'waitUntil': 'domcontentloaded'}),
    page.click('a.account'),
)
assert '/account' in page.url
await page.waitForSelector('main.account', {'visible': True})

For a hash-only change, inspect the URL after the click or wait for the section selected by the hash:

await page.click('a[href="#details"]')
await page.waitForFunction(
    "window.location.hash === '#details'",
    {'timeout': 5000},
)
await page.waitForSelector('#details', {'visible': True})

4. A complete diagnostic script

This script logs the URL, captures the failure class and separates document navigation from an in-page result:

import asyncio
from pyppeteer import launch
from pyppeteer.errors import TimeoutError

async def click_and_wait_for_navigation(page, selector):
    try:
        await asyncio.gather(
            page.waitForNavigation({
                'waitUntil': 'domcontentloaded',
                'timeout': 30000,
            }),
            page.click(selector),
        )
    except TimeoutError:
        raise RuntimeError(
            f'No completed navigation after clicking {selector!r}; '
            f'current URL is {page.url}'
        )

async def main():
    browser = await launch(headless=True)
    page = await browser.newPage()
    page.setDefaultNavigationTimeout(30000)
    try:
        await page.goto('https://example.com', {
            'waitUntil': 'domcontentloaded',
            'timeout': 30000,
        })

        # Use this only when the click causes a document navigation.
        await click_and_wait_for_navigation(page, 'a.my-link')
        await page.waitForSelector('main', {
            'visible': True,
            'timeout': 10000,
        })
        print({'url': page.url, 'title': await page.title()})
    finally:
        await browser.close()

asyncio.get_event_loop().run_until_complete(main())

Replace https://example.com, a.my-link and main with selectors from the target site. The first Pyppeteer run downloads Chromium; the project documentation describes using pyppeteer-install to install it before running scripts (Pyppeteer documentation).

5. Understand timeout settings

Pyppeteer navigation methods use a documented 30-second default timeout. You can change a single call:

await page.goto(url, {'timeout': 60000, 'waitUntil': 'domcontentloaded'})
await page.waitForNavigation({'timeout': 60000, 'waitUntil': 'load'})

Or set the default navigation timeout for the page:

page.setDefaultNavigationTimeout(60000)

A timeout of 0 disables the timeout:

page.setDefaultNavigationTimeout(0)

Only increase the timeout when the expected event is correct and the site is genuinely slow. A longer timeout cannot make a DOM-only click emit a navigation event, and it cannot fix a selector that never clicked.

6. Troubleshooting common failures

“Navigation Timeout Exceeded” after a click

• Cause: The click updates the current page instead of navigating.
• Fix: Remove waitForNavigation() and wait for a result selector or function.

The click succeeds but the destination is missed

• Cause: The script called click() before creating the navigation wait.
• Fix: Use asyncio.gather(page.waitForNavigation(...), page.click(...)).

networkidle0 never completes

• Cause: Persistent polling, analytics, streaming or other background requests keep connections open.
• Fix: Use domcontentloaded or load, then wait for the specific element required by the next operation. Try networkidle2 only when limited background traffic is acceptable.

The selector wait times out

• Cause: The selector is wrong, the element is inside an iframe or shadow DOM, or the element exists but is not visible.
• Fix: Confirm the selector in the page, switch to the correct frame, and decide whether you need presence or visibility.

Invalid URL or SSL error

• Cause: The URL is malformed, unreachable or has a certificate problem.
• Fix: Log the exact URL, test it in the same environment, and inspect the Pyppeteer exception. Do not hide certificate problems by default; address the certificate or use an explicitly approved environment setting.

Main resource failed

• Cause: The server returned an unusable response or the browser could not load the main document.
• Fix: Check DNS, redirects, authentication, proxy configuration and the response status. A timeout increase does not repair a failed main resource.

Chromium is missing

• Cause: Pyppeteer has not downloaded its browser binary in the current environment.
• Fix: Run pyppeteer-install as documented, or configure an existing Chromium executable explicitly.

The page changed URL but waitForNavigation() returned None

• Cause: The transition may be a same-document hash change.
• Fix: Wait for the hash, URL value or destination element directly.

7. A decision checklist

1. Confirm the click target exists and is clickable.
2. Determine whether the action causes a document navigation, History API update, hash change or DOM-only update.
3. For document navigation, create waitForNavigation() before the click with asyncio.gather().
4. Choose domcontentloaded, load, networkidle0 or networkidle2 based on the next operation.
5. Wait for a meaningful selector or function condition after navigation.
6. Inspect the exact exception for SSL, invalid URL, timeout or main-resource failure.
7. Increase the timeout only when the event and selector assumptions are already correct.
8. Check the installed Pyppeteer version against the API documentation.

8. Performance, reliability and maintenance

Use the narrowest readiness condition that satisfies the task. Waiting for domcontentloaded followed by one result selector is usually more predictable than waiting indefinitely for network idle on an application with background traffic. Reuse one browser process where appropriate, create pages per task, and always close pages and the browser in a finally block.

Keep navigation and result waits separate in your code so a failure identifies the missing state. Log the URL, selector, chosen waitUntil value and timeout. For intermittent failures, capture a screenshot and HTML dump at the point of failure and retry only errors that are safe to retry.

Pyppeteer’s repository currently says the project is unmaintained and recommends considering Playwright Python (Pyppeteer repository). Playwright has different APIs and behavior, so treat migration as a project decision rather than assuming a drop-in rewrite. Its current guidance emphasizes locator auto-waiting and web assertions; verify the migration against your own selectors and workflows.

Or skip the browser setup

If your goal is to obtain a clean screenshot after a page has settled, ScreenshotNeo provides a single HTTP request instead of maintaining Chromium and navigation-wait code. See the ScreenshotNeo API documentation for all options.

cURL

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups and chat widgets. Bot checks, blank pages, failed loads, timeouts and cache hits are not billed; response headers identify the page verdict and whether the shot was billed. Its MCP server includes take_screenshot, get_page_info and capture_pdf tools for Claude, Cursor and other MCP clients. The free plan includes 1,000 screenshots each month with no card, and paid plans start at $5 for 3,000 shots.

Create a free ScreenshotNeo account and start with 1,000 screenshots a month at no charge.

FAQ

Should I always use networkidle0 for screenshots?

No. Pages with polling or analytics may never become idle. Use the earliest reliable state and wait for the element that proves the page is ready.

Can I call waitForNavigation() after click()?

You can, but it risks missing a fast navigation. Create both operations together with asyncio.gather().

Does a History API route change count as navigation?

Pyppeteer treats History API URL changes as navigation. A same-document hash change may instead complete without a normal navigation response.

Will disabling the timeout fix a hanging script?

No. It only removes the deadline. First verify that the click really navigates and that your readiness condition can occur.

When should I move from Pyppeteer to Playwright?

The Pyppeteer repository identifies the project as unmaintained and suggests considering Playwright Python. Evaluate API differences, selectors and deployment behavior before migrating.