ScreenshotNeo

BlogHow-to

How to Get Fully Loaded HTML from a Page with Puppeteer

Wait for the page state your scraper needs, then extract its rendered HTML with Puppeteer’s page.content() or the live DOM.

By the ScreenshotNeo team30 September 202610 min read

How to Get Fully Loaded HTML from a Page with Puppeteer

To get rendered HTML with Puppeteer, navigate to the page, wait for a condition that proves the content you need is ready, then call page.content(). For a whole document, this is the usual extraction method; it returns the page’s full HTML, including the DOCTYPE. There is no universal signal for “fully loaded”: network quietness can help, but a page-specific selector or application-ready condition is often more reliable.

This guide shows a runnable baseline, how to choose a wait condition, the difference between page.content() and document.documentElement.outerHTML, and how to handle common incomplete-render cases.

1. Install Puppeteer and extract rendered HTML

Use a current Node.js installation and install Puppeteer in your project. The standard package downloads a compatible Chrome for Testing browser during installation. If your environment supplies its own Chrome, Puppeteer also offers the puppeteer-core package, which does not download a browser; configure its executable path explicitly.

npm install puppeteer

Save this as get-html.mjs and run it with node get-html.mjs https://example.com. It waits for the DOM to be parsed, then for a representative content element to become visible. Change the selector to one that proves the content you intend to collect exists.

import puppeteer from 'puppeteer';

const url = process.argv[2];
if (!url) throw new Error('Usage: node get-html.mjs <url>');

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();

try {
  await page.goto(url, {
    waitUntil: 'domcontentloaded',
    timeout: 30_000,
  });

  await page.waitForSelector('main', {
    visible: true,
    timeout: 15_000,
  });

  const html = await page.content();
  process.stdout.write(html);
} finally {
  await page.close();
  await browser.close();
}

The finally block closes the page and browser even if navigation or the selector wait throws. In a service that reuses a browser for many jobs, close each page after the job and close the shared browser during shutdown.

2. Choose a readiness condition that matches the page

Navigation lifecycle events describe browser milestones, not whether the exact data your scraper needs has appeared. Pick the least expensive condition that proves the target state, and give it a timeout so a stuck page cannot wait forever.

Wait for the page-specific content state before serializing the DOM.
Wait for the page-specific content state before serializing the DOM.
Wait option What it indicates Good fit Limitation
commit The navigation response has started committing. Very early access before DOM work. Content may not be parsed yet.
domcontentloaded The initial HTML has been parsed. Pages where you will wait separately for data or UI. Client-side rendering may still be in progress.
load The page’s load event fired. Pages relying on ordinary load-event resources. Does not guarantee later JavaScript or API data is done.
networkidle0 No more than zero network connections for an idle period. Pages that settle their requests after rendering. Polling, analytics, or persistent connections may prevent it.
networkidle2 No more than two network connections for an idle period. Pages with a small amount of continuing background traffic. Quiet network does not prove the relevant component rendered.
Selector or predicate A DOM condition you specify became true. Application content has a stable element or state. You must choose a condition that actually represents readiness.

Puppeteer exposes navigation, network-idle, selector, and function waits as separate tools; combine them around the target site rather than relying on one universal wait. The official Page API reference documents these primitives, and the Chrome rendering example uses networkidle0 for rendering JavaScript pages.

Wait for a selector

A selector is usually a stronger readiness signal than an arbitrary delay. Use visible: true if a hidden placeholder with the same selector could appear before real content.

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#main-content', {
  visible: true,
  timeout: 15_000,
});
const html = await page.content();

Wait for an application flag or DOM predicate

If the app publishes a readiness flag, wait for it directly. You can also express a DOM condition such as a non-empty article body. The callback runs in the browser page context.

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForFunction(() => window.appReady === true, {
  timeout: 20_000,
});
const html = await page.content();

For a DOM condition, make it specific enough to reject a shell or loading placeholder:

await page.waitForFunction(() => {
  const article = document.querySelector('article');
  return article && article.textContent.trim().length > 200;
}, { timeout: 15_000 });

Wait for network quietness

Use waitForNetworkIdle() when network quietness is meaningful for the target. It waits at least the configured idle time. A common sequence is to navigate early, wait for the page-specific element, then allow a short quiet period for follow-up requests.

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('#main-content', { visible: true });
await page.waitForNetworkIdle({ idleTime: 500, timeout: 10_000 });
const html = await page.content();

Network idle is only a heuristic. A WebSocket or polling loop can keep traffic active indefinitely. Conversely, a page can go quiet before a delayed timer fetches the data you want. If network idle times out, switch to a selector, app flag, or known response condition rather than simply increasing the timeout without limit.

Wait for a known API response

When a page’s content comes from a known endpoint, you can wait for its response, then wait for the DOM update. Do not assume that receiving JSON means the framework has already rendered it.

const responsePromise = page.waitForResponse(response =>
  response.url().includes('/api/items') && response.status() === 200
);

await page.goto(url, { waitUntil: 'domcontentloaded' });
await responsePromise;
await page.waitForSelector('[data-loaded="true"]', { timeout: 10_000 });
const html = await page.content();

Set the response predicate to the site’s actual endpoint and expected status. A loose URL substring may match the wrong request when a page makes multiple similar calls.

3. Choose the right HTML extraction method

page.content() is the normal choice for the serialized whole document. Puppeteer describes it as “The full HTML contents of the page, including the DOCTYPE.” See the content() reference.

page.evaluate() runs a function in the page context and returns its result; if the function returns a Promise, Puppeteer waits for it to resolve. Use it when you specifically need the live document DOM or need to read values from the page:

const html = await page.evaluate(() =>
  document.documentElement.outerHTML
);

These approaches usually serialize the current document DOM, including changes made by client-side JavaScript. Neither is a copy of the original HTTP response: scripts may have changed the DOM, and the serialized output does not include runtime state such as JavaScript event listeners, canvas pixels, or values that exist only in memory.

For one section, extract only that element:

const articleHtml = await page.$eval(
  'article',
  element => element.outerHTML
);

The evaluate() reference and $eval() reference cover page-context evaluation and element serialization. A missing selector makes $eval() fail, so wait for it first when rendering is asynchronous.

4. Use a bounded production pattern

For a reusable extraction function, keep navigation and readiness timeouts distinct. The separate bounds identify which stage failed and avoid treating a partial render as a successful extraction.

async function getRenderedHtml(browser, url) {
  const page = await browser.newPage();
  try {
    await page.goto(url, {
      waitUntil: 'domcontentloaded',
      timeout: 30_000,
    });
    await page.waitForSelector('#main-content', {
      visible: true,
      timeout: 15_000,
    });
    await page.waitForNetworkIdle({
      idleTime: 500,
      timeout: 10_000,
    });
    return await page.content();
  } finally {
    await page.close();
  }
}

This pattern assumes the selector means the content is useful and that the site eventually becomes quiet. If either assumption is false, replace that wait with a condition tied to the page. At the caller, catch failures and record the URL, stage, and timeout; do not label the result complete when extraction never reached its readiness condition.

5. Handle common edge cases

Lazy-loaded content

Content may load only after scrolling into view. If the required section is below the fold, scroll it into view and then wait for its content marker. For pages that load more items as you scroll, repeat until the desired item count or end marker appears; there is no generic Puppeteer flag that proves every lazy component on every page has loaded.

await page.evaluate(() => {
  document.querySelector('#target-section')?.scrollIntoView();
});
await page.waitForSelector('#target-section .loaded-item', {
  visible: true,
  timeout: 10_000,
});

When a click or form submission triggers navigation, start waiting for navigation before performing the action to avoid missing a fast transition. Then wait for the resulting page’s content condition too.

await Promise.all([
  page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
  page.click('a.next-page'),
]);
await page.waitForSelector('#main-content', { visible: true });
const html = await page.content();

Some single-page apps change content without a full navigation. In that case, wait for the expected URL or DOM change with waitForFunction() instead of expecting a navigation event.

Frames and shadow DOM

page.content() serializes the main frame’s document. An embedded iframe has its own document; find its frame and evaluate within it if you need that HTML. Shadow-root contents also require querying the relevant shadow root rather than assuming they appear as ordinary descendants in the document serialization.

A page may show a consent overlay, sign-in screen, bot check, or access-denied page instead of the content you expected. First inspect the page title, URL, and a short text excerpt to identify the state. Only automate interactions you are authorized to perform, and do not mistake a challenge or denial page for the target document.

Saving output safely

For larger documents, write the result to a file rather than a terminal, which may truncate or mangle long output. In Node.js, use writeFile with UTF-8. Avoid logging HTML that may contain personal information, form values, or page-specific tokens.

6. Troubleshoot incomplete or failed captures

Symptom Likely cause Fix
HTML contains only a loading shell Navigation completed before client rendering. Wait for a page-specific selector, app-ready flag, or data response followed by a DOM condition.
TimeoutError in goto() The page did not reach the selected lifecycle event within the bound, or the server is slow. Try domcontentloaded and separately wait for the needed content. Check the URL and browser logs.
Selector wait times out Wrong selector, content never arrived, or it is in another frame. Inspect the rendered DOM and frames; verify the selector manually; confirm the page did not redirect or show an error.
networkidle0 never resolves Polling, analytics, streaming, or persistent requests keep activity alive. Use a selector or app condition. Consider networkidle2 only if it matches the page’s behavior.
Output is missing content that appears later A timer, delayed request, interaction, or lazy-load has not run. Wait for the exact content marker; scroll or perform the required action, then wait again.
Execution context was destroyed A navigation replaced the page context while evaluation was running. Wait for navigation to finish, then evaluate in the new page context.
Browser fails to launch in a container Missing system libraries, incompatible executable, or sandbox/container configuration. Use a supported Puppeteer browser install or configure the environment’s Chrome path and dependencies. Avoid disabling browser security controls without understanding the environment.
HTML is unexpectedly large The entire app shell, embedded data, or many repeated nodes were serialized. Extract a specific section with $eval(), or inspect only the fields needed.

7. Performance, reliability, and cost

Launching a browser for every URL adds startup work. For a batch process, reuse a browser and create a fresh page per task, while limiting simultaneous pages to the capacity of the machine and the target site. Close pages in finally blocks so failed jobs do not accumulate browser resources. Timeouts should reflect actual page behavior and be monitored by stage; increasing every timeout can hide a broken readiness rule.

Choose waits with intent. networkidle0 can delay extraction or wait forever on active sites, while a long fixed sleep wastes time on fast pages and still misses slow ones. A selector or explicit application condition tends to be both faster and more meaningful. If HTML is for data extraction, capture only the relevant element where possible to reduce memory, output size, and downstream parsing work.

Reliability also depends on the page being accessible from the machine running Chrome. Redirects, rate limits, network failures, consent steps, and bot challenges can change what gets returned. Check final URL and page content for expected markers before storing a result as complete. Puppeteer does not make a page’s rendering deterministic: content can vary by locale, time, cookies, geolocation, viewport, and account state.

Self-hosted Puppeteer has no per-screenshot API charge, but browser compute, memory, storage, network traffic, and maintenance still have costs. If the actual need is a visual screenshot rather than HTML, a hosted screenshot API can remove browser installation and capture infrastructure from the workflow.

8. Or skip the browser setup

If you need a screenshot of the rendered page rather than its HTML, ScreenshotNeo returns a PNG, JPEG, WebP, or PDF from one GET request. Its API documentation covers the request options.

A hosted screenshot can handle common overlays before capture.
A hosted screenshot can handle common overlays before capture.
curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

ScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server gives AI agents tools for screenshots, page information, and PDF capture. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. Every feature is available on every plan.

Sign up free for 1,000 screenshots a month, with no card.

9. Frequently asked questions

Does page.content() include the DOCTYPE?

Yes. Puppeteer documents page.content() as returning the full HTML contents of the page, including the DOCTYPE.

Does “fully loaded” mean all future content is present?

No. A page can continue fetching or rendering after any browser lifecycle event. Define completion by the content your job requires, such as a selector, application flag, or known data state.

Can Puppeteer return the original server HTML?

page.content() returns the current serialized document after browser-side changes. To inspect the original response body, capture the navigation response separately; that response will not include later client-rendered changes.

Should I use a fixed delay?

Only when the page offers no better signal and the delay is a deliberate fallback. A fixed delay cannot adapt to fast and slow runs, so pair it with a timeout and validate the content afterward.

Can I extract a single article instead of the whole page?

Yes. Wait for the article selector, then use page.$eval('article', el => el.outerHTML) to return that element’s serialized markup.