ScreenshotNeo

BlogHow-to

How to Load the Full HTML Content of a Page in Puppeteer

Use Puppeteer’s `page.content()` to read a page’s current HTML, including the DOCTYPE. Learn when to wait, how to handle dynamic pages, and how to troubleshoot incomplete markup.

By the ScreenshotNeo team30 September 202610 min read

How to Load the Full HTML Content of a Page in Puppeteer

To load the full HTML content of a page in Puppeteer, navigate to the page, wait until the content you need is ready, then call await page.content(). It returns a string containing the current document markup, including the DOCTYPE. For pages modified by JavaScript, the readiness wait is what determines whether that string includes the content you expect.

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch();
try {
  const page = await browser.newPage();
  await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
  await page.waitForSelector('main'); // Choose a signal that means your content is ready.
  const html = await page.content();
  console.log(html);
} finally {
  await browser.close();
}

This gets the browser’s current DOM serialization, not necessarily every resource, lazy-loaded image, future update, or hidden frame. Use a page-specific wait condition that matches the data you intend to extract.

1. What page.content() returns

Puppeteer’s Page.content() method returns a promise for the full HTML contents of the page, including the DOCTYPE. The result is a JavaScript string. Because Puppeteer reads the page after it has been rendered, the markup can include DOM changes made by JavaScript that ran before the call.

That does not mean it retrieves the original response body or freezes the page. It serializes the document state at the moment you call it. If an application inserts a section later, you need to wait for that section and then call page.content(). If content changes after extraction, the string you already saved does not update.

The term “full HTML” also has boundaries. page.content() captures the current document, not the contents of every iframe as part of one combined document, and it does not guarantee that images, stylesheets, or other referenced assets have been downloaded. It returns markup with references to resources.

2. Install Puppeteer and run a complete example

In a new Node.js project, install Puppeteer and save the following as get-html.mjs:

npm install puppeteer
import puppeteer from 'puppeteer';

const url = process.argv[2] ?? 'https://example.com';
const browser = await puppeteer.launch();

try {
  const page = await browser.newPage();
  await page.goto(url, {
    waitUntil: 'domcontentloaded',
    timeout: 30_000,
  });

  // Replace this with a selector that represents the content you need.
  await page.waitForSelector('body', { timeout: 10_000 });

  const html = await page.content();
  console.log(html);
} finally {
  await browser.close();
}

Run it with an optional URL:

node get-html.mjs https://example.com

The example uses domcontentloaded so it does not wait for every image and subresource before beginning the page-specific wait. For a real target, replace body with a selector that appears only when the information you need is present. If you do not close the browser in a finally block, exceptions can leave a browser process running.

3. Choose the right readiness condition

Navigation completing and application content being ready are different events. A page can finish its initial document navigation before a client-side app fetches and renders its data. Pick the wait condition based on the page’s behavior and the content you need.

Wait for the content your task depends on before serializing the current DOM.
Wait for the content your task depends on before serializing the current DOM.
Wait strategy Useful when Watch for
waitUntil: 'domcontentloaded' You want the document parsed, then will wait for a specific app condition. JavaScript may still be fetching and rendering the main content.
waitUntil: 'load' The page’s load event is a useful boundary for your task. It waits for dependent resources; that may be longer than needed.
waitForSelector() A particular element signals that the content you need was rendered. A generic selector such as body can exist before meaningful content.
waitForNetworkIdle() The site settles its network activity predictably. Polling, analytics, streaming, ads, or persistent connections can delay or prevent network quiet.
Wait for a response or app-specific condition A known API response or readiness flag is the strongest signal. Use a condition tied to the actual page behavior and allow a timeout.

For dynamic content, a selector is often a straightforward readiness check:

await page.goto('https://example.com/catalog', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('[data-testid="product-list"]', { timeout: 15_000 });
const html = await page.content();

To wait for a known response, register the wait before navigation or before the action that triggers the request. This avoids missing a fast response:

const responsePromise = page.waitForResponse(response =>
  response.url().includes('/api/catalog') && response.ok()
);
await page.goto('https://example.com/catalog', { waitUntil: 'domcontentloaded' });
await responsePromise;
await page.waitForSelector('[data-testid="product-list"]');
const html = await page.content();

A response arriving does not always mean its data has already been rendered into the DOM, so the example also waits for the rendered list.

When network activity becomes quiet reliably on a target, Puppeteer provides page.waitForNetworkIdle():

await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
await page.waitForNetworkIdle({ idleTime: 500, timeout: 10_000 });
const html = await page.content();

Network idle is not the same as application completeness. It waits at least for the configured idle time, but pages with background requests can make it a poor fit. Prefer a selector, response, or app-specific readiness condition when one is available. Avoid arbitrary sleeps as the default: they can be too short on a slow run and waste time on a fast one.

4. Handle navigation, actions, and delayed updates

If clicking a link or button triggers a navigation, wait for the navigation and click together. Otherwise, the code can race the transition and read the old document or encounter a navigation-related error.

await page.goto('https://example.com/start');
await Promise.all([
  page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
  page.click('a.next-page'),
]);
await page.waitForSelector('main .results');
const html = await page.content();

For a click that updates the current page without navigation, wait for the resulting content instead:

await page.click('button.load-more');
await page.waitForSelector('.result-card:nth-child(21)');
const html = await page.content();

Some sites lazy-load content only when it approaches the viewport or after scrolling. If that content is part of your task, trigger the site behavior, wait for the resulting elements, and extract afterward. Calling page.content() alone does not cause every lazy-loaded section to appear.

5. Save the HTML to a file

Use Node’s file system module to write the returned string. This example writes UTF-8 HTML and accepts the URL as a command-line argument:

import puppeteer from 'puppeteer';
import { writeFile } from 'node:fs/promises';

const url = process.argv[2] ?? 'https://example.com';
const browser = await puppeteer.launch();
try {
  const page = await browser.newPage();
  await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30_000 });
  await page.waitForSelector('main', { timeout: 15_000 });
  const html = await page.content();
  await writeFile('page.html', html, 'utf8');
} finally {
  await browser.close();
}

If you need one file per URL, derive a safe filename rather than using an arbitrary URL as a filesystem path. For large documents, consider whether retaining the string in memory is necessary; writing it immediately avoids keeping multiple page snapshots in memory at once.

6. page.content() versus outerHTML

For ordinary extraction, use page.content(): it is the documented Puppeteer method and includes the DOCTYPE. The browser-side alternative document.documentElement.outerHTML returns the markup for the document element. It can be useful when you need to perform custom processing in the page context, but it is less direct for “give me the full HTML document.”

const html = await page.evaluate(() => document.documentElement.outerHTML);

Choose outerHTML when the in-page evaluation itself is useful, such as returning a transformed fragment or combining markup with a value read from the page. Choose page.content() when you want the current full document serialization with DOCTYPE handling provided by Puppeteer.

7. Frames and what the result includes

A page can contain iframes, each with its own document. The main page’s HTML contains the iframe element and its attributes; do not assume that this string embeds the child frame’s rendered document. To extract a frame’s document, locate the frame and read its content separately:

The main page and an iframe have separate document contents to extract.
The main page and an iframe have separate document contents to extract.
const frame = page.frames().find(item => item.url().includes('/embedded/'));
if (!frame) throw new Error('Expected embedded frame was not found');
const frameHtml = await frame.content();

Frame URLs can change, and cross-origin restrictions may prevent page JavaScript from inspecting an iframe. Puppeteer’s frame APIs operate through the browser automation protocol, but you still need to identify the intended frame and wait for its content. For pages with multiple matching frames, use a more specific URL or another stable property rather than taking the first match blindly.

8. Troubleshooting incomplete or missing HTML

Symptom Likely cause Fix
HTML contains an empty app root The client-side app has not rendered yet. Wait for a meaningful content selector or known response, then verify the rendered element exists.
Content appears sometimes but not always A race with asynchronous fetches or rendering. Replace fixed delays with a page-specific condition and set a realistic timeout.
Network-idle wait times out Persistent connections, polling, analytics, or other background traffic. Wait for the specific selector or response relevant to the task.
New page content is missing after a click The click and navigation were not synchronized, or a same-page update was mistaken for navigation. For navigation, use Promise.all with waitForNavigation; for in-place updates, wait for the new content selector.
Iframe content is absent page.content() serializes the page document, not a combined serialization of child documents. Find the relevant frame and call its content() after waiting for that frame’s content.
Lazy-loaded section is missing The site has not loaded it until scrolling or another interaction. Trigger the expected behavior, wait for the section, and extract again.
Navigation timeout The selected lifecycle event takes too long or never occurs. Choose a suitable navigation event, configure a task-appropriate timeout, and use a content-specific wait after navigation.
Browser process remains running An exception prevented the close call from running. Place browser use in try/finally and close it in the finally block.

When output looks incomplete, inspect both the wait condition and the document state. A useful diagnostic is to check for the exact element you expect before extracting:

const present = await page.$('[data-testid="article-body"]');
if (!present) throw new Error('Article body did not render before extraction');
const html = await page.content();

9. Performance, reliability, and cost considerations

Browser automation starts and controls a real browser process, so it has more setup and resource overhead than making a plain HTTP request. Reuse a browser process for multiple pages when appropriate, but isolate page state and close pages when finished. Avoid launching a new browser for every URL in a high-volume loop. Keep concurrency bounded because each active page consumes resources, and a target site may impose its own limits.

Reliability mostly comes from choosing useful wait conditions, setting timeouts, and cleaning up browser resources. A selector that represents the required content makes failures easier to detect than extracting immediately and discovering later that the page was empty. Treat navigation failures, missing selectors, and unexpected page states as explicit errors in a production pipeline, and record which URL and wait stage failed.

There is no universal runtime or cost figure for page.content(); page complexity, network behavior, browser startup, and your execution environment affect the work. Your own costs can include compute, browser hosting, and engineering time spent maintaining browser dependencies. If you only need the initial server response HTML, a browser may be unnecessary; if you need JavaScript-rendered DOM, Puppeteer is useful because it observes the page after it runs in the browser.

10. Or skip the browser setup

If your goal is a screenshot or PDF rather than the HTML string itself, ScreenshotNeo provides a website screenshot API and MCP server. A single GET request captures a URL as an image or PDF. Its API can also wait for a selector, delay, or network idle, and its parameter names are compatible with those used by other screenshot APIs. This does not return the page’s HTML source.

See the ScreenshotNeo API documentation for request options and setup. Here is the one-call cURL example:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

ScreenshotNeo accepts cookie and consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each of those steps can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers identify the page verdict and whether the shot was billed. Its MCP server gives AI agents tools to take screenshots, get page information, and capture PDFs.

The free plan includes 1,000 shots a month with no card. Paid plans start at $5 for 3,000 shots; every feature is on every plan. Sign up for 1,000 free screenshots a month, with no card required.

11. FAQ

Does page.content() return a string?

Yes. Await the promise to get the current document serialization as a JavaScript string.

Does it include the DOCTYPE?

Yes, Puppeteer documents that the full HTML content includes the DOCTYPE.

Will it include changes made by JavaScript?

It includes DOM changes already made when you call it. Wait for the relevant update before extracting.

Does page.content() download images and stylesheets?

No. It serializes the document markup, including references present in the DOM; it is not a guarantee that every referenced asset has loaded.

Is waitForNetworkIdle() always the best wait?

No. Use it when network quiet is a reliable signal for the target. A page-specific selector or response is often a better signal for dynamic content.