How to bulk screenshot websites with a custom browser user agent

To bulk screenshot websites with a custom browser user agent, use shot-scraper for a declarative YAML batch, or use Playwright when you need per-URL logic or browser interactions. In shot-scraper, pass --user-agent to the multi command. In Playwright, set userAgent when creating a browser context, then navigate to each URL and save a screenshot.

A user-agent string can influence what a server returns, but it does not by itself emulate a phone or tablet. For consistent results, set the viewport, capture mode, and waits explicitly, and handle failed pages so the batch report does not hide missing images.

1. Choose a batch method

Need	Choose	Why
A list of URL/output pairs with minimal code	shot-scraper	Define jobs in YAML and run them together.
Per-page branching, custom error handling, or scripted interactions	Playwright	Control browser contexts, pages, navigation, and screenshots directly.
A screenshot API call without managing a local browser	ScreenshotNeo	Clean shots with cookie banners, popups, and chat widgets removed; only clean shots are billed.

The examples below use the documented interfaces. Check the installed shot-scraper version’s --help output before relying on command options, because CLI syntax can change between versions.

2. Bulk capture with shot-scraper

Create a YAML file with one URL and output filename for each capture. For example, save this as shots.yml:

- output: example-home.png
  url: https://example.com/
- output: example-org-docs.png
  url: https://example.org/docs/

Run the batch with your chosen user-agent string:

shot-scraper multi shots.yml --user-agent 'Example User Agent'

Replace the example string with the user agent you need. The option applies to the multi command invocation; use separate runs if different groups of pages need different user agents. The YAML makes the mapping between each requested URL and its resulting file explicit.

Useful shot-scraper batch controls

• --user-agent TEXT sets the user agent for the batch.
• --timeout sets a time limit; choose a value suitable for the slowest pages in your set.
• --fail and --skip control how HTTP errors are treated. Pick behavior that makes failures visible to the process consuming the batch.
• --no-clobber skips outputs that already exist, which helps avoid overwriting prior captures.
• -o selects particular configured outputs when you want to run only a subset.
• Multi configuration supports dimensions and wait or wait-for settings. Keep those consistent across jobs when comparing captures.

Consult the shot-scraper multi documentation for the exact syntax supported by your installed version and for other per-job options.

3. Bulk capture with Playwright in Node.js

Install Playwright and its browser using the official setup instructions. The following is a runnable script pattern: save it as bulk-screenshots.mjs in a project where Playwright is installed, then run it with Node.js. It creates one context with the custom user agent and viewport, then processes a URL list sequentially and records individual failures instead of silently stopping at the first one.

import { chromium } from 'playwright';

const userAgent = 'Example User Agent';
const jobs = [
  { url: 'https://example.com/', path: 'example-home.png' },
  { url: 'https://example.org/docs/', path: 'example-org-docs.png' },
];

const browser = await chromium.launch();
const context = await browser.newContext({
  userAgent,
  viewport: { width: 1440, height: 900 },
});

const results = [];
try {
  for (const job of jobs) {
    const page = await context.newPage();
    try {
      const response = await page.goto(job.url, {
        waitUntil: 'load',
        timeout: 45000,
      });
      if (response && response.status() >= 400) {
        throw new Error(`HTTP ${response.status()} for ${job.url}`);
      }
      await page.screenshot({ path: job.path, fullPage: true });
      results.push({ url: job.url, path: job.path, ok: true });
    } catch (error) {
      results.push({ url: job.url, ok: false, error: String(error) });
    } finally {
      await page.close();
    }
  }
} finally {
  await context.close();
  await browser.close();
}

for (const result of results) {
  console.log(JSON.stringify(result));
}

if (results.some((result) => !result.ok)) {
  process.exitCode = 1;
}

The script captures the full scrollable page. For a viewport-only image, change fullPage: true to fullPage: false or omit the option. The load event is a baseline, not a guarantee that a single-page app or delayed content is ready; use an appropriate wait condition for the target site.

Wait for the content you need

For pages that render content after navigation, wait for a known element before capturing:

await page.goto(job.url, { waitUntil: 'domcontentloaded', timeout: 45000 });
await page.locator('main').waitFor({ state: 'visible', timeout: 15000 });
await page.screenshot({ path: job.path, fullPage: true });

Replace main with a selector that reliably identifies the content on that site. A fixed delay can be used when there is no stable selector, but it makes every job wait for the full delay even when the page is ready sooner.

4. Set the right user agent and device behavior

Choose a user-agent value appropriate to your authorized testing or rendering task. A user agent can affect server-side user-agent detection and page content selection. It does not automatically set viewport size, screen dimensions, device scale factor, touch behavior, or other device properties.

For a mobile layout, configure a device profile or set the relevant browser context properties alongside the user agent. Playwright documents device descriptors and browser context options in its Page and browser API documentation. Keep the selected settings fixed for the entire comparison batch.

• Use one browser engine and version for a comparison run.
• Set the viewport width and height explicitly.
• Decide whether each file should capture the viewport or the full page.
• Use consistent navigation and content-ready waits.
• Use distinct output paths for different user agents or capture configurations so runs do not overwrite one another.

5. cURL, Python, and Node.js with ScreenshotNeo

If you do not need to run a local browser, ScreenshotNeo accepts a URL in one GET request and returns a PNG, JPEG, WebP, or PDF. For a batch, call the endpoint once per URL, or use its bulk capture option for up to 100 URLs per call. These examples use the documented API base and parameter pattern; see the ScreenshotNeo API documentation for available parameters, formats, and authentication details.

cURL

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

Python

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
r.raise_for_status()
with open("shot.webp", "wb") as f:
    f.write(r.content)

Node.js

const q = new URLSearchParams({
  access_key: 'YOUR_API_KEY',
  url: 'https://stripe.com',
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
const bytes = new Uint8Array(await res.arrayBuffer());
await import('node:fs/promises').then(({ writeFile }) => writeFile('shot.webp', bytes));

For a URL list, iterate over the URLs and save each response under a distinct filename, or submit a bulk capture request as described in the docs. Keep API keys out of source control and logs. ScreenshotNeo supports custom user agents along with viewport, full-page capture, waits, cookies, headers, and many other options; consult the docs for the parameter names and formats.

Or skip the browser setup

ScreenshotNeo can take the screenshot without you installing or managing a browser. Cookie banners, newsletter popups, and chat widgets are removed before the shot, and each of those cleanup steps can be turned off. Bot checks, blank pages, timeouts, and failed loads are never billed; cache hits also cost nothing, and the response includes X-Page-Verdict and X-Billed headers so you can see the outcome.

It also provides an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 screenshots; every feature is available on every plan.

curl -G "https://api.screenshotneo.com/v1/shot" \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

Create a free ScreenshotNeo account to get 1,000 screenshots a month with no card.

6. Make batches repeatable

1. Use explicit inputs and outputs. Store the URL list and output mapping in versioned data or a checked-in YAML file.
2. Pin the capture environment. Keep browser engine, browser version, viewport, user agent, and relevant device settings stable when comparing runs.
3. Choose a readiness condition. Prefer a site-specific selector or known application signal where navigation alone does not mean the page is ready.
4. Make failures visible. Record URL, output path, status or exception, and whether a screenshot was written. Return a nonzero process status if failures should fail the job.
5. Protect prior results. Use unique run directories, unique names, or shot-scraper’s no-clobber option when retaining prior captures matters.
6. Review a sample. Check representative pages from the batch before relying on all outputs; user-agent-specific content and page behavior vary by site.

7. Performance, reliability, and cost

Performance

Sequential Playwright processing is easy to reason about and reduces simultaneous browser load, but total runtime grows with the number of pages and each page’s navigation and wait time. If you add concurrency, use a bounded number of pages or workers rather than opening an unbounded set; browser memory and target-site load both increase with concurrency. Full-page screenshots can take more time and produce larger files than viewport captures. No universal capture-speed figure applies across sites and environments.

Reliability

Pages can redirect, require authentication, fail to load, return an HTTP error, or render essential content after the chosen wait. A navigation completing does not ensure the desired content is present. Log failures per URL, preserve the exact user-agent and viewport with each run, and retry only errors that are likely transient. Avoid treating an HTTP response status alone as proof that the screenshot is visually correct.

Cost

With a local Playwright or shot-scraper workflow, account for the machine and browser environment you operate; the research sources provide no comparable cost or speed figures. ScreenshotNeo has a Free plan with 1,000 shots per month and no card, then Starter at $5 for 3,000, Growth at $15 for 15,000, Pro at $39 for 60,000, Scale at $99 for 250,000, and Business at $249 for 1,000,000. Yearly billing gives two months free. Only clean shots are billed, and every feature is on every plan.

8. Troubleshooting

Symptom	Likely cause	Fix
CLI reports an unknown option	Installed shot-scraper version has different flags.	Check shot-scraper multi --help and the documentation for that version; confirm the option is attached to the multi command.
Some expected files are missing	A URL failed, timed out, or HTTP errors were skipped or treated according to the selected policy.	Use deliberate --fail or --skip behavior, set an adequate timeout, and review per-URL output and logs.
Existing images did not change	--no-clobber skipped files, or outputs point to prior filenames.	Choose new output names or remove the old files when replacement is intended.
Page looks like desktop despite mobile user agent	User-agent override does not configure viewport, screen, or touch settings.	Use a device profile or set the associated browser context properties as well.
Screenshot is blank or incomplete	Capture happened before client-rendered or delayed content appeared.	Wait for a relevant visible selector or application-ready signal; then capture.
Playwright navigation times out	The page is slow, long-lived, or waiting for the selected navigation event.	Choose an appropriate timeout and wait condition, then separately wait for the content required in the image.
One bad URL stops the whole script	Error handling is outside the per-job loop.	Catch errors inside each job, record the failing URL, continue, and set the final process status according to your batch policy.
Screenshot differs between runs	Browser version, viewport, user agent, page state, timing, or dynamic content changed.	Keep environment settings stable, wait for meaningful readiness, and account for content that changes on each visit.

9. Frequently asked questions

Does changing the user agent make a browser undetectable?

No. A user-agent override changes that browser-reported value; it is not a guarantee about how a website identifies or handles automation.

Should I use full-page screenshots for every URL?

Only when the whole document is needed. Viewport captures are usually more comparable and smaller when the task concerns the initial visible screen.

Can every URL in a batch use a different user agent?

With Playwright, group jobs by browser context and create each context with its desired user agent. For shot-scraper, run groups with the appropriate command-level option, after checking the installed version’s documented syntax.

Can ScreenshotNeo capture a list of URLs?

Yes. Its bulk capture feature accepts up to 100 URLs per call; see the API docs for request details.

Sources

• Playwright screenshots documentation
• Playwright Page API documentation
• shot-scraper: Taking multiple screenshots
• shot-scraper project documentation