How to Capture Bulk Screenshots of URLs from a PostgreSQL Table

To capture bulk screenshots of URLs stored in PostgreSQL, select the rows you want, then send each URL to a browser automation script and save the resulting image. This guide uses Node.js, the Playwright Page API, and the PostgreSQL COPY command to export query results as CSV. The script parses CSV safely, captures URLs one at a time, and records a success or failure for every row.

Use a stable database identifier in filenames and a CSV parser rather than splitting lines on commas. CSV fields may contain commas, quotes, or line breaks. Choose viewport or full-page capture based on what reviewers need to see.

1. Choose a database handoff

There are two practical workflows:

• Export to CSV: easiest to inspect and rerun. PostgreSQL executes a query and writes the selected rows as CSV.
• Query through a database driver: convenient when the capture script already runs inside an application that connects to PostgreSQL. Choose the driver and connection handling used in your environment.

The runnable example below uses CSV, so the database export and browser capture are separate steps. Replace public.pages and the column names with your table and selection rules. Select an identifier as well as the URL so each image has a stable filename.

COPY (
  SELECT id, url
  FROM public.pages
  WHERE url IS NOT NULL
  ORDER BY id
) TO '/tmp/page-urls.csv'
WITH (FORMAT CSV, HEADER TRUE);

COPY ... TO writes to a path accessible to the PostgreSQL server process. If that is not appropriate in your deployment, use psql‘s client-side \copy command with the same query and CSV options:

\copy (SELECT id, url FROM public.pages WHERE url IS NOT NULL ORDER BY id) TO 'page-urls.csv' WITH (FORMAT CSV, HEADER TRUE)

Keep the query narrow and deliberate: filter to the intended records, select only needed columns, and order rows for repeatable processing. For PostgreSQL versions other than 18, check the matching version of the COPY documentation for available syntax and options.

2. Install the capture dependencies

Use a current Node.js release supported by your deployment, then create a small project and install Playwright and a CSV parser:

mkdir pg-screenshots
cd pg-screenshots
npm init -y
npm install playwright csv-parse
npx playwright install chromium

Playwright installs browser binaries separately from the package. In a container or CI environment, install the browser and operating-system dependencies required by the Playwright version you use.

3. Capture each CSV row with Playwright

Save this as capture.mjs. It reads the CSV generated above, validates basic URL structure, captures either the visible viewport or the full page, and writes a JSONL result record for every row. One row failing does not end the batch.

import fs from 'node:fs';
import path from 'node:path';
import { parse } from 'csv-parse/sync';
import { chromium } from 'playwright';

const inputPath = process.env.INPUT_CSV ?? '/tmp/page-urls.csv';
const outputDir = process.env.OUTPUT_DIR ?? './screenshots';
const fullPage = process.env.FULL_PAGE === '1';
const timeoutMs = Number(process.env.NAVIGATION_TIMEOUT_MS ?? 30000);

if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) {
  throw new Error('NAVIGATION_TIMEOUT_MS must be a positive number');
}

fs.mkdirSync(outputDir, { recursive: true });
const rows = parse(fs.readFileSync(inputPath), {
  columns: true,
  skip_empty_lines: true,
  bom: true,
  trim: true,
});

if (!rows.length) {
  console.log('No rows to capture.');
  process.exit(0);
}

const browser = await chromium.launch({ headless: true });
const resultsPath = path.join(outputDir, 'results.jsonl');
const results = fs.createWriteStream(resultsPath, { flags: 'w' });

try {
  const page = await browser.newPage({ viewport: { width: 1440, height: 900 } });

  for (const row of rows) {
    const id = String(row.id ?? '').trim();
    const rawUrl = String(row.url ?? '').trim();
    const startedAt = new Date().toISOString();

    if (!id || !rawUrl) {
      results.write(JSON.stringify({ id: id || null, url: rawUrl || null, status: 'skipped', error: 'Missing id or url', startedAt }) + '\n');
      continue;
    }

    let url;
    try {
      url = new URL(rawUrl);
      if (!['http:', 'https:'].includes(url.protocol)) throw new Error('Only http and https URLs are allowed');
    } catch (error) {
      results.write(JSON.stringify({ id, url: rawUrl, status: 'failed', error: `Invalid URL: ${error.message}`, startedAt }) + '\n');
      continue;
    }

    // Use a sanitized identifier, not a URL, for a predictable filename.
    const safeId = id.replace(/[^a-zA-Z0-9_-]/g, '_').slice(0, 100) || 'row';
    const screenshotPath = path.join(outputDir, `${safeId}.png`);

    try {
      const response = await page.goto(url.href, { waitUntil: 'load', timeout: timeoutMs });
      if (!response) throw new Error('Navigation returned no main-resource response');
      if (response.status() >= 400) throw new Error(`Main resource returned HTTP ${response.status()}`);

      await page.screenshot({ path: screenshotPath, fullPage });
      results.write(JSON.stringify({
        id,
        url: url.href,
        status: 'ok',
        httpStatus: response.status(),
        screenshot: screenshotPath,
        fullPage,
        startedAt,
        finishedAt: new Date().toISOString(),
      }) + '\n');
    } catch (error) {
      results.write(JSON.stringify({
        id,
        url: url.href,
        status: 'failed',
        error: error.message,
        startedAt,
        finishedAt: new Date().toISOString(),
      }) + '\n');
    }
  }
} finally {
  results.end();
  await browser.close();
}

console.log(`Processed ${rows.length} rows. Results: ${resultsPath}`);

Run it with:

INPUT_CSV=/tmp/page-urls.csv OUTPUT_DIR=./screenshots node capture.mjs

# Capture the full scrollable page instead of just the viewport:
FULL_PAGE=1 INPUT_CSV=/tmp/page-urls.csv OUTPUT_DIR=./screenshots node capture.mjs

The code uses waitUntil: 'load', which waits for the page load event. Some modern sites continue loading content after that event; a site-specific wait condition may be needed. The script treats HTTP error responses as failures, but sites can return an error page with a 2xx status, so inspect the captured results when that distinction matters.

4. Select capture scope and output deliberately

Playwright’s screenshot guide documents the screenshot operation and its options. Match the capture to the downstream task:

Choice	Use it when	Consideration
Viewport screenshot	You need a consistent visible frame for review or comparison.	Content below the viewport is omitted.
Full-page screenshot	You need the entire scrollable document.	Very long pages can create large images and take longer to capture.
PNG	Lossless output matters, such as visual review.	Files are often larger than compressed alternatives.
JPEG	Smaller photographic output is useful.	Compression can affect fine visual detail.
CSS scale	Default-size captures suit the review workflow.	Output dimensions follow CSS pixels.
Device scale	Higher-density output is needed for retina-style review.	Image dimensions and storage use increase.

To change the example’s output format or scale, set type and scale in page.screenshot(), for example await page.screenshot({ path: screenshotPath, fullPage, type: 'jpeg', quality: 80, scale: 'css' }). Use a .jpg filename when saving JPEG output. The supported combinations depend on the Playwright version; consult the Page API.

5. Consider a direct PostgreSQL connection

If the capture process already has application database access, you can query rows directly instead of exporting CSV. The relevant steps are the same: execute a parameterized query, iterate over the returned { id, url } rows, and run the navigation and screenshot block for each row. Use your application’s established PostgreSQL driver and credentials management. Avoid constructing SQL by concatenating untrusted input, and never put database credentials in source control.

CSV is often easier to audit and replay. Direct database access avoids managing an intermediate file, but it couples the capture job to database availability and connection configuration. For either handoff, define whether rows are a snapshot at job start or can change during processing.

6. Make the batch reliable and safe

• Keep an outcome for each row. The JSONL file records successes, invalid rows, and navigation failures so you can retry selected records.
• Use stable identifiers. A database key avoids collisions that can happen when filenames are derived from URLs. If IDs are not unique, include another stable field in the filename.
• Bound time and resources. Set a navigation timeout appropriate to the target sites. Process a limited number of pages at once, especially when captures are full-page or the batch is large.
• Choose readiness criteria per site. A page’s load event may not mean its final content is ready. For known pages, wait for a selector, a specific event, or an application condition. Avoid a universal fixed delay when a meaningful readiness signal exists.
• Retry selectively. Retry transient timeouts or connection errors with a bounded retry count and delay. Do not endlessly retry invalid URLs, blocked destinations, or persistent HTTP errors.
• Protect internal systems. If URLs come from users or an untrusted table, restrict allowed schemes and destinations. A headless browser can reach network resources available to its host; apply network egress controls to avoid exposing internal services.
• Plan output storage. Full-page images and repeated runs can consume disk quickly. Use an output retention policy and ensure a failed disk write is surfaced as a job failure.

For visual comparisons, rendering is not perfectly reproducible across environments. Playwright notes that output can vary with operating system, browser version, settings, hardware, power source, and headless mode. Pin the browser and runtime environment when consistency matters, and compare captures made under the same conditions. See Playwright’s visual comparison guidance.

7. Performance, reliability, and cost

This workflow’s primary costs are browser runtime, machine capacity, and image storage. No universal throughput figure applies: sites differ in response time, page weight, client-side rendering, and access controls. Measure a representative sample in the target environment before scheduling a large recurring job.

• Throughput: sequential processing is simple and limits pressure on target sites. If increasing throughput, use a small bounded number of independent pages or workers, and respect each site’s access policies and rate limits.
• Memory: close pages or contexts when they are no longer needed. Full-page capture and unusually long pages can require more memory than a viewport capture.
• Reliability: persist row-level results and make reruns idempotent. Decide whether a rerun overwrites existing images or writes to a run-specific directory.
• Storage: estimate output size from a representative sample, then include JSONL results and failed-run artifacts in retention planning.
• Database load: export only the rows needed for the run. For a long-running capture, a saved CSV provides a fixed input snapshot that can be retried without repeating the database query.

8. Troubleshooting

Symptom	Likely cause	Fix
COPY cannot write the destination file	The path is interpreted on the database server and the server process cannot access it.	Choose a server-accessible location, or use psql client-side \copy.
CSV rows appear shifted or split	The file was parsed by splitting on commas or newlines.	Use a CSV parser such as csv-parse, which handles quoted delimiters and embedded newlines.
Playwright says the browser executable is missing	The package is installed but its browser binary is not.	Run npx playwright install chromium in the environment that runs the script.
Navigation times out	The site is slow, never reaches the chosen load event, or blocks automated browsing.	Check reachability and the site’s policy, adjust a reasonable timeout, and choose a site-appropriate readiness condition. Record and retry transient failures selectively.
The screenshot is blank or incomplete	Content may be rendered after navigation, lazy-loaded on scroll, or hidden behind a client-side state.	Wait for a meaningful selector or state. For lazy content, scroll through the page before capturing and allow the content to settle.
Some sites show a bot check or CAPTCHA	The site is restricting automated visits.	Respect the site’s access rules. Do not attempt to bypass its challenge; mark the row as blocked and use an authorized access method.
Filenames collide	IDs are not unique or sanitize to the same filename.	Include a unique database key or another stable disambiguating value in the filename.
Images differ between runs	Browser or operating-system rendering, timing, dynamic page content, or remote assets changed.	Keep browser and host settings consistent, wait for stable content, and account for dynamic regions in downstream comparisons.

Or skip the browser setup

For a batch driven by application code, send each selected URL to ScreenshotNeo’s screenshot API instead of installing and maintaining a browser in the capture job. The request below saves a WebP screenshot; see the ScreenshotNeo API documentation for the available options and the product site at ScreenshotNeo.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

In a bulk job, substitute each PostgreSQL row’s URL and a safe per-row output filename. ScreenshotNeo removes cookie and consent banners, newsletter popups, and chat widgets before the shot. Bot checks, blank pages, and failed loads are never billed. Its MCP server lets AI agents take screenshots, and 1,000 screenshots per month are free with no card; paid plans start at $5 for 3,000 screenshots.

Create a free ScreenshotNeo account to get an API key and start with 1,000 screenshots a month at no charge and no card.

FAQ

Should I save screenshots in PostgreSQL?

This workflow writes image files and a JSONL manifest. Store images in the storage system suited to your application; keep database references or job results separately if that fits your data model.

Can I capture only part of a page?

Yes. Playwright supports element screenshots through a locator’s screenshot operation. Identify the element with a selector that is stable for the target page, then capture that element instead of the whole page.

Does the example guarantee identical images every run?

No. Remote content and rendering conditions can change. Keep the browser environment and capture timing consistent when repeatability matters, and consult Playwright’s guidance on rendering variation.