ScreenshotNeo
Sign in Get a free key

BlogEngineering

How to Create a Webpage Screenshot Service with Headless Chrome and a Queue

Build a Node.js screenshot API with Puppeteer, BullMQ, and Redis. This guide covers job design, browser workers, security, delivery, retries, and operations.

By the ScreenshotNeo team4 October 202614 min read

A practical webpage screenshot service separates fast HTTP intake from slower browser work: validate and authenticate a capture request, enqueue a small job, return its ID, let a worker render the page in headless Chrome, store the result, and expose job status and a protected result reference. The example below uses Node.js, Puppeteer, BullMQ, and Redis. The architecture is a synthesis of Puppeteer’s browser and screenshot APIs and BullMQ’s queue and worker model, not a prescribed service design.

Puppeteer is a high-level JavaScript browser automation API that runs headless by default and supports page and element screenshots. BullMQ is a Node.js queue built on Redis; workers pick up queued jobs and support concurrency, retries, and rate limiting. See the Puppeteer screenshot guide, BullMQ queue guide, and BullMQ worker guide.

1. Choose the service shape

Design Good fit Tradeoffs
Synchronous API capture Local tools, low volume, short and predictable navigations The client connection stays open while Chrome loads and renders. Slow pages tie up request capacity, and bursts can overwhelm browser resources.
API plus queue and workers Public APIs, bursty traffic, longer captures, independent scaling Requires Redis-compatible queue operations, job status and result delivery, cleanup, and worker monitoring.

A queue smooths bursts and lets browser capacity scale independently. It does not make rendering faster or remove the need to control resource use. Compare designs using measured request latency, queue delay, throughput, failure rate, recovery behavior, operational complexity, and worker isolation. There is no universal concurrency or latency figure: benchmark representative target pages in the deployment environment.

2. Define a bounded capture request

Keep the public API intentionally small at first. For example:

POST /v1/captures
Authorization: Bearer YOUR_SERVICE_TOKEN
Content-Type: application/json

{
  "url": "https://example.com",
  "width": 1440,
  "height": 900,
  "format": "png",
  "fullPage": false,
  "waitUntil": "domcontentloaded"
}

Return 202 Accepted with a job ID, and let the caller poll a status route:

{"id":"cap_...","status":"queued","statusUrl":"/v1/captures/cap_..."}

Choose and document service-level bounds for URL length, viewport dimensions, capture duration, output size, and allowed formats. Validate types and options before enqueueing. If callers may retry submissions, support an idempotency key and define how long it is retained. Put only the data a worker needs in Redis; store screenshot bytes in object or file storage, not in the queue payload.

3. Create the Node.js API and browser worker

This runnable starting point uses a local output directory for simplicity. The API and worker can run as separate processes using the same Redis and output volume; in production replace local files with object storage or another durable store accessible to both processes. The example uses polling for status. Add authentication, production storage, and reviewed URL security controls before exposing it publicly.

Install and configure

npm init -y
npm install express bullmq ioredis puppeteer

Set REDIS_URL and a long random API_TOKEN in the environment. Install a supported Node.js release and allow Puppeteer to install/use its compatible Chrome. In containers, follow Puppeteer’s current installation guidance and configure the required system libraries; do not assume a browser binary from a different image is compatible.

queue.js: shared queue configuration

const { Queue } = require('bullmq');
const IORedis = require('ioredis');

const connection = new IORedis(process.env.REDIS_URL || 'redis://127.0.0.1:6379', {
  maxRetriesPerRequest: null,
});
const captures = new Queue('captures', { connection });

module.exports = { connection, captures };

api.js: validate, enqueue, and expose status

const express = require('express');
const crypto = require('node:crypto');
const { captures } = require('./queue');

const app = express();
app.use(express.json({ limit: '16kb' }));
const token = process.env.API_TOKEN;
if (!token) throw new Error('Set API_TOKEN');

app.use((req, res, next) => {
  const supplied = req.get('authorization') || '';
  if (supplied !== `Bearer ${token}`) return res.status(401).json({ error: 'unauthorized' });
  next();
});

function validate(body) {
  if (!body || typeof body.url !== 'string' || body.url.length > 2048) return 'url must be a string up to 2048 characters';
  let parsed;
  try { parsed = new URL(body.url); } catch { return 'url must be absolute'; }
  if (!['http:', 'https:'].includes(parsed.protocol)) return 'only http and https URLs are accepted';
  const width = body.width ?? 1440;
  const height = body.height ?? 900;
  if (!Number.isInteger(width) || width < 320 || width > 2560) return 'width must be an integer from 320 to 2560';
  if (!Number.isInteger(height) || height < 240 || height > 2560) return 'height must be an integer from 240 to 2560';
  if (!['png', 'jpeg', 'webp'].includes(body.format ?? 'png')) return 'format must be png, jpeg, or webp';
  if (body.fullPage !== undefined && typeof body.fullPage !== 'boolean') return 'fullPage must be boolean';
  if (body.selector !== undefined && (typeof body.selector !== 'string' || body.selector.length > 500)) return 'selector must be a string up to 500 characters';
  if (body.waitUntil !== undefined && !['load', 'domcontentloaded', 'networkidle0', 'networkidle2'].includes(body.waitUntil)) return 'unsupported waitUntil value';
  return null;
}

app.post('/v1/captures', async (req, res, next) => {
  try {
    const problem = validate(req.body);
    if (problem) return res.status(400).json({ error: problem });
    const { url, width = 1440, height = 900, format = 'png', fullPage = false, selector, waitUntil = 'domcontentloaded' } = req.body;
    const clientKey = req.get('idempotency-key');
    // A caller-provided key lets BullMQ reuse the same job ID for duplicate submissions.
    // Validate and scope keys to the authenticated caller in a multi-tenant service.
    const jobId = clientKey ? `client-${clientKey.replace(/[^a-zA-Z0-9_-]/g, '').slice(0, 100)}` : `cap-${crypto.randomUUID()}`;
    const job = await captures.add('capture', { url, width, height, format, fullPage, selector, waitUntil }, {
      jobId,
      attempts: 3,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: { age: 3600, count: 1000 },
      removeOnFail: { age: 86400, count: 5000 },
    });
    res.status(202).json({ id: job.id, status: 'queued', statusUrl: `/v1/captures/${encodeURIComponent(job.id)}` });
  } catch (err) { next(err); }
});

app.get('/v1/captures/:id', async (req, res, next) => {
  try {
    const job = await captures.getJob(req.params.id);
    if (!job) return res.status(404).json({ error: 'capture not found' });
    const state = await job.getState();
    const response = { id: job.id, status: state };
    if (state === 'completed') response.resultUrl = `/v1/captures/${encodeURIComponent(job.id)}/result`;
    if (state === 'failed') response.error = job.failedReason;
    res.json(response);
  } catch (err) { next(err); }
});

// Local demo result route. Protect it with the same authorization as the API.
app.get('/v1/captures/:id/result', async (req, res, next) => {
  try {
    const job = await captures.getJob(req.params.id);
    if (!job || await job.getState() !== 'completed') return res.status(404).json({ error: 'result not ready' });
    res.type(job.returnvalue.contentType).sendFile(job.returnvalue.path);
  } catch (err) { next(err); }
});

app.use((err, req, res, next) => {
  console.error(err);
  res.status(500).json({ error: 'internal error' });
});
app.listen(process.env.PORT || 3000, () => console.log('API listening'));

worker.js: render and persist

const fs = require('node:fs/promises');
const path = require('node:path');
const puppeteer = require('puppeteer');
const { Worker } = require('bullmq');
const { connection } = require('./queue');

const outputDir = path.resolve(process.env.OUTPUT_DIR || './captures');
const mime = { png: 'image/png', jpeg: 'image/jpeg', webp: 'image/webp' };
let browser;

async function getBrowser() {
  if (!browser || !browser.connected) browser = await puppeteer.launch({ headless: true });
  return browser;
}

const worker = new Worker('captures', async job => {
  const { url, width, height, format, fullPage, selector, waitUntil } = job.data;
  const chrome = await getBrowser();
  const page = await chrome.newPage();
  try {
    await page.setViewport({ width, height, deviceScaleFactor: 1 });
    await page.goto(url, { waitUntil, timeout: 45000 });
    let target = page;
    if (selector) {
      target = await page.waitForSelector(selector, { timeout: 10000 });
      if (!target) throw new Error('selector not found');
    }
    const bytes = await target.screenshot({
      type: format,
      fullPage: selector ? undefined : fullPage,
      ...(format === 'jpeg' || format === 'webp' ? { quality: 85 } : {}),
    });
    await fs.mkdir(outputDir, { recursive: true });
    const filename = `${job.id}.${format}`;
    const filepath = path.join(outputDir, filename);
    await fs.writeFile(filepath, bytes);
    return { path: filepath, contentType: mime[format] };
  } finally {
    await page.close();
  }
}, {
  connection,
  concurrency: Number(process.env.WORKER_CONCURRENCY || 2),
  limiter: { max: 20, duration: 1000 },
});

worker.on('failed', (job, err) => console.error('capture failed', job?.id, err.message));
worker.on('error', err => console.error('worker error', err));

async function shutdown() {
  await worker.close();
  if (browser) await browser.close();
  await connection.quit();
  process.exit(0);
}
process.once('SIGTERM', shutdown);
process.once('SIGINT', shutdown);

Start Redis, then run node api.js and node worker.js in separate processes. Set OUTPUT_DIR to a shared persistent volume for this demo. The sample’s per-process limiter is not a complete global quota mechanism; BullMQ limiter behavior and configuration depend on the BullMQ version deployed, so confirm the current documentation for a distributed rate policy.

4. Submit jobs and retrieve the result

cURL

curl -sS -X POST http://localhost:3000/v1/captures \
  -H "Authorization: Bearer $API_TOKEN" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: example-homepage-1' \
  -d '{"url":"https://example.com","width":1440,"height":900,"format":"png","fullPage":true}'

Poll the returned statusUrl using the same authorization header until status is completed or failed. When complete, fetch its resultUrl and save the response body as an image.

Python client

import os
import time
import requests

base = 'http://localhost:3000'
headers = {'Authorization': f"Bearer {os.environ['API_TOKEN']}"}
response = requests.post(
    f'{base}/v1/captures', headers=headers,
    json={'url': 'https://example.com', 'format': 'png', 'fullPage': True},
    timeout=15,
)
response.raise_for_status()
job = response.json()

while True:
    status = requests.get(base + job['statusUrl'], headers=headers, timeout=15)
    status.raise_for_status()
    result = status.json()
    if result['status'] == 'completed':
        image = requests.get(base + result['resultUrl'], headers=headers, timeout=60)
        image.raise_for_status()
        with open('capture.png', 'wb') as output:
            output.write(image.content)
        break
    if result['status'] == 'failed':
        raise RuntimeError(result.get('error', 'capture failed'))
    time.sleep(1)

Node.js client

const base = 'http://localhost:3000';
const headers = {
  authorization: `Bearer ${process.env.API_TOKEN}`,
  'content-type': 'application/json',
};
const created = await fetch(`${base}/v1/captures`, {
  method: 'POST', headers,
  body: JSON.stringify({ url: 'https://example.com', format: 'webp', fullPage: true }),
});
if (!created.ok) throw new Error(`submit failed: ${created.status}`);
const job = await created.json();

while (true) {
  const response = await fetch(base + job.statusUrl, { headers });
  if (!response.ok) throw new Error(`status failed: ${response.status}`);
  const state = await response.json();
  if (state.status === 'failed') throw new Error(state.error || 'capture failed');
  if (state.status === 'completed') {
    const image = await fetch(base + state.resultUrl, { headers });
    if (!image.ok) throw new Error(`result failed: ${image.status}`);
    const fs = await import('node:fs/promises');
    await fs.writeFile('capture.webp', Buffer.from(await image.arrayBuffer()));
    break;
  }
  await new Promise(resolve => setTimeout(resolve, 1000));
}

5. Select navigation readiness and screenshot options

Puppeteer’s page.goto() supports readiness conditions such as load, domcontentloaded, networkidle0, and networkidle2. The screenshot guide demonstrates networkidle2, but it is not universally right: sites with analytics, polling, streaming, or other persistent requests may never become idle. Start with an appropriate navigation condition, then add an explicit selector wait or a bounded delay for a known late-rendering element when needed. Set a firm overall navigation timeout.

Need Approach Watch for
Viewport image page.screenshot() with fullPage: false Captures only visible page dimensions.
Full document fullPage: true Very tall pages can create large images and use substantial memory.
One element Wait for a selector and call its screenshot() Missing selectors should fail clearly; Puppeteer attempts to scroll a hidden element into view.
Format and quality PNG, JPEG, or WebP; set quality for lossy output Quality is relevant to JPEG/WebP support in Puppeteer; validate option behavior against the deployed version.
Viewport and device scale page.setViewport({ width, height, deviceScaleFactor }) Higher device scale produces more pixels and larger output.
Lower-level control Chrome DevTools Protocol Page.captureScreenshot Use when direct protocol control is needed; it exposes format, quality, clipping, and beyond-viewport parameters.

See Puppeteer’s ScreenshotOptions API and the Chrome DevTools Protocol screenshot method. The sample API intentionally omits arbitrary JavaScript, custom headers, cookies, proxy settings, PDF, and request interception. Each adds security and resource implications; add only requirements you can validate and isolate.

6. Secure arbitrary URL capture

A public capture API is an outbound browsing proxy: an attacker may try to make Chrome reach internal services or sensitive network destinations. The API’s basic URL parser is input validation, not an SSRF defense. Before public launch, have an engineer define and review policy and controls for:

  • Private, loopback, link-local, and otherwise internal IP ranges, including IPv4 and IPv6.
  • DNS resolution and rebinding; a hostname can resolve differently between validation and connection.
  • Redirect destinations, alternate encodings, non-HTTP schemes, and local browser resources.
  • Subresource requests initiated by the page, not just the original URL.
  • Browser downloads, file access, request sizes, navigation duration, and overall job duration.
  • Outbound network restrictions and isolation from service credentials, metadata endpoints, and sensitive networks.
  • Abuse controls: caller authentication, quotas, request logging with care, and result access scoped to the owner.

This checklist identifies design areas; it is not a complete or verified security policy. A queue does not solve these risks. Run Chrome workers in a restricted environment with no access to application secrets or internal services, and validate the chosen controls against current security guidance and adversarial testing before accepting arbitrary public URLs.

7. Retries, concurrency, and reliability

BullMQ documents retries, backoff, worker concurrency, multiple workers, and rate limiting. Start with low concurrency per worker and increase only after measuring memory, CPU, browser crashes, queue wait, capture duration, and failure rates on representative sites. A browser page is a resource-heavy job; example values are not capacity guarantees. Scale by adding workers only while Redis, storage, and outbound network capacity remain healthy.

  • Retry transient failures: temporary navigation errors or browser restarts may succeed on another attempt. Exponential backoff limits rapid repeated attempts.
  • Do not retry permanent input failures indefinitely: reject invalid options at intake and classify unsupported or inaccessible targets for a clear terminal result.
  • Assume work can repeat: BullMQ describes its goal as exactly-once semantics with at-least-once delivery in the worst case. A job may execute again after failures. Use stable IDs where appropriate and make output writes idempotent or deliberately versioned.
  • Handle shutdown: stop accepting work, allow active jobs to finish or be requeued, close pages and browsers, and let the queue worker shut down cleanly.
  • Monitor the pipeline: queued, active, completed, and failed counts; queue age; duration; retry count; browser restarts; storage errors; and result expiry.

BullMQ’s own description says queues can smooth processing peaks and offload work across workers; it does not guarantee a particular throughput or that a screenshot’s side effects never repeat. See its retry guidance and rate-limiting guide.

8. Results, privacy, and retention

The sample stores output locally, which is suitable only when API and worker share a persistent volume. A multi-host service needs durable shared storage. Keep result references authenticated or use appropriately scoped, expiring access links. Define retention and deletion rules before launch: screenshots can contain personal or confidential page content. Avoid logging credentials, full sensitive URLs, cookies, or screenshot bytes. Decide what metadata callers can see and how long job records and files remain available.

Keep Redis job records compact and expire completed and failed jobs. Ensure file/object cleanup follows the same retention policy; removing queue records alone does not delete stored images. Consider maximum output dimensions and bytes, and reject or stop work that exceeds the service budget.

9. Performance and cost planning

The major cost drivers are browser CPU and memory, Redis, storage and egress, and the operational work of securing and monitoring the worker pool. Full-page and high-scale captures increase pixel count and output size. Slow pages occupy a worker longer. Queueing changes when work happens and helps absorb bursts; it does not reduce the compute required for each capture.

  • Record queue wait separately from browser render time so overload is visible.
  • Use a representative mix of fast, slow, long, and failure-prone pages for capacity trials.
  • Keep concurrency conservative; observe memory and browser stability before raising it.
  • Set timeouts and limits for dimensions, retries, job age, and storage retention.
  • Choose PNG for lossless output and JPEG/WebP when smaller lossy images suit the use case.
  • Do not publish latency, throughput, or per-screenshot cost estimates without measurements in the target environment.

10. Troubleshooting

Symptom Likely cause Fix
Job remains queued No worker is running, Redis connection is unhealthy, or the worker uses a different queue/Redis configuration. Check worker logs and Redis connectivity; confirm queue name and environment match.
Navigation timeout The page is slow, blocked, or never reaches the selected readiness condition. Use a suitable waitUntil, set an explicit navigation timeout, and wait for a specific selector when appropriate. Do not blindly wait for network idle on persistent connections.
Screenshot is blank or incomplete Capture happened before client-rendered content appeared, a required selector was absent, or the target failed to load. Wait for the actual content selector, verify page status and worker logs, and distinguish page failure from successful capture.
Selector wait fails Selector is wrong, content is in a frame/shadow root, or element appears later than the timeout. Confirm selector in the target document, increase a bounded wait if justified, or add frame-aware handling for the application’s structure.
Chrome fails to launch in container Missing shared libraries, incompatible browser binary, or sandbox/container configuration. Use Puppeteer’s installation guidance and compatible browser; build an image with dependencies. Do not disable browser sandbox protections as a casual fix for a public service.
Workers crash or memory rises Too much concurrency, very large pages, leaked pages, or browser instability. Close pages in finally, lower concurrency, enforce page/dimension limits, and recycle a browser deliberately after failures.
Duplicate capture or overwritten result Client retries or queue recovery repeated a job. Use scoped stable job IDs and idempotent output writes; decide whether duplicates return the original job or create a new capture.
Result route returns missing file Worker and API do not share storage, or cleanup removed the object. Use shared durable storage and align record/file retention policies.
Unexpected internal URL access Application URL validation was mistaken for SSRF protection. Stop public exposure and implement a reviewed network egress and destination policy, including redirects and subresources.

11. When to use a hosted screenshot API

Building this service makes sense when you need control over browser behavior, networking, storage, and job policy and can own the security and operations. If your goal is to get screenshots without operating Chrome workers, ScreenshotNeo is a website screenshot API and MCP server from Yorker Media. Its API takes one GET request and returns an image or PDF. The details and options are in the ScreenshotNeo API documentation.

Or skip the browser setup

One GET request can return the capture directly:

cURL

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
await import('node:fs/promises').then(fs => fs.writeFile('shot.webp', Buffer.from(await res.arrayBuffer())));

See the API docs for request options. Cookie banners, popups, and chat widgets are removed before the shot; each cleanup step can be turned off. Bot checks, blank pages, and failed loads are never billed, and response headers identify page verdict and billing. Its MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. Sign up for 1,000 free screenshots a month, no card required.

FAQ

Should the API wait until a screenshot is finished before replying?

For variable or bursty browser work, return a job ID and let callers poll or receive a completion callback. A synchronous route is simpler when captures are predictably short and request timeouts are acceptable.

Does a queue make the screenshot service exactly once?

No. Queue recovery may cause work to run again. Design result writes and caller retries to be idempotent, and make duplicate behavior explicit.

Can I make this endpoint public if it only accepts HTTPS?

No. HTTPS validation alone does not prevent unsafe redirects, DNS changes, or page subrequests to internal destinations. Public arbitrary-URL capture needs a reviewed security and network isolation policy.

When should I use Chrome DevTools Protocol directly?

Use Puppeteer for conventional Node.js browser automation. Consider CDP when you need direct access to protocol-level behavior such as clipping and capture parameters.

Keep reading