How to Capture Screenshots of Multiple URLs With a Screenshot API and Respect Rate Limits

To capture screenshots of many URLs without hitting rate limits, first check whether your provider offers a batch endpoint and status tracking. If it does, submit appropriately sized batches and track each URL’s result. Otherwise, put individual URLs in a queue, limit concurrency to the provider’s documented capacity, and use usage data or response headers to pace requests. Treat temporary rate limits separately from monthly quota exhaustion: honor Retry-After when supplied, retry transient failures with bounded backoff and jitter, and do not retry invalid requests, authentication failures, or depleted quota.

Batching reduces submission overhead, but it does not necessarily reduce the number of screenshots counted against a plan. Limits and accounting differ by provider, so use its current documentation and account usage as the source of truth.

1. Choose a batch endpoint or a paced queue

Use a batch endpoint when the provider supports your workload, the batch size fits its documented maximum, and you can retrieve each item’s result later. Use a paced queue when no suitable batch API exists or when you need fine-grained control over retries and per-URL status.

Approach	Use it when	What to check
Batch endpoint	The provider accepts groups of URLs and returns a batch ID or per-URL identifiers.	Maximum batch size, whether each URL consumes quota, how partial failures appear, and how to poll or receive completion events.
Queue of individual requests	You need controlled pacing, provider has no fitting batch endpoint, or each URL needs an independent retry policy.	Per-minute/request limits, concurrency semantics, usage headers or endpoint, and retry guidance.

“Concurrency” can mean different things. For example, ScreenshotOne describes concurrency.remaining and concurrency.reset as the current one-minute request bucket, not the number of browser renders running simultaneously. Read the provider’s definitions before using a field to set worker count. See [ScreenshotOne’s bulk screenshot guide](https://screenshotone.com/docs/guides/bulk-screenshots/).

Screenshot API documents POST /api/v1/screenshot/batch, batch IDs, and status tracking. ScreenshotRun documents per-screenshot status and webhooks, and states that each URL counts against monthly quota even when submitted in a batch. These are provider-specific examples, not universal behavior. Check [Screenshot API’s REST documentation](https://screenshot-api.org/docs/) and [ScreenshotRun’s batch documentation](https://screenshotrun.com/docs/batch) for current details.

2. Validate and normalize the URL list

Before sending work, remove malformed entries, normalize URLs consistently, and deduplicate only when duplicate captures are not intentional. Decide whether redirects, authentication, query strings, and URL fragments are meaningful to your job. Most HTTP servers do not send URL fragments to the server, but a page’s client-side app may use them; confirm how the screenshot provider navigates to fragment URLs.

• Accept only the schemes supported by the provider, commonly https and sometimes http.
• Reject empty hosts, malformed URLs, and unsupported schemes before spending request capacity.
• Preserve query parameters when they select different content. Avoid logging secrets embedded in query strings.
• Apply your own allowlist or denylist if the input list is user-controlled; do not let an arbitrary URL list turn your service into an unintended proxy.
• Assign each input a stable job ID so duplicate URLs can still be tracked as distinct requested items when required.

Do not retry a URL validation error. Correct or remove the input instead.

3. Track both request rate and monthly quota

Model rate capacity and plan quota as separate budgets:

• Request-rate budget: how quickly submissions may be made within a time window. A provider may apply it per API call, per URL, per IP, or at more than one layer.
• Monthly screenshot budget: how many rendered URLs the account may consume over a billing period. A single batch submission may count as many screenshots.

Use the provider’s usage endpoint or documented response headers to decide when to pause and resume. Store the latest remaining-capacity and reset-time values with your job state. Do not assume a rate-limit reset also replenishes monthly quota.

The provider examples in the research illustrate why you must check the specific contract: Screenshot API documents plan limits and rate/quota headers; ScreenshotEngine lists plan-specific monthly screenshot and per-minute request limits; Screenshot Studio documents a per-IP request limit and a Retry-After response for 429s. Those figures belong to those providers and can change. Consult [ScreenshotEngine’s error and limit documentation](https://www.screenshotengine.com/docs/errors-and-limits) and [Screenshot Studio’s authentication documentation](https://www.screenshot-studio.com/docs/authentication) for their current behavior.

4. Implement a queue with bounded retries

The following Python example shows a synchronous worker queue for APIs that return the image in the response body. Replace the endpoint, authentication, and response handling with the provider’s documented API. It deliberately stops on monthly quota exhaustion and permanent client errors, honors Retry-After for 429 responses, and retries only selected transient failures. A provider that returns asynchronous job IDs needs a submission-and-polling adapter instead.

import random
import time
from pathlib import Path
from urllib.parse import urlparse

import requests

API_URL = "https://api.example.com/v1/screenshot"
API_KEY = "YOUR_API_KEY"
URLS = [
    "https://example.com/",
    "https://www.python.org/",
]
OUT = Path("screenshots")
MAX_ATTEMPTS = 5
MAX_BACKOFF_SECONDS = 60

session = requests.Session()


def valid_url(value):
    parsed = urlparse(value)
    return parsed.scheme in {"http", "https"} and bool(parsed.netloc)


def retry_after_seconds(response):
    value = response.headers.get("Retry-After")
    if not value:
        return None
    try:
        return max(0.0, float(value))
    except ValueError:
        # Retry-After may also be an HTTP date. Let the provider-specific
        # adapter parse that form if the API uses it.
        return None


def backoff(attempt):
    ceiling = min(MAX_BACKOFF_SECONDS, 2 ** (attempt - 1))
    return random.uniform(0, ceiling)


def capture(url):
    if not valid_url(url):
        return {"url": url, "state": "invalid_url", "attempts": 0}

    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            response = session.get(
                API_URL,
                params={"access_key": API_KEY, "url": url},
                timeout=(10, 90),
            )
        except (requests.Timeout, requests.ConnectionError) as exc:
            if attempt == MAX_ATTEMPTS:
                return {"url": url, "state": "transient_failure", "error": str(exc), "attempts": attempt}
            time.sleep(backoff(attempt))
            continue

        if response.status_code == 429:
            # Some providers use 429 for monthly quota exhaustion too.
            # Inspect the provider's documented error code/body before retrying.
            code = response.headers.get("X-Error-Code", "").lower()
            if code in {"monthly_quota_exceeded", "quota_exceeded"}:
                return {"url": url, "state": "quota_exhausted", "attempts": attempt}
            wait = retry_after_seconds(response)
            if wait is None:
                wait = backoff(attempt)
            if attempt == MAX_ATTEMPTS:
                return {"url": url, "state": "rate_limited", "attempts": attempt}
            time.sleep(wait)
            continue

        if response.status_code in {408, 500, 502, 503, 504}:
            if attempt == MAX_ATTEMPTS:
                return {"url": url, "state": "transient_failure", "status": response.status_code, "attempts": attempt}
            time.sleep(backoff(attempt))
            continue

        if response.status_code in {400, 401, 403, 404, 422}:
            return {"url": url, "state": "permanent_failure", "status": response.status_code, "detail": response.text[:500], "attempts": attempt}

        response.raise_for_status()
        OUT.mkdir(parents=True, exist_ok=True)
        path = OUT / f"{attempt:02d}-{abs(hash(url))}.png"
        path.write_bytes(response.content)
        return {"url": url, "state": "complete", "path": str(path), "attempts": attempt}

    return {"url": url, "state": "unknown_failure", "attempts": MAX_ATTEMPTS}


# This serial loop is intentionally conservative. Add workers only after
# checking the provider's current limits and pacing policy.
results = []
for item in URLS:
    results.append(capture(item["url"] if isinstance(item, dict) else item))
    # Add a documented inter-request delay here if your provider requires it.

for result in results:
    print(result)

For production use, replace Python’s process-randomized hash() filename with a stable digest or your own job ID; persist results instead of keeping them only in memory. Also inspect the provider’s documented error payload before classifying 429s: status alone may not distinguish a temporary throttle from exhausted monthly quota.

5. Submit batches and track partial results

Batch APIs are provider-specific, so there is no safe universal request body or polling path. The general flow is:

1. Validate URLs and divide them into groups no larger than the documented maximum.
2. Submit a group and persist the returned batch ID and any per-URL IDs immediately.
3. Poll the documented status endpoint or register the supported webhook mechanism.
4. Persist each URL’s terminal result independently. A completed batch can contain failed items.
5. Retry only unfinished items that failed transiently. Do not resubmit successful URLs unless duplicate captures are intended.

Use idempotency keys if the provider documents them. If a submission times out after the server accepted it, blindly resubmitting can create duplicate jobs. First query status using a returned identifier, idempotency key, or provider request ID if available.

Choose polling intervals that do not create another rate-limit problem. Prefer webhooks for long-running jobs when the provider supports them, and verify webhook signatures as documented before trusting events.

6. cURL example for a single screenshot request

For an individual-response API, a cURL request can capture one URL. A shell loop can run the same request sequentially, but a queue worker is preferable when you need persistent state, adaptive pacing, or retry classification. The example endpoint and parameter names below are placeholders; use the provider’s documented values.

curl --fail-with-body --get "https://api.example.com/v1/screenshot" \
  --data-urlencode "access_key=YOUR_API_KEY" \
  --data-urlencode "url=https://example.com/" \
  --output example.png

For a local list in urls.txt, a conservative sequential shell loop is:

while IFS= read -r url; do
  [ -z "$url" ] && continue
  name=$(printf '%s' "$url" | sha256sum | cut -d ' ' -f 1)
  curl --fail-with-body --get "https://api.example.com/v1/screenshot" \
    --data-urlencode "access_key=YOUR_API_KEY" \
    --data-urlencode "url=$url" \
    --output "${name}.png" || printf 'Capture failed: %s\n' "$url" >&2
  # Set this delay from the provider's documented rate policy.
  sleep 1
done < urls.txt

The one-second delay is merely an example of where pacing belongs; it is not a universal safe rate. Do not put a real API key in a shared script or commit it to source control.

7. Node.js queue example

This runnable Node.js example uses built-in fetch, processes items sequentially, and applies bounded retries. It expects a provider that returns image bytes synchronously. Adjust status and quota detection to match the provider’s documented response format.

import { createHash } from 'node:crypto';
import { mkdir, writeFile } from 'node:fs/promises';

const endpoint = 'https://api.example.com/v1/screenshot';
const apiKey = process.env.SCREENSHOT_API_KEY;
const urls = ['https://example.com/', 'https://www.python.org/'];
const maxAttempts = 5;
const maxBackoffMs = 60_000;

if (!apiKey) throw new Error('Set SCREENSHOT_API_KEY');

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
function backoffMs(attempt) {
  const cap = Math.min(maxBackoffMs, 1000 * (2 ** (attempt - 1)));
  return Math.floor(Math.random() * cap);
}
function validUrl(value) {
  try {
    const parsed = new URL(value);
    return ['http:', 'https:'].includes(parsed.protocol);
  } catch {
    return false;
  }
}
function retryAfterMs(response) {
  const value = response.headers.get('retry-after');
  if (!value) return null;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
  const date = Date.parse(value);
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

async function capture(url) {
  if (!validUrl(url)) return { url, state: 'invalid_url', attempts: 0 };
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    let response;
    try {
      const query = new URLSearchParams({ access_key: apiKey, url });
      response = await fetch(`${endpoint}?${query}`, { signal: AbortSignal.timeout(90_000) });
    } catch (error) {
      if (attempt === maxAttempts) return { url, state: 'transient_failure', error: String(error), attempts: attempt };
      await sleep(backoffMs(attempt));
      continue;
    }

    if (response.status === 429) {
      const code = (response.headers.get('x-error-code') || '').toLowerCase();
      if (['monthly_quota_exceeded', 'quota_exceeded'].includes(code)) {
        return { url, state: 'quota_exhausted', attempts: attempt };
      }
      if (attempt === maxAttempts) return { url, state: 'rate_limited', attempts: attempt };
      await sleep(retryAfterMs(response) ?? backoffMs(attempt));
      continue;
    }
    if ([408, 500, 502, 503, 504].includes(response.status)) {
      if (attempt === maxAttempts) return { url, state: 'transient_failure', status: response.status, attempts: attempt };
      await sleep(backoffMs(attempt));
      continue;
    }
    if ([400, 401, 403, 404, 422].includes(response.status)) {
      return { url, state: 'permanent_failure', status: response.status, detail: (await response.text()).slice(0, 500), attempts: attempt };
    }
    if (!response.ok) throw new Error(`Unexpected HTTP ${response.status}: ${(await response.text()).slice(0, 500)}`);

    await mkdir('screenshots', { recursive: true });
    const id = createHash('sha256').update(url).digest('hex').slice(0, 20);
    const bytes = Buffer.from(await response.arrayBuffer());
    await writeFile(`screenshots/${id}.png`, bytes);
    return { url, state: 'complete', path: `screenshots/${id}.png`, attempts: attempt };
  }
}

for (const url of urls) {
  console.log(await capture(url));
  // Apply any provider-required inter-request delay here.
}

8. Reliability: preserve enough state to resume safely

Keep a durable record for each requested URL, including:

• Input URL and normalized URL, plus a stable job/item ID.
• State such as queued, submitted, running, complete, permanent failure, transient failure, or quota exhausted.
• Attempt count, last status/error code, next eligible retry time, and the provider’s request or batch ID when available.
• Output path or object-storage key, content type, and completion timestamp.

Persist state before and after submission so a process restart does not lose accepted work. Make output writes atomic where possible: write to a temporary file and rename only after the full response has been received. For asynchronous batches, reconcile local state with provider status after restart before resubmitting anything.

9. Performance and cost considerations

More workers can shorten elapsed time only until you reach the provider’s allowed request rate, concurrency, or your own storage/network capacity. Start conservatively, then raise worker count based on documented limits and observed usage data. Avoid treating a 429 as a signal to keep increasing concurrency.

Batch submission can cut HTTP submission overhead, but each URL may still count against the monthly screenshot allowance. Estimate work in screenshots, not just API calls. Leave headroom for retries and reruns, and stop or defer jobs when the provider reports depleted quota. Compare plan limits, overage behavior, and reset dates in the account’s current documentation; figures from one provider are not transferable to another.

For large jobs, stream image bodies to disk or object storage rather than holding all screenshots in memory. Limit queued work and apply backpressure when storage is slower than capture. Cache or skip URLs only when the desired freshness permits it.

10. Troubleshooting

Symptom	Likely cause	Fix
HTTP 429	Temporary rate limit, per-URL throttle, or monthly quota exhaustion; providers differ.	Inspect documented error codes and quota/rate headers. Honor Retry-After for temporary limits. Pause until reset or adjust the plan for exhausted monthly quota.
Repeated 429 after waiting	Retry delay ignores the provider’s reset window, multiple workers share one account limit, or the account quota is depleted.	Coordinate pacing across workers and inspect usage/reset fields. Do not let each process independently consume the full limit.
401 or 403	Missing, invalid, expired, or insufficiently scoped credential.	Check the key and required permissions. Do not retry unchanged credentials.
400 or 422	Malformed URL, unsupported option, or invalid batch shape.	Validate request fields against the provider’s API schema and correct the input.
Batch accepted but some screenshots fail	Per-item navigation/render errors or partial batch completion.	Read each item’s result and retry only transiently failed items.
Submission timed out and job may have started	Network failure occurred after provider acceptance.	Look up the batch/job using its ID or idempotency mechanism before submitting again.
Empty or invalid image file	Error JSON/HTML was saved as an image, or capture failed while the client assumed success.	Check HTTP status, content type, and provider error body before writing output.
Requests remain queued	Per-URL processing limits, account concurrency, or provider-side backlog.	Check job status and documented processing limits; reduce submission rate and use status tracking rather than repeated resubmission.
Quota appears to fall faster than request count	Batch items count individually, retries create billable captures, or another worker shares the account.	Compare URL-level accounting with usage data and include all producers in one quota monitor.

Some screenshot APIs report that bot checks, CAPTCHA pages, blank pages, timeouts, or failed loads were not successful captures; others may account for them differently. Verify the selected service’s billing definition and response headers rather than assuming all failures are free.

11. Or skip the browser setup

For a simple one-URL capture, ScreenshotNeo provides a single GET request that returns an image or PDF. Its [API documentation](https://screenshotneo.com/docs/) covers the request options and response behavior. For a multi-URL job, submit one request per URL through your own paced queue, or use the documented bulk capture option for up to 100 URLs per call.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before the shot. Bot checks, blank pages, timeouts, and failed loads are not billed, and cache hits are not billed. Its MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. [See ScreenshotNeo](https://screenshotneo.com) and its [API options](https://screenshotneo.com/docs/) for details.

Sign up free for 1,000 screenshots a month, with no card required.

12. FAQ

Does one batch request count as one screenshot?

Not necessarily. Some providers count each URL in the batch against monthly screenshot quota. Check the provider’s accounting rules.

Should I retry every 429 response?

No. First determine whether it means temporary throttling or exhausted quota. Retry temporary limits according to Retry-After or the documented reset; wait for quota renewal or change the plan when the monthly allowance is depleted.

How many concurrent workers should I use?

There is no cross-provider number. Start with a conservative queue and set concurrency from the provider’s current limits and usage data.

Is polling better than webhooks?

Use the provider’s supported mechanism. Webhooks can avoid frequent status requests for long-running work; polling is useful when events are unavailable, provided polling itself stays within limits.