ScreenshotNeo
Sign in Get a free key

BlogEngineering

Parallel Lighthouse Performance Testing APIs

Compare Google PSI, Lighthouse CI, and regional APIs, then build reliable parallel Lighthouse tests with bounded concurrency and repeat runs.

By the ScreenshotNeo team1 October 20268 min read

Direct answer: For parallel Lighthouse testing, use Google PageSpeed Insights (PSI) with your own bounded worker pool when you need the official hosted runner; use Lighthouse CI when you want URL arrays, repeat runs, and CI integration; use a regional Lighthouse service when geography and monitoring are first-class requirements. Keep concurrency bounded, run each URL several times, aggregate with medians or percentiles, and label every result with URL, strategy, device, Lighthouse version, region, and fetch time.

PSI exposes GET https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed. Each request analyzes one URL and accepts category and desktop/mobile strategy parameters. Lighthouse CI’s psiCollectCron accepts URL arrays and maxNumberOfParallelUrls; its documented numberOfRuns default is 5. PSI requires publicly reachable URLs, while Lighthouse CI’s Node method can test private environments. A hosted regional API can create one check per region from a single request.

Choose an API by workload

Option Execution Parallelism Repeat runs Geography Private URLs Best fit
Google PageSpeed Insights API Google-hosted runner Your client fans out requests Implement in your client Google runner location No Direct, official PSI data
Lighthouse CI psiCollectCron Google PSI through LHCI maxNumberOfParallelUrls; default is Infinity numberOfRuns; default is 5 PSI locations No in PSI mode Scheduled or CI collections over URL arrays
Lighthouse CI Node mode Your Node/browser environment Control workers in your CI Configure repeated runs Your runner’s location Yes Staging, localhost, authenticated internal pages
Lighthouse Metrics API Third-party hosted regions One check can fan out to regions One run per region Regions array Depends on service access Geographic comparisons and monitoring

What “parallel” means in each system

Google PageSpeed Insights

The PSI endpoint analyzes one URL per request. For 100 URLs, issue 100 requests through a bounded queue, observe service quotas, and retry transient failures with exponential backoff. A request can select categories, locale, and strategy=desktop or strategy=mobile. See the PageSpeed Insights API endpoint for the request surface.

Lighthouse CI

In psiCollectCron, put URLs in sites[i].urls. Set an explicit maxNumberOfParallelUrls; the documented Infinity default can create an unexpectedly large burst. Set numberOfRuns explicitly when repeatability matters. The same configuration supports category arrays and mobile or desktop strategy.

Regional hosted checks

Lighthouse Metrics API’s POST /v1/lighthouse/checks accepts a URL and a regions array. The service creates one run per region. Optional Lighthouse version and device settings let you compare controlled configurations. Bearer authentication is required, and endpoint rate limits can return HTTP 429.

Run parallel PSI checks with a bounded worker pool

This Python example runs desktop and mobile checks for several public URLs with a fixed concurrency limit. Add your API authentication as required by your Google Cloud project and service quota.

import asyncio
import os
from typing import Iterable

import httpx

ENDPOINT = "https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed"
API_KEY = os.environ["PAGESPEED_API_KEY"]
URLS = [
    "https://example.com/",
    "https://example.com/pricing",
    "https://example.com/docs",
]
CONCURRENCY = 4
CATEGORIES = ["performance", "accessibility", "best-practices", "seo"]

async def run_one(client: httpx.AsyncClient, url: str, strategy: str):
    params = {
        "url": url,
        "strategy": strategy,
        "locale": "en",
        "key": API_KEY,
    }
    # httpx repeats category parameters when passed as a list of tuples.
    query = list(params.items()) + [("category", category) for category in CATEGORIES]
    for attempt in range(5):
        response = await client.get(ENDPOINT, params=query)
        if response.status_code not in (429, 500, 502, 503, 504):
            response.raise_for_status()
            return {"url": url, "strategy": strategy, "data": response.json()}
        await asyncio.sleep(2 ** attempt)
    response.raise_for_status()

async def bounded(items: Iterable[tuple[str, str]]):
    semaphore = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient(timeout=120) as client:
        async def wrapped(url: str, strategy: str):
            async with semaphore:
                return await run_one(client, url, strategy)
        return await asyncio.gather(*(wrapped(url, strategy) for url, strategy in items))

results = asyncio.run(
    bounded((url, strategy) for url in URLS for strategy in ("desktop", "mobile"))
)
for result in results:
    lighthouse = result["data"].get("lighthouseResult", {})
    print(result["strategy"], result["url"], lighthouse.get("categories", {}).get("performance", {}).get("score"))

Equivalent cURL request

curl -G "https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed" \
  --data-urlencode "url=https://example.com/" \
  --data "strategy=mobile" \
  --data "category=performance" \
  --data "category=seo" \
  --data "key=$PAGESPEED_API_KEY" \
  -o psi-mobile.json

Node.js fan-out with a concurrency cap

const endpoint = 'https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed';
const apiKey = process.env.PAGESPEED_API_KEY;
const urls = ['https://example.com/', 'https://example.com/pricing'];
const limit = 4;

async function request(url, strategy) {
  const q = new URLSearchParams({ url, strategy, key: apiKey, locale: 'en' });
  q.append('category', 'performance');
  q.append('category', 'seo');
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${endpoint}?${q}`);
    if (![429, 500, 502, 503, 504].includes(res.status)) {
      if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
      return { url, strategy, body: await res.json() };
    }
    await new Promise(r => setTimeout(r, 2 ** attempt * 1000));
  }
  throw new Error(`Retries exhausted for ${url} (${strategy})`);
}

async function mapWithLimit(items, workerLimit) {
  const output = [];
  let next = 0;
  async function worker() {
    while (true) {
      const index = next++;
      if (index >= items.length) return;
      output[index] = await request(items[index].url, items[index].strategy);
    }
  }
  await Promise.all(Array.from({ length: workerLimit }, worker));
  return output;
}

const items = urls.flatMap(url => ['desktop', 'mobile'].map(strategy => ({ url, strategy })));
const results = await mapWithLimit(items, limit);
console.log(results.map(r => ({ url: r.url, strategy: r.strategy })));

Configure Lighthouse CI for URL arrays

A minimal PSI collection configuration looks like this:

module.exports = {
  ci: {
    collect: {
      psiCollectCron: {
        sites: [
          {
            urls: [
              'https://example.com/',
              'https://example.com/pricing',
              'https://example.com/docs'
            ]
          }
        ],
        maxNumberOfParallelUrls: 4,
        numberOfRuns: 5,
        settings: {
          strategy: 'mobile',
          onlyCategories: ['performance', 'accessibility', 'best-practices', 'seo']
        }
      }
    }
  }
};

Use a finite parallelism value that your quota and CI environment can sustain. Keep numberOfRuns high enough to reduce noise, then select a median or percentile for gates while retaining all raw runs for diagnosis. PSI collection cannot reach private URLs; switch to Lighthouse CI’s Node method for internal or local targets.

Run one check across regions

For a hosted regional service, send a URL and regions in one authenticated request. The exact region identifiers and optional device/version fields come from the provider account.

curl -X POST "https://api.example.com/v1/lighthouse/checks" \
  -H "Authorization: Bearer $LIGHTHOUSE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/",
    "regions": ["us-east", "eu-west", "asia-east"],
    "device": "mobile"
  }'

Expect HTTP 429 when the regional endpoint rate limit is exceeded. Retry with backoff, and avoid pooling scores from different regions, devices, or Lighthouse versions without preserving those labels.

Make parallel results statistically useful

  • Repeat each condition. CPU contention, network timing, cache state, and third-party scripts create run-to-run variation.
  • Use medians or percentiles. Lighthouse guidance says the median score of five runs is twice as stable as one run.
  • Keep dimensions separate. Do not combine desktop and mobile, regions, devices, or Lighthouse versions into one aggregate.
  • Store raw evidence. Retain the JSON report, URL, strategy, device, version, region, timestamp, and fetch status.
  • Gate conservatively. Base CI decisions on a representative median or a percentile threshold, and investigate outliers separately.

Performance, reliability, and cost considerations

Concern Practical handling
Throughput Use bounded workers. More simultaneous jobs can increase quota errors and make the target or runner contend for resources.
Rate limits Handle 429 and transient 5xx responses with exponential backoff and a maximum retry count.
Timeouts Use a client timeout longer than a normal page load, but cap retries so one URL cannot block the batch indefinitely.
Quota and billing Check your Google API quota and each hosted vendor’s plan limits before increasing concurrency. These limits can change.
Report retention Save raw JSON and metadata in your own artifact store when long-term comparisons matter.
Regional fidelity Use a regional service when location is part of the question; PSI fan-out alone does not guarantee one run per geography.

Troubleshooting parallel Lighthouse jobs

HTTP 429 Too Many Requests

Cause: Your request rate exceeded a quota or vendor limit. Fix: Lower worker count, add exponential backoff with jitter, and inspect quota headers or account limits.

PSI cannot analyze a staging URL

Cause: PSI collection requires a publicly accessible URL. Fix: Use Lighthouse CI Node mode inside the private network.

Scores vary widely between runs

Cause: Lighthouse measurements are noisy and affected by network, CPU, cache, and page changes. Fix: Run five or more samples, report medians or percentiles, and retain raw runs.

Parallel jobs overload CI or the target

Cause: An unbounded queue, including Lighthouse CI’s documented Infinity default, starts too many audits. Fix: Set an explicit maxNumberOfParallelUrls or worker limit.

Comparisons mix incompatible results

Cause: Results use different strategy, device, region, or Lighthouse version. Fix: Include those fields in the result key and compare like with like.

A regional request returns 401 or 403

Cause: Missing or invalid bearer authentication. Fix: Send the token in the Authorization: Bearer header and verify its scope.

A run times out on a single URL

Cause: The page may have slow resources, redirects, or scripts that never settle. Fix: isolate the URL, inspect its trace, apply a bounded timeout, and retry only transient failures.

Or skip the browser setup

ScreenshotNeo is a website screenshot API and MCP server when your workflow needs clean visual captures alongside performance checks. It accepts one GET request and returns PNG, JPEG, WebP, or PDF. Cookie and consent banners, newsletter popups, and chat widgets are removed before capture; bot checks, blank pages, timeouts, failed loads, and cache hits are not billed. Its MCP server lets Claude, Cursor, and other MCP clients call take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots per month with no card, and paid plans start at $5 for 3,000.

See the ScreenshotNeo API documentation for all options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Create a free ScreenshotNeo account with 1,000 screenshots each month and no card.

FAQ

Can I send many URLs in one PSI request?

No. PSI analyzes one URL per request, so bulk testing requires client-side fan-out or Lighthouse CI.

Should I maximize concurrency?

No. Choose a bounded value that respects quotas and keeps the runner and target stable.

How many runs should a CI gate use?

Five is Lighthouse CI’s documented default for repeated collection. Use a median or percentile and keep raw runs.

Can PSI test localhost?

No. Use Lighthouse CI Node mode for private or local pages.

When do regions matter?

Use regional checks when latency or content differs by geography. Keep each region as a separate labeled result.

Are screenshot APIs a replacement for Lighthouse?

No. Lighthouse measures performance and quality audits; a screenshot API captures visual output. They can be used together in a monitoring pipeline.

Keep reading