How to Turn Web Scrapers into Data APIs

Short answer: keep your scraper workers separate from the public API. The API authenticates callers, validates a request, creates a run, and returns either data immediately for short jobs or a job ID for longer work. Workers execute Scrapy spiders or browser code, normalize records, persist results, and expose status and paginated exports. Version the response schema so parser changes do not silently break clients.

This design gives consumers a stable contract while your extraction code changes behind it. It also lets you add queues, retries, schedules, rate limits, and multiple site adapters without turning one HTTP request into an unreliable monolith.

1. Define the scraper-to-API contract

Start with the interface that callers will depend on. A useful request includes a target, an operation, and optional extraction parameters. A useful response contains stable metadata as well as records.

POST /v1/runs
Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

{
  "spider": "product_search",
  "start_urls": ["https://example.com/products"],
  "options": {"query": "laptop", "max_items": 100}
}

For a short scrape, the endpoint can return 200 OK and a result body. For a queued scrape, return 202 Accepted:

{
  "run_id": "run_01J...",
  "status": "queued",
  "status_url": "/v1/runs/run_01J...",
  "created_at": "2026-09-30T12:00:00Z",
  "schema_version": "2026-01"
}

Every item should use explicit, documented fields. Include the source URL, retrieval timestamp, parser or schema version, and a stable item identifier. Represent missing values as JSON null rather than changing field types or omitting fields unpredictably.

Suggested result envelope

{
  "run_id": "run_01J...",
  "status": "succeeded",
  "schema_version": "2026-01",
  "started_at": "2026-09-30T12:00:02Z",
  "finished_at": "2026-09-30T12:00:18Z",
  "source_urls": ["https://example.com/products"],
  "item_count": 42,
  "items": [
    {
      "id": "product-123",
      "name": "Example laptop",
      "price": 999.0,
      "currency": "USD",
      "source_url": "https://example.com/products/123",
      "retrieved_at": "2026-09-30T12:00:07Z"
    }
  ],
  "next_cursor": null,
  "error": null
}

Keep site-specific CSS selectors, XPath expressions, browser actions, and retry rules inside adapters. The public contract should remain stable when one target changes. A parser failure must be visible as a failed run; never return an apparently successful response containing silently partial data.

2. Separate the API, queue, workers, and storage

A production layout normally has four components:

A durable scraper API separates the public contract from queued extraction workers and dataset storage.

API layer: validates input, authenticates the caller, creates a run, and reads status or exports.
Queue: carries run IDs to workers and absorbs bursts.
Extraction workers: run Scrapy spiders or browser automation and emit normalized items.
Storage: stores run metadata and item datasets for pagination and export.

Scrapy’s core entry point is the Crawler object, and its optimization guidance recommends controlling concurrency and download delay while respecting robots.txt and directives such as Crawl-delay or Request-rate. Excessive pressure can cause throttling, errors, or bans. An API or bulk export is faster for you and cheaper for the website than crawling pages when one is available. See the Scrapy settings documentation and Scrapy core API.

A minimal FastAPI boundary

The following example is a runnable API skeleton. Replace the in-memory dictionaries and background function with Redis, a durable queue, and a database before production.

from datetime import datetime, timezone
from typing import Any
from fastapi import Depends, FastAPI, Header, HTTPException, status
from pydantic import BaseModel, HttpUrl
from uuid import uuid4

app = FastAPI(title="Scraper Data API", version="1.0.0")
RUNS: dict[str, dict[str, Any]] = {}
API_KEYS = {"scraper_demo_key"}

class RunRequest(BaseModel):
    spider: str
    start_urls: list[HttpUrl]
    options: dict[str, Any] = {}

async def require_key(authorization: str | None = Header(default=None)):
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="missing bearer token")
    key = authorization.removeprefix("Bearer ")
    if key not in API_KEYS:
        raise HTTPException(status_code=401, detail="invalid API key")
    return key

@app.post("/v1/runs", status_code=status.HTTP_202_ACCEPTED)
async def create_run(req: RunRequest, _: str = Depends(require_key)):
    run_id = "run_" + uuid4().hex
    now = datetime.now(timezone.utc).isoformat()
    RUNS[run_id] = {
        "run_id": run_id,
        "status": "queued",
        "schema_version": "2026-01",
        "created_at": now,
        "items": [],
        "error": None,
    }
    # enqueue(run_id, req.model_dump()) in a real deployment
    return {"run_id": run_id, "status": "queued", "status_url": f"/v1/runs/{run_id}"}

@app.get("/v1/runs/{run_id}")
async def get_run(run_id: str, _: str = Depends(require_key)):
    run = RUNS.get(run_id)
    if not run:
        raise HTTPException(status_code=404, detail="run not found")
    return {k: v for k, v in run.items() if k != "items"}

@app.get("/v1/runs/{run_id}/items")
async def get_items(run_id: str, limit: int = 100, cursor: int = 0, _: str = Depends(require_key)):
    if limit < 1 or limit > 1000:
        raise HTTPException(status_code=422, detail="limit must be 1..1000")
    run = RUNS.get(run_id)
    if not run:
        raise HTTPException(status_code=404, detail="run not found")
    items = run["items"][cursor:cursor + limit]
    next_cursor = cursor + limit if cursor + limit < len(run["items"]) else None
    return {"items": items, "next_cursor": next_cursor}

Run it with uvicorn app:app --reload. In a real worker, transition runs through queued, running, succeeded, failed, or cancelled. Persist the last error and timestamps so clients can make safe retry decisions.

3. Authenticate at the boundary

Use HTTPS and keep credentials out of URLs, logs, browser code, and source repositories. A common pattern is Authorization: Bearer ...; an X-API-Key header can be supported for compatibility. Missing or invalid keys should return 401. Derive tenant ownership from the key and apply it to every run, status, dataset, and export lookup. FastAPI's security utilities add API-key schemes to generated OpenAPI documentation.

Give each client a scoped key, rotate keys, and record an owner and creation time. Add quotas and per-key concurrency limits before exposing expensive browser work. Never accept an account identifier supplied only in the request body as the authorization boundary.

4. Choose synchronous or asynchronous execution

Mode	Use it when	Response pattern
Synchronous	A small scrape reliably finishes inside your HTTP timeout	`200` with normalized items
Asynchronous	Pagination, many URLs, browser rendering, or uncertain latency is involved	`202` with run ID, then poll and export

Do not keep a request open while waiting on a long crawl. Queue the work, return a run ID, and let clients poll with backoff or receive a signed webhook. A managed platform such as Scrapy Cloud documents separate synchronous and asynchronous execution, run polling, dataset retrieval, and schedules; use its current API reference for exact paths and limits.

Polling with cURL

curl -X POST https://api.example.com/v1/runs \
  -H 'Authorization: Bearer scraper_demo_key' \
  -H 'Content-Type: application/json' \
  -d '{"spider":"product_search","start_urls":["https://example.com/products"]}'

curl https://api.example.com/v1/runs/run_01J... \
  -H 'Authorization: Bearer scraper_demo_key'

curl 'https://api.example.com/v1/runs/run_01J.../items?limit=100' \
  -H 'Authorization: Bearer scraper_demo_key'

Python client

import time
import requests

base = "https://api.example.com/v1"
headers = {"Authorization": "Bearer scraper_demo_key"}
r = requests.post(f"{base}/runs", headers=headers, json={
    "spider": "product_search",
    "start_urls": ["https://example.com/products"]
}, timeout=30)
r.raise_for_status()
run_id = r.json()["run_id"]

for _ in range(60):
    s = requests.get(f"{base}/runs/{run_id}", headers=headers, timeout=30)
    s.raise_for_status()
    state = s.json()
    if state["status"] in {"succeeded", "failed", "cancelled"}:
        break
    time.sleep(2)

if state["status"] != "succeeded":
    raise RuntimeError(state.get("error") or state["status"])
items = requests.get(f"{base}/runs/{run_id}/items", headers=headers, timeout=30).json()
print(items)

Node.js client

const headers = {
  'Authorization': 'Bearer scraper_demo_key',
  'Content-Type': 'application/json'
};
const created = await fetch('https://api.example.com/v1/runs', {
  method: 'POST', headers,
  body: JSON.stringify({ spider: 'product_search', start_urls: ['https://example.com/products'] })
});
if (!created.ok) throw new Error(await created.text());
const { run_id } = await created.json();
let state;
for (let i = 0; i < 60; i++) {
  const res = await fetch(`https://api.example.com/v1/runs/${run_id}`, { headers });
  state = await res.json();
  if (['succeeded', 'failed', 'cancelled'].includes(state.status)) break;
  await new Promise(resolve => setTimeout(resolve, 2000));
}
if (state.status !== 'succeeded') throw new Error(state.error || state.status);
const data = await fetch(`https://api.example.com/v1/runs/${run_id}/items`, { headers });
console.log(await data.json());

5. Make exports and schemas durable

Return JSON for ordinary API calls and offer CSV or JSONL for bulk consumers. Paginate by an opaque cursor rather than an array offset when datasets can grow or be appended. Keep a schema version in every response and publish a migration policy. Add contract tests that assert field names, types, nullability, and error shapes.

Store raw response samples and adapter versions for debugging. If a selector stops matching, mark the run failed and alert on item-count changes or parse-error rates. Never convert a blocked page, CAPTCHA, or empty response into a successful empty dataset without recording the reason.

6. Rate limits, retries, and site policy

Read each target's terms, authentication requirements, and robots.txt. Translate Crawl-delay or Request-rate into worker delay and concurrency. Treat HTTP 429 as a first-class result. Use bounded exponential backoff with jitter, cap attempts, and persist the final error. Retry transient DNS, connection, and 5xx failures; do not blindly retry deterministic 4xx responses or parser errors.

delay = min(60, 2 ** attempt) + random.uniform(0, 0.5)
# retry at most 5 times; record status code and response headers

7. Scheduling and observability

For recurring datasets, create schedules that enqueue runs rather than running spiders inside a scheduler process. Record queue wait, fetch duration, item count, HTTP status distribution, retry count, and parser version. Alert on repeated failures, unusual zero-item runs, and sustained 429 responses. A run detail endpoint should expose enough information for an operator to diagnose a failure without searching worker logs.

8. Self-hosted workers versus a managed scraper API

Concern	Self-hosted Scrapy workers	Managed scraper API
Control	Full code, network, dependencies, and deployment control	Provider controls runtime; you integrate documented endpoints
Maintenance	You maintain browsers, proxies, queues, and site adapters	Less infrastructure work, but provider limits and pricing apply
Execution	You design sync, async, polling, and webhooks	Often includes run and dataset APIs
Exports	Implement pagination and JSON/CSV/JSONL yourself	Usually provided as dataset retrieval features
Policy fit	You are responsible for robots, terms, and request pressure	You still remain responsible for lawful use and target rules

Choose based on control over the network environment, maintenance capacity, tenant isolation, scheduling, observability, rate-limit handling, and per-result cost. No scraper service should be treated as a promise to bypass blocks or terms of service.

9. Or skip the browser setup

If your job is to capture a page as an image or PDF rather than extract records, ScreenshotNeo provides a single HTTP endpoint. It accepts 63 options including full-page capture with lazy images, CSS-selector element capture, dark mode, device presets, retina scale, PDF paper and margin settings, custom CSS and JavaScript, click and wait actions, blocked resource types, headers, cookies, user agents, timezone, geolocation, transparent backgrounds, resizing, caching, signed links, async jobs, webhooks, bulk capture, and a usage API. See the ScreenshotNeo documentation for parameter details.

Consent banners and overlays can be handled before a screenshot is returned.

cURL

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie and consent banners, newsletter popups, and chat widgets are removed before the shot. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result. Its MCP server lets Claude, Cursor, and other MCP clients call take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

10. Troubleshooting checklist

Symptom	Likely cause	Fix
401 from the API	Missing, malformed, expired, or wrongly scoped key	Send the Bearer header, rotate the key, and verify tenant ownership
202 run never completes	Worker cannot reach the queue or target	Inspect queue depth, worker heartbeat, timeout, and last error
Empty successful dataset	Selector drift, consent wall, or blocked response	Save raw HTML, validate selectors, and report a visible parser or access error
Repeated 429 responses	Concurrency or delay exceeds target tolerance	Reduce concurrency, honor crawl directives, and apply jittered backoff
Clients break after a deploy	Fields or types changed without a schema version	Keep the old version, add a new version, and publish migration notes
Duplicate records	Retries or pagination replayed a page	Use deterministic item IDs and upserts keyed by source plus identifier

11. Performance, reliability, and cost notes

Use queues and bounded worker concurrency so bursts do not overload targets or your own database.
Cache immutable pages and normalize once; expose cached datasets with a retrieval timestamp.
Set separate connect, read, and overall job timeouts. A single large timeout hides stuck workers.
Batch exports as JSONL or compressed files when item counts are large.
Measure cost per successful item, proxy or browser usage, storage, and retry volume rather than only API request count.
Make run creation idempotent with a client supplied idempotency key when duplicate submissions would be expensive.

FAQ

Should every scrape be asynchronous?

No. Keep a synchronous path for small, predictable requests, but make asynchronous execution the default for browser rendering, pagination, batches, and uncertain sites.

Can I expose raw HTML as well as structured data?

Yes, if policy and storage requirements allow it. Keep raw artifacts private, apply retention limits, and make normalized fields the stable public contract.

How do I notify clients without polling?

Offer signed webhooks for terminal run states. Include the run ID, status, schema version, and a URL to fetch the dataset; clients should still verify the signature and retrieve results over HTTPS.

What belongs in an OpenAPI document?

Document authentication, request validation, every status code, pagination cursors, error fields, schema versions, and runnable examples. Generate it from the API layer and review it as part of each contract change.