Scrape Any Website to JSON with CSS Selectors

Scrapy translates CSS queries to XPath internally, so XPath is available when CSS cannot express a relationship clearly. Its feed exports can write JSON, JSON Lines, CSV, or XML; -O products.json overwrites the output, while -o products.json appends according to the feed format.

4. A small Python extractor for one page

For a single static page, Requests plus Beautiful Soup is often enough. This example implements text, attributes, repeated records, URL resolution, and explicit nulls.

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"
response = requests.get(
    url,
    headers={"User-Agent": "Mozilla/5.0 (compatible; JsonExtractor/1.0)"},
    timeout=30,
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

def text_or_none(node):
    return node.get_text(" ", strip=True) if node else None

def attr_or_none(node, name):
    return node.get(name) if node and node.has_attr(name) else None

result = {
    "title": text_or_none(soup.select_one("h1")),
    "canonical_url": urljoin(url, attr_or_none(soup.select_one('link[rel="canonical"]'), "href"))
                     if attr_or_none(soup.select_one('link[rel="canonical"]'), "href") else None,
    "products": [],
}

for card in soup.select(".product-card"):
    href = attr_or_none(card.select_one("a.product-link"), "href")
    image = attr_or_none(card.select_one("img"), "src")
    result["products"].append({
        "name": text_or_none(card.select_one(".product-name")),
        "url": urljoin(url, href) if href else None,
        "price": text_or_none(card.select_one(".price")),
        "image": urljoin(url, image) if image else None,
    })

import json
print(json.dumps(result, ensure_ascii=False, indent=2))

This fetches returned HTML only. It does not execute page JavaScript, click controls, wait for network activity, or maintain a browser session.

5. JavaScript-rendered pages and wait conditions

Use a rendered browser when the field appears only after JavaScript executes. A useful readiness strategy is, in order:

1. Wait for a selector that proves the required data exists, such as [data-loaded="true"] or .product-card.
2. Use network-idle waiting when no stable readiness selector exists.
3. Add a short fixed delay only for pages with animations or delayed third-party widgets.

Cloudflare Browser Run documents gotoOptions.waitUntil values such as networkidle0 and networkidle2, plus waitForSelector. Browserless documents selectors against the fully rendered DOM. Microlink describes rules that run on a rendered page when needed. Rendering too early produces empty arrays; waiting forever makes jobs slow and unreliable.

// Generic browser pseudocode
await page.goto(url, {waitUntil: "networkidle2"});
await page.waitForSelector(".product-card", {timeout: 15000});
const data = await page.$$eval(".product-card", cards => cards.map(card => ({
  name: card.querySelector(".product-name")?.textContent.trim() ?? null,
  url: card.querySelector("a.product-link")?.href ?? null
})));

For infinite scroll, repeatedly scroll and collect records until the count stops increasing or a next-page control disappears. Set a maximum page count and record count so a broken page cannot run forever.

6. Selector patterns that cover common fields

Use fallback selectors only when they represent the same concept. If both selectors match different elements, validate and choose the first non-empty value rather than silently merging unrelated text.

7. cURL, Node.js, and hosted extraction choices

cURL is useful for checking the raw response and headers:

curl -L --fail --compressed \
  -A 'Mozilla/5.0 (compatible; JsonExtractor/1.0)' \
  'https://example.com/products' \
  -o page.html

It cannot execute JavaScript or apply CSS selectors by itself. Pipe the saved document into a parser such as Scrapy, Beautiful Soup, Cheerio, or another DOM library.

import * as cheerio from "cheerio";

const response = await fetch("https://example.com/products");
if (!response.ok) throw new Error(`${response.status} ${response.statusText}`);
const html = await response.text();
const $ = cheerio.load(html);

const data = {
  title: $("h1").first().text().trim() || null,
  products: $(".product-card").map((_, card) => ({
    name: $(card).find(".product-name").first().text().trim() || null,
    url: $(card).find("a.product-link").attr("href") || null,
    price: $(card).find(".price").first().text().trim() || null
  })).get()
};
console.log(JSON.stringify(data, null, 2));

Hosted extraction APIs reduce browser and parser operations to one request. Compare them on JavaScript rendering, selector grammar, nested and repeated data, type conversion, null behavior, wait controls, authentication, proxy and session support, output formats, quotas, and operational ownership. Scrapy is a good fit when you need custom crawling, pipelines, retries, or on-premise execution. A hosted service is useful when you want fetch, render, and extraction in one request.

8. Or skip the browser setup

ScreenshotNeo is a website screenshot API and MCP server. It can render a page before you inspect or archive it, with waits for a selector, delay, or network idle; custom headers, cookies, user agents and authorization; custom JavaScript and CSS; click and hide-selector actions; and request or resource blocking. Those controls help you capture the same rendered state that a selector-based scraper needs, while the extraction itself remains your JSON parser’s job. See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/products -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://example.com/products"}, timeout=90)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://example.com/products' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`${res.status} ${res.statusText}`);
const bytes = new Uint8Array(await res.arrayBuffer());
await Bun.write("shot.webp", bytes);

Cookie banners, newsletter popups, and chat widgets are removed before the shot. Bot checks, blank pages, timeouts, failed loads, and cache hits cost nothing; response headers report X-Page-Verdict and X-Billed. The MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients. The free plan includes 1,000 screenshots each month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

9. Reliability checklist

• Store the source URL, fetch time, HTTP status, and parser version with every record.
• Prefer semantic selectors, IDs, data attributes, and schema markup.
• Keep a fallback selector for known template variants.
• Measure null rates per field and alert when they change sharply.
• Validate arrays, numeric ranges, URLs, and required keys before writing output.
• Use bounded retries with backoff for transient network errors.
• Cache unchanged pages where your freshness requirements allow it.
• Respect terms of service, robots directives, rate limits, and applicable law.
• Keep authentication data and cookies out of logs.

10. Performance, cost, and scale

Raw HTTP fetching is usually faster and cheaper than launching a browser. Browser rendering is necessary when content is client-rendered, but it adds startup, JavaScript, network, and waiting time. Reduce work by selecting only required fields, avoiding unnecessary resources, reusing browser sessions where supported, and setting a maximum wait.

For large jobs, queue URLs, limit concurrency per host, deduplicate URLs, and write JSON Lines incrementally so one failed page does not lose the batch. Track cost as requests plus browser time or hosted API units, and distinguish cache hits from fresh captures. A successful HTTP response can still contain an empty field after a redesign, so correctness metrics matter as much as latency.

11. Troubleshooting

12. FAQ

Can CSS selectors extract JSON directly?

Selectors identify and extract DOM values; your code or a hosted API maps those values into JSON. The schema defines names, nesting, repetition, types, and missing-value behavior.

What is the difference between text and HTML extraction?

Text extraction returns readable text nodes. HTML extraction preserves markup and is useful when formatting or embedded links must be retained. Choose one explicitly for each field.

How do I scrape pages behind login?

Use an authorized session, cookies, headers, or an authenticated browser context. Keep credentials secret and confirm that your use complies with the site’s rules.

Should I use Scrapy or a hosted API?

Choose Scrapy for control over crawling, pipelines, retries, and deployment. Choose a hosted API when browser rendering and extraction operations should be managed as a service. Compare rendering, schema support, wait controls, quotas, cost, and ownership before deciding.

How do I know a scraper still works?

Validate required fields, monitor null and error rates, save representative fixtures, and run selector checks against each important template variant.