ScreenshotNeo

BlogHow-to

Parsing JSON in Web Scraping

Learn how to parse API JSON, extract JSON-LD from HTML, validate responses, handle failures, and build reliable scraping pipelines.

By the ScreenshotNeo team30 September 20269 min read

Parsing JSON in Web Scraping

Parsing JSON in web scraping starts with one question: did the server return JSON as the response, or did it return HTML that contains JSON inside a script element? Those cases need different code. A reliable scraper also checks HTTP status separately from JSON decoding, validates the resulting data shape, and handles empty, malformed, or dynamically generated content.

This guide shows complete Python, cURL, and Node.js workflows for both response JSON and JSON embedded in HTML, including JSON-LD. It also covers encoding, parser selection, dynamic pages, retries, performance, cost, troubleshooting, and a practical validation checklist.

1. Identify where the JSON lives

There are two common shapes:

Choose a parsing path based on whether JSON is the response or embedded in HTML.
Choose a parsing path based on whether JSON is the response or embedded in HTML.
Situation What you receive How to parse
JSON endpoint Body is JSON, usually with application/json Decode the response body directly
HTML with embedded data Markup containing a <script>, attribute, or page state object Parse HTML, select the data-bearing element, then decode its text
JavaScript-rendered data Initial HTML lacks the needed values Find the underlying data request or render the page before extraction

JSON-LD is a JSON-based format for serializing Linked Data. A page can contain several JSON-LD blocks, and not every script element is JSON-LD. Select script[type="application/ld+json"] explicitly and use a JSON-LD-aware processor when linked-data semantics such as contexts and expanded entities matter. See the W3C JSON-LD 1.1 Recommendation.

2. Parse JSON returned directly by an endpoint

Always perform these operations in order:

  1. Send the request and retain status, headers, and body.
  2. Check the expected HTTP status (or call raise_for_status()).
  3. Decode JSON.
  4. Validate that the decoded value has the expected type and keys.

A valid JSON error document can accompany a failed status. Requests explicitly warns that the success of r.json() does not indicate that the response succeeded; status handling is a separate check (Requests JSON response documentation).

Python with Requests

import requests
from typing import Any


def fetch_json(url: str) -> dict[str, Any]:
    response = requests.get(
        url,
        headers={"Accept": "application/json"},
        timeout=(10, 60),
    )
    response.raise_for_status()

    try:
        data = response.json()
    except ValueError as exc:
        sample = response.text[:200].replace("\n", " ")
        raise RuntimeError(f"Invalid JSON from {url}: {sample!r}") from exc

    if not isinstance(data, dict):
        raise TypeError(f"Expected a JSON object, got {type(data).__name__}")
    return data

payload = fetch_json("https://example.com/api/products")
products = payload.get("products", [])
if not isinstance(products, list):
    raise TypeError("products must be an array")
for product in products:
    if isinstance(product, dict):
        print(product.get("name"))

cURL

curl --fail-with-body --silent --show-error \
  -H 'Accept: application/json' \
  'https://example.com/api/products' \
  -o response.json

python -m json.tool response.json

--fail-with-body makes HTTP failures visible while preserving the response body for diagnosis. For a shell pipeline, check the exit status before passing the file to another parser.

Node.js

const res = await fetch('https://example.com/api/products', {
  headers: { Accept: 'application/json' },
  signal: AbortSignal.timeout(60_000),
});

const raw = await res.text();
if (!res.ok) {
  throw new Error(`HTTP ${res.status}: ${raw.slice(0, 200)}`);
}

let data;
try {
  data = JSON.parse(raw);
} catch (error) {
  throw new Error(`Invalid JSON: ${error.message}`);
}

if (!data || typeof data !== 'object' || Array.isArray(data)) {
  throw new Error('Expected a JSON object');
}
console.log(data.products ?? []);

3. Extract JSON from an HTML page

When the response is HTML, do not run json.loads() on the entire page. Parse the document first, select the exact element, then decode its text. Beautiful Soup documents that malformed markup can produce different trees with different parsers, so specify one consistently (Beautiful Soup parser documentation).

Python: JSON-LD and an embedded state object

import json
import requests
from bs4 import BeautifulSoup

url = "https://example.com/article"
r = requests.get(url, timeout=(10, 60), headers={"User-Agent": "my-research-bot/1.0"})
r.raise_for_status()
soup = BeautifulSoup(r.content, "html.parser")

jsonld_items = []
for tag in soup.select('script[type="application/ld+json"]'):
    text = tag.string or tag.get_text()
    if not text.strip():
        continue
    try:
        value = json.loads(text)
    except json.JSONDecodeError as exc:
        print(f"Skipping malformed JSON-LD: {exc}")
        continue
    jsonld_items.extend(value if isinstance(value, list) else [value])

for item in jsonld_items:
    if isinstance(item, dict):
        print(item.get("@type"), item.get("headline"))

# Example for a site-specific serialized state script:
state_tag = soup.select_one("script#__NEXT_DATA__")
if state_tag:
    state = json.loads(state_tag.string or state_tag.get_text())
    print(state.keys())

JSON-LD may be an object, an array, or an object containing a @graph array. Treat the shape as untrusted input. If you need linked-data operations rather than simple field extraction, use a JSON-LD processor and follow the W3C processing algorithms (JSON-LD 1.1 API).

Node.js: Cheerio and JSON.parse

import * as cheerio from 'cheerio';

const res = await fetch('https://example.com/article');
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const html = await res.text();
const $ = cheerio.load(html);

const records = [];
$('script[type="application/ld+json"]').each((_, el) => {
  const text = $(el).text().trim();
  if (!text) return;
  try {
    const value = JSON.parse(text);
    records.push(...(Array.isArray(value) ? value : [value]));
  } catch (error) {
    console.warn('Malformed JSON-LD:', error.message);
  }
});
console.log(records);

4. Handle encoding and raw bytes deliberately

Requests exposes decoded text through response.text and raw bytes through response.content. Character encoding can come from HTTP headers or be inferred, and assumptions can fail on mixed or incorrectly labeled sites. Use raw bytes when passing content to a parser that should determine encoding, inspect response.encoding, and set it only when you have evidence for the correct charset. Never silently replace decoding errors in data that must remain exact.

5. Deal with dynamic pages

If the needed value is absent from the initial HTML, inspect the browser’s network requests. Many applications fetch JSON from an API after page load; calling that endpoint is simpler and less fragile than scraping rendered text when the endpoint is legitimately accessible. Other pages serialize initial state in a script tag. Scrapy describes these as distinct cases: data may be in the response, in an embedded script, or loaded by later requests (Scrapy dynamic content documentation).

  • Prefer a documented or stable data endpoint when it supplies the required fields.
  • Record request parameters, pagination cursors, and required headers.
  • Use a browser only when execution is required to obtain the data.
  • Respect the site’s terms, access controls, and applicable requirements. A robots.txt file manages crawler traffic for a site’s stated crawlers; it is not a universal permission or privacy mechanism.

6. Validate the data you extracted

Parsing answers “is this syntactically valid?” It does not answer “is this the record I need?” Add checks at the boundary:

def require_product(value):
    if not isinstance(value, dict):
        raise TypeError("product must be an object")
    required = ("id", "name", "price")
    missing = [key for key in required if key not in value]
    if missing:
        raise ValueError(f"Missing keys: {missing}")
    if not isinstance(value["id"], (str, int)):
        raise TypeError("id must be a string or integer")
    if not isinstance(value["price"], (int, float)):
        raise TypeError("price must be numeric")
    return value

Keep the source payload separate from normalized application data. Store a redacted response fixture, URL, status, content type, parser name, and extraction selector so a failed run can be reproduced without repeatedly fetching the site.

7. Troubleshooting common failures

Symptom Likely cause Fix
JSONDecodeError or “Unexpected token <” HTML, an error page, or an empty body was returned Log status and content type, inspect a bounded body sample, then call raise_for_status() before decoding
Valid JSON but wrong records Wrong endpoint, pagination page, or nested key Print the top-level type and keys; validate IDs and required fields
No JSON-LD found Selector mismatch or content inserted after load Check the exact script type and inspect raw HTML plus network requests
Different results on two machines Different HTML parsers or versions Specify the parser explicitly and pin dependencies
Unicode is corrupted Incorrect charset assumption Inspect headers and encoding; parse raw bytes when appropriate
Works in a browser, fails in code Cookies, headers, authentication, rate limits, or bot checks Use permitted access, reproduce required request headers, slow down, and handle authentication securely
Intermittent timeouts Slow origin, large response, or overloaded service Set connect/read timeouts, retry only idempotent requests with backoff, and cap concurrency

8. Performance, reliability, and cost

  • Choose the smallest representation. A JSON endpoint avoids downloading and parsing unrelated HTML.
  • Stream large JSON carefully. Loading a multi-megabyte document into memory is simple but expensive; use a streaming parser when records are independent and the format supports it.
  • Cache immutable responses. Cache keys should include URL and meaningful query parameters. Honor cache headers and site rules.
  • Bound concurrency. A worker pool with per-host limits reduces timeouts and avoids creating an accidental denial-of-service pattern.
  • Retry selectively. Retry transient network errors and selected 5xx responses, with exponential backoff and jitter. Do not blindly retry malformed JSON or 4xx responses.
  • Measure stages. Record DNS/connect time, time to first byte, download time, parse time, extracted-record count, and failure category.
  • Estimate cost. Your main costs are bandwidth, compute, proxy or browser infrastructure, and engineering time. Rendering JavaScript is usually more expensive than fetching a JSON endpoint, so discover the data request first.

9. Or skip the browser setup

If your workflow starts from a web page and you need a clean visual capture alongside extracted data, ScreenshotNeo provides a website screenshot API and MCP server. It accepts one GET request and returns PNG, JPEG, WebP, or PDF. Cookie and consent banners are accepted and 60+ known consent platforms, newsletter popups, and chat widgets are removed before capture; each step can be disabled. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and the response reports the result in X-Page-Verdict and X-Billed headers.

A capture service can clean common overlays before producing an image.
A capture service can clean common overlays before producing an image.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

See the ScreenshotNeo API documentation for all options, including full-page and element capture, dark mode, device presets, retina scale, custom CSS and JavaScript, waits, request blocking, cookies and headers, geolocation, PDF output, caching, signed links, async webhooks, bulk capture, and usage reporting. Its MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients. The Free plan includes 1,000 screenshots each month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

10. A production checklist

  • Confirm whether the source is JSON, HTML-embedded JSON, or dynamically loaded.
  • Record status, headers, URL, and a safe body sample.
  • Check HTTP success before trusting decoded data.
  • Catch empty and malformed JSON explicitly.
  • Select JSON-LD by script type and account for arrays and @graph.
  • Choose and pin an HTML parser.
  • Validate required fields, types, pagination, and record counts.
  • Use bounded timeouts, concurrency, retries, and caching.
  • Preserve fixtures and extraction metadata for debugging.
  • Follow the target site’s terms and applicable access requirements.

FAQ

How do I know whether a response is JSON?

Inspect the HTTP Content-Type, but still validate by attempting a bounded decode. Servers sometimes label content incorrectly or return an HTML error page with a JSON-looking URL.

Can I parse JSON-LD with a normal JSON parser?

Yes for syntax and straightforward field extraction. Use a JSON-LD-aware processor when you need linked-data expansion, framing, or context semantics.

Why does a parser fail only on some pages?

Those pages may contain malformed markup, a different encoding, multiple payload shapes, or data inserted after JavaScript execution. Save the failing response and compare each layer.

Should I scrape visible text or an API?

Use the permitted route that supplies the required fields most reliably. A stable API-like response generally requires less parsing; a rendered page may be necessary when the data exists only after execution.