How to Use a JavaScript-Rendering Web Scraping API

Short answer: use a JavaScript-rendering web scraping API when the data you need is inserted or changed by client-side JavaScript after the initial HTTP response. The API loads the page in a browser engine, runs its scripts, waits for the content you specify, optionally performs a few interactions, and returns rendered HTML or extracted data.

Start with an ordinary HTTP request. If the required field is already in the response HTML, rendering only adds latency and cost. If it appears after hydration, an API call, a click, or a scroll, enable rendering and wait for a stable selector. ScraperAPI documents this pattern with render=true and wait_for_selector; selector waits require rendering to be enabled. See the provider’s JavaScript rendering documentation for its current parameter syntax.

1. Decide whether you need browser rendering

Use this decision process before choosing an API option:

1. Request the URL with a normal HTTP client.
2. Inspect the response body for the exact field, not just a placeholder such as <div id="app">.
3. If the field is present, parse the HTML without rendering.
4. If the field is absent and appears in a real browser, enable JavaScript rendering.
5. Identify a selector that represents the data being ready, such as .product-price or [data-testid="results"].

Rendering executes page scripts in a headless browser. It does not automatically make every page accessible: authentication, bot checks, rate limits, geo restrictions and application errors still need to be handled. Only collect data you are permitted to access. RFC 9309 explains that robots.txt is crawler guidance and “not a form of access authorization”; a permissive file is not legal, contractual or authenticated permission.

2. Make a rendered request with cURL

This conceptual ScraperAPI request enables rendering and returns the browser-rendered HTML:

curl --get 'https://api.scraperapi.com/' \
  --data-urlencode 'api_key=API_KEY' \
  --data-urlencode 'render=true' \
  --data-urlencode 'url=https://example.com/' \
  --output rendered.html

Always URL-encode the target. Keep the API key in an environment variable or secret manager rather than shell history, source control or application logs. The response is HTML, so your next step is to parse it with an HTML parser and validate that the target element exists.

Wait for a selector

A fixed sleep guesses how long a page needs. A selector wait expresses the condition you actually need. For example:

curl --get 'https://api.scraperapi.com/' \
  --data-urlencode 'api_key=API_KEY' \
  --data-urlencode 'render=true' \
  --data-urlencode 'wait_for_selector=.product-price' \
  --data-urlencode 'url=https://example.com/product/123' \
  --output product.html

Choose a stable selector tied to the content, preferably an attribute intended for testing or data extraction. Avoid selectors based on generated class names, deep positional paths or visual styling. A selector that never appears causes a timeout, so make sure it is present for all legitimate page states or implement a fallback.

3. Complete Python example

The following script requests rendered HTML, checks the HTTP response, and extracts a value with Beautiful Soup. Install dependencies with pip install requests beautifulsoup4.

import os
import requests
from bs4 import BeautifulSoup

API_KEY = os.environ["SCRAPERAPI_KEY"]
TARGET = "https://example.com/product/123"

params = {
    "api_key": API_KEY,
    "render": "true",
    "wait_for_selector": ".product-price",
    "url": TARGET,
}

response = requests.get(
    "https://api.scraperapi.com/",
    params=params,
    timeout=90,
)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
price = soup.select_one(".product-price")
if price is None:
    raise RuntimeError("Rendered response did not contain .product-price")

print(price.get_text(" ", strip=True))

In production, record the target URL, elapsed time, status code, response length and parser result. Do not log the API key or sensitive cookies. Treat an HTTP 200 response without the expected selector as an extraction failure, not a successful scrape.

4. Complete Node.js example

This example uses the built-in fetch available in current Node.js releases. It saves the rendered response and checks for the expected marker.

const apiKey = process.env.SCRAPERAPI_KEY;
const target = 'https://example.com/product/123';

const query = new URLSearchParams({
  api_key: apiKey,
  render: 'true',
  wait_for_selector: '.product-price',
  url: target,
});

const response = await fetch(`https://api.scraperapi.com/?${query}`);
if (!response.ok) {
  throw new Error(`Scraping API returned ${response.status}`);
}

const html = await response.text();
if (!html.includes('product-price')) {
  throw new Error('Expected product selector was not returned');
}

console.log(html);

For robust parsing, use an HTML parser such as Cheerio rather than string matching. Check the page’s actual markup because a class name appearing in a script bundle or embedded JSON does not prove that the element was rendered.

5. Add browser interactions when rendering alone is not enough

Some pages expose data only after an action. ScraperAPI’s instruction feature supports actions such as entering text, clicking, scrolling, waiting for browser events and waiting for selectors. A typical sequence is:

1. Enter a search term into the input.
2. Click the submit button.
3. Wait for the result selector.

Keep an instruction set to roughly three or four actions. Each additional action adds browser work and increases timeout risk. Prefer a direct URL with query parameters when the site offers one. Use a selector wait after the final action instead of a long arbitrary delay.

Conceptually, your request needs rendering plus the provider’s instruction syntax:

curl --get 'https://api.scraperapi.com/' \
  --data-urlencode 'api_key=API_KEY' \
  --data-urlencode 'render=true' \
  --data-urlencode 'url=https://example.com/search' \
  --data-urlencode 'instructions=...' \
  --output results.html

Instruction syntax is provider-specific, so copy the current format from the provider documentation rather than assuming that another API’s action format will work.

6. Parse rendered HTML safely

Rendered HTML can contain several representations of the same value:

• Visible text in the DOM.
• Attributes such as data-price.
• Embedded JSON in a script tag.
• Links generated after hydration.

Prefer semantic attributes and stable data markers. Normalize whitespace, currency symbols and localized number formats before storing values. Validate required fields and preserve the raw response or a content hash when you need auditability. If the site returns an error component inside a normal HTTP 200 page, detect it explicitly.

7. Rendering options and trade-offs

Option	Use it when	Trade-off
Plain HTTP	Data is in initial HTML	Lowest latency and cost
render=true	JavaScript inserts or changes the data	Browser startup adds latency and provider credits
wait_for_selector	A known element signals readiness	Bad selectors create timeouts
Fixed wait	No reliable selector exists	Can return too early or wait unnecessarily
Interaction instructions	Data requires input, click or scroll	More actions increase failure risk
Premium proxy tiers	Your permitted workload needs them	Higher credit consumption

ScraperAPI documents 10 API credits for an ordinary JavaScript-rendered request, 25 credits with premium proxies and 75 with ultra-premium proxies. These are provider-specific values and can change. Its FAQ also documents a default rendering burst limit of 10 requests per second. Treat both as current configuration details to verify, not industry benchmarks.

8. Performance, reliability and cost planning

Reduce latency

• Do not render pages whose data is available from ordinary HTML.
• Wait for the target selector instead of sleeping for a large fixed interval.
• Keep interaction sequences short.
• Request only the pages and fields you need.
• Use bounded client timeouts and cancel work that has exceeded your service-level limit.

Build for intermittent failures

• Retry transient network errors and provider 5xx responses with exponential backoff and jitter.
• Do not blindly retry a deterministic selector timeout; inspect the page state first.
• Cap retries so one URL cannot consume your whole queue.
• Record whether a failure happened during navigation, rendering, waiting or parsing.
• Use idempotent jobs and a deduplication key when workers may retry.

Rendering increases latency and can reduce success rates, according to ScraperAPI’s own guidance. Measure your permitted representative workload: record p50 and p95 latency, selector success, timeout rate, response size and credits per successful item. Vendor feature pages are not independent performance benchmarks.

9. Troubleshooting common errors

Symptom	Likely cause	Fix
Placeholder HTML only	Rendering was not enabled	Add the provider’s render flag and verify the response body.
Selector wait times out	Wrong selector, consent wall or failed page script	Inspect returned HTML, choose a stable selector, and handle the consent or error state.
HTTP 200 but no data	Application returned an error page inside a successful HTTP response	Check for error headings, empty-state markers and the expected data field.
Frequent timeouts	Too many actions, slow origin or an overly short client timeout	Reduce actions, use a readiness selector, and set a bounded timeout appropriate to the workload.
429 or throttling	Rate or burst limit exceeded	Queue requests, add backoff and stay within the provider’s documented limits.
Blocked or CAPTCHA page	Bot mitigation or access policy	Do not attempt to bypass controls without authorization; review access terms and use an allowed integration.
Wrong language or prices	Locale, timezone or region differs	Use the provider’s supported location controls or an explicit URL locale when permitted.
Parser breaks after a redesign	Selector depended on presentation classes	Switch to semantic attributes and add fixture-based parser checks.

10. Or skip the browser setup

If your goal is a clean visual capture rather than rendered HTML extraction, ScreenshotNeo provides a single GET request that runs the page and returns a PNG, JPEG, WebP or PDF. It can wait for a selector, delay or network idle; click elements; load lazy images; capture one CSS-selected element or the full page; set a device, viewport, dark mode, retina scale, headers, cookies, user agent, timezone and geolocation; block ads, trackers or resource types; inject CSS or JavaScript; resize images; cache with a chosen TTL; and submit asynchronous jobs with signed webhooks. See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Before capture, cookie and consent banners are accepted and more than 60 known consent platforms, newsletter popups and chat widgets can be removed; each cleanup step can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and response headers identify the page verdict and billing result. ScreenshotNeo also offers an MCP server with take_screenshot, get_page_info and capture_pdf for Claude, Cursor and other MCP clients. The Free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000.

Create a free ScreenshotNeo account and use the 1,000 monthly screenshots without a card.

11. A production checklist

• Confirm the target is permitted and review its access requirements.
• Prove that ordinary HTML does not contain the required data.
• Enable rendering only for pages that need it.
• Choose a stable readiness selector.
• Keep interactions to the minimum sequence.
• Set client, job and queue timeouts.
• Retry only transient failures with capped exponential backoff.
• Validate the extracted fields, not just HTTP status.
• Track latency, selector success, errors and credits.
• Redact credentials, cookies and personal data from logs.

12. FAQ

Does JavaScript rendering execute every script?

It runs the page in a browser engine, but scripts can still fail because of network errors, authentication, bot controls, browser incompatibilities or application bugs. Verify the target element before treating a response as usable.

Is waiting for network idle always best?

No. Analytics, ads and long-lived connections can prevent true idleness. A selector that represents the data being ready is usually more precise.

Can I use a rendered scraping API for screenshots?

Some providers advertise screenshot output, but HTML rendering and image capture are different outputs. If you need a clean PNG, WebP, JPEG or PDF, use a capture API designed for that result.

How should I compare providers?

Compare rendered HTML versus structured extraction, selector waits, interaction support, proxy and location controls, credit rules and measured results on the same permitted URLs. Do not treat vendor marketing pages as independent benchmarks.

What should I do when robots.txt allows crawling?

Check the site’s terms, contracts, authentication requirements and applicable law as well. RFC 9309 makes clear that robots.txt is not access authorization.