How to Connect a Web Scraping API with Make

To connect a web scraping API to Make, add HTTP > Make a request to your scenario, configure the provider’s HTTPS endpoint and authentication, put the target URL in the documented query or request body field, enable Parse response, and map the returned fields into the next module. For an asynchronous scraper, either poll the job status or give the provider a Make Webhooks > Custom webhook URL.

Make’s HTTP app is the general connector for API services without a native integration. The request URL must use https://; Make rejects unverified self-signed certificates. See the HTTP app documentation for the current module fields.

1. Decide how the scraping API works

Read the scraping provider’s API documentation before building the scenario. Identify these details:

• Endpoint and method: synchronous scrape calls are commonly GET or POST; asynchronous APIs usually expose a start-job endpoint and a status or result endpoint.
• Authentication: API key, Basic Auth, OAuth 2.0, or no authentication.
• Target field: the exact parameter or JSON key containing the URL to scrape.
• Response shape: flat JSON, nested result arrays, a downloadable file, or a status/error object.
• Pagination: offset, page number, next-link, or cursor/token.
• Callback support: whether the provider can POST a completion payload to a URL you provide.

Provider parameter names and response paths are authoritative. Do not rename fields in Make until you know the provider’s documented schema.

2. Build a synchronous request in Make

1. Create or open a Make scenario and add HTTP > Make a request.
2. Set the URL to the provider’s documented HTTPS endpoint.
3. Select the required authentication type. Store credentials in Make’s dedicated Credentials or keychain field instead of exposing a token in a query string or ordinary header field. Make documents No authentication, API key, Basic Auth, and OAuth 2.0 options.
4. Choose the provider’s method, usually GET or POST.
5. For GET or DELETE inputs, add each value under Query parameters. Map the source URL from an earlier module, such as a spreadsheet row, webhook, or data store.
6. For POST requests, select the body format required by the provider: application/JSON, multipart form data, URL-encoded form data, or a custom content type.
7. For JSON, use Data structure when possible. Make maps keys and escapes JSON-reserved characters for you. Use JSON string only when you need to send a raw provider-specific document and can manage escaping yourself.
8. Turn on Parse response.
9. Run the module once. Make exposes the returned fields for mapping only after it has seen a response.
10. Map the result into a database, spreadsheet, CRM, notification, file store, or AI module.

Example GET configuration

Assume the provider documents GET https://api.example-scraper.com/v1/scrape with an API-key credential and a url query parameter. Configure the HTTP module as follows:

Make field	Value
URL	https://api.example-scraper.com/v1/scrape
Method	GET
Authentication	Provider API-key credential in Make’s Credentials field
Query parameter	url mapped from the previous module
Parse response	Yes

Replace the example hostname and parameter names with values from your provider. The example hostname is illustrative, not a claim that this endpoint exists.

Example POST configuration

If the provider expects JSON, set the content type to application/json and send only documented keys:

{
  "url": "https://example.com/products",
  "country": "US",
  "render_js": true
}

Use Make’s data-structure mode to map these values safely. Only send options the provider supports; names such as render_js vary by service.

3. Pass URLs and options safely

• Map the URL from the source module instead of concatenating untrusted text into a raw JSON string.
• Use query parameters for GET inputs and the documented body for POST inputs.
• Keep API tokens in Make’s credential store. Dedicated credential storage reduces accidental exposure, centralizes access, and supports key rotation.
• Use HTTPS for every request and reject workflows that silently downgrade to HTTP.
• Store the original URL alongside the scraped result so a later retry can reproduce the request.
• For large HTML or file responses, check the provider’s response limits before sending data to modules that have smaller field limits.

4. Parse and map the response

With Parse response enabled, run the HTTP module once and inspect the output bundle. Map fields such as status, title, extracted records, pagination metadata, and provider request IDs into later modules.

Example response shape (illustrative):
{
  "status": "complete",
  "data": [
    {"name": "Item A", "price": "12.00"}
  ],
  "next_cursor": "abc123",
  "request_id": "provider-id"
}

Map data[] with an Iterator when each record needs its own downstream operation. Map the whole array when the destination accepts a batch. Preserve the provider’s status and request ID for diagnostics.

5. Handle asynchronous scraping jobs

An asynchronous API returns a job identifier instead of scraped data. There are two standard designs.

Polling from Make

1. HTTP module A starts the scrape and returns a job ID.
2. Store the job ID and source URL in a data store or other durable record.
3. Schedule a scenario to call the provider’s status endpoint after a delay.
4. Use a router or filter for queued, running, complete, and failed states.
5. When complete, call the result endpoint and map the data.
6. Stop after the provider’s documented timeout or a retry count you choose; mark the job as failed and notify an operator.

Polling is easy to reason about, but every status request consumes operations and can delay completion.

Callback with a Custom webhook

1. Add Webhooks > Custom webhook as the first module in a new scenario.
2. Create the webhook and copy the unique URL Make generates.
3. Put that URL in the scraping provider’s callback or webhook setting when starting the job.
4. Send a test callback so Make can infer the data structure.
5. Validate the event status and job ID before fetching or storing results.

Make accepts query-string, form, multipart, and JSON webhook input. You can require an API key in the x-make-apikey header, validate a declared data structure, expose request headers, and pass the original JSON through as text. The documented maximum webhook payload is 5 MB (5,242,880 bytes); have the callback contain a result URL or job ID when the scraped document could exceed that size.

Callbacks are usually more efficient for long jobs because the scenario starts when the provider reports completion. Protect the webhook with the provider’s supported secret or Make’s API-key option, validate the job ID and status, and make the storage step idempotent.

6. Configure pagination

Make supports offset-based, page-based, URL/link-based, and token/cursor-based pagination. Configure the items array and next-page value exactly as the provider documents them.

Provider response	Make approach
page and total_pages	Repeat requests while the next page is within the total.
offset and limit	Increase offset by the number of returned records.
next URL	Pass the returned URL into the next HTTP request.
next_cursor	Send the cursor in the provider’s documented query or body field.

Set a maximum page count and retain the cursor or next URL with the job record. This prevents an incorrect termination condition from creating an endless scenario.

7. Test the same request outside Make

Testing a provider call with a small script helps separate provider errors from Make mapping errors. Replace placeholders with the provider’s documented endpoint and fields.

cURL

curl -G 'https://api.example-scraper.com/v1/scrape' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  --data-urlencode 'url=https://example.com'

Python

import requests

r = requests.get(
    'https://api.example-scraper.com/v1/scrape',
    headers={'Authorization': 'Bearer YOUR_API_KEY'},
    params={'url': 'https://example.com'},
    timeout=90,
)
r.raise_for_status()
print(r.json())

Node.js

const q = new URLSearchParams({ url: 'https://example.com' });
const res = await fetch(
  `https://api.example-scraper.com/v1/scrape?${q}`,
  { headers: { Authorization: 'Bearer YOUR_API_KEY' } }
);
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
console.log(await res.json());

8. Common errors and fixes

Error	Likely cause	Fix
Invalid URL or certificate error	Endpoint is not HTTPS or uses an unverified self-signed certificate.	Use the provider’s public HTTPS endpoint with a valid certificate.
401 or 403	Wrong credential type, expired token, or missing required scope.	Recheck the provider’s authentication scheme and store the token in Make’s Credentials field.
400 validation error	Wrong parameter name, body type, or URL encoding.	Compare the module with the provider’s example request; use query parameters for GET and a documented JSON/form body for POST.
Parse response shows no fields	The module has not received a sample response, or the response is not JSON.	Run the module once; check the content type and disable JSON parsing only when the provider returns text or a file.
Scenario loops forever	Pagination or polling termination condition is missing.	Stop on the documented next-page absence, completion state, timeout, or maximum attempt count.
Webhook never triggers	Wrong callback URL, provider cannot reach it, or secret validation fails.	Copy the current Make webhook URL, send a test event, inspect headers, and verify the provider’s callback delivery logs.
Webhook payload too large	Payload exceeds Make’s 5 MB limit.	Ask the provider to send a job ID or result URL, then fetch the result with HTTP.
Duplicate records	Provider retries a callback or Make retries a module.	Use a provider job ID or source URL plus run ID as an idempotency key before inserting.

9. Reliability, performance, and cost

• Retries: retry transient 429 and 5xx responses with backoff when the provider permits it. Do not blindly retry validation errors.
• Rate limits: respect the provider’s requests-per-minute and concurrent-job limits. Use Make scheduling, queues, or a sleep between pages.
• Timeouts: set a request timeout compatible with the provider’s maximum response time and record failures for replay.
• Idempotency: deduplicate by provider job ID, canonical URL, and extraction date before writing to a destination.
• Payload size: keep webhook messages small and download large results in a separate HTTP step.
• Observability: save HTTP status, provider request ID, job ID, source URL, attempt count, and error body.
• Scenario operations: asynchronous callbacks can reduce repeated status requests; pagination and per-record modules increase Make operations.
• Provider charges: check whether the scraper bills per request, page, bandwidth, render time, or successful result. Make operation usage and provider usage are separate costs.

10. Or skip the browser setup

If your goal is clean website screenshots inside a Make workflow, ScreenshotNeo provides a single HTTPS request and an MCP server for AI agents. It accepts the cookie or consent banner before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed; each response identifies the result with X-Page-Verdict and X-Billed headers.

See the ScreenshotNeo API documentation for all options. In Make, add HTTP > Make a request with method GET, URL https://api.screenshotneo.com/v1/shot, and query parameters access_key and url. Save the binary response to your file or storage module.

cURL

curl -G 'https://api.screenshotneo.com/v1/shot' \
  -d access_key=YOUR_API_KEY \
  --data-urlencode url=https://stripe.com \
  -o shot.webp

Python

import requests

r = requests.get(
    'https://api.screenshotneo.com/v1/shot',
    params={'access_key': 'YOUR_API_KEY', 'url': 'https://stripe.com'},
    timeout=90,
)
open('shot.webp', 'wb').write(r.content)

Node.js

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo also supports PNG, JPEG, WebP, and PDF output; full-page capture with lazy images loaded; CSS-selector element capture; dark mode; device presets and custom viewports; retina scale; PDF paper, margin, landscape, and page-range options; custom CSS and JavaScript; clicks; waits; request blocking; headers, cookies, user agents, and Authorization; timezone and geolocation; transparent backgrounds; resizing; configurable caching; signed links; asynchronous jobs with signed webhooks; bulk capture of up to 100 URLs per call; a usage API; and an OpenAPI specification.

An MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients. The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

11. Short FAQ

Can Make call any scraping API?

Yes, when the service exposes an HTTPS API that accepts the authentication and request format you configure. Make’s HTTP module is intended for APIs without a native integration.

Should I put the API key in the URL?

Use Make’s dedicated Credentials field when the provider supports it. Avoid query-string tokens because URLs are more likely to appear in logs and history.

When should I use a webhook instead of polling?

Use a webhook when the provider supports reliable callbacks and jobs may take long enough that repeated status requests add operations. Poll when callbacks are unavailable or when you need a simple scheduled design.

How do I map nested scraper results?

Run the HTTP module once with Parse response enabled, then map the nested array into an Iterator or map the complete array into a batch-capable destination.

What is the safest way to handle duplicate callbacks?

Store a provider job ID or another stable idempotency key before writing the result, and skip records whose key already exists.