ScreenshotNeo

BlogHow-to

How to Use Web Scraping for Online Research

A practical, responsible workflow for collecting web data: define the question, choose authorized sources, scrape gently, validate results, and preserve provenance.

By the ScreenshotNeo team30 September 202610 min read

How to Use Web Scraping for Online Research

Web scraping is the automated collection of information from web pages. For online research, the hard part is rarely writing a parser. The hard part is designing a collection that answers a specific question, uses an appropriate source, respects access conditions and rights, and produces data another researcher can evaluate.

A reliable workflow is:

  1. Define the research question and the smallest useful data schema.
  2. Look for an official API, download, published dataset or archive before collecting live pages.
  3. Read the target site’s terms, API rules and robots.txt for the correct host.
  4. Collect only necessary pages and fields at a restrained rate.
  5. Minimize personal information and protect the data you do collect.
  6. Validate extracted values and preserve source provenance.
  7. Report selection rules, dates, missing data and limitations.

The sections below show a complete process, including a small Python collector you can adapt. Legal and ethical conclusions depend on your jurisdiction, the site, the data and your purpose; this guide is a research workflow, not legal advice.

1. Start with the question and a data schema

Write the question in one sentence before choosing a library. “How did the listed price of product X change between January and June?” leads to a different collection than “What themes appear in public comments?” Define the unit of analysis (page, post, product, organization or event), the time range, the population you intend to cover and explicit exclusions.

Begin with the research question and choose the least burdensome source that can answer it.
Begin with the research question and choose the least burdensome source that can answer it.

Turn that definition into a schema. For a product-price study, fields might be source_url, collected_at, product_id, price, currency and availability. Add a field only when it supports the question. Extra fields increase storage, privacy exposure, cleaning work and opportunities for extraction errors.

Write inclusion and exclusion rules

  • Which hostnames and URL paths are in scope?
  • Which language, date and content types qualify?
  • Will duplicate pages, syndicated copies or login-only material be excluded?
  • What happens when a field is absent or rendered only after JavaScript runs?

Record these rules in a short project note and version it. A future reader should be able to tell why a page was collected and why another was omitted.

2. Prefer an authorized or curated source

Before crawling live pages, check the publisher’s API, an open-data download, a research dataset, a sitemap or a web archive. These sources can reduce requests and make your work easier to reproduce. An API may provide stable field names and explicit usage limits; a download may include historical records that are no longer visible on the site.

Common Crawl is one archive option. Its terms explain that archived material can remain subject to the original owner’s terms and that Common Crawl does not guarantee the truthfulness, authenticity, quality, lawfulness or accuracy of crawled content. Treat it as a source to evaluate, not a blanket license to reuse everything. Compare archive coverage and capture dates with the needs of your question.

Source Strength Questions to check
Official API Structured fields and documented access Authentication, quotas, retention and version changes
Download or dataset Efficient, repeatable bulk access License, update schedule, missing fields and provenance
Web archive Historical snapshots and reduced live-site load Capture coverage, original terms and archive accuracy
Direct page collection Current presentation and fields unavailable elsewhere Terms, robots rules, rate, consent, privacy and maintenance

3. Check terms and robots.txt

Read the site’s terms of service and any API documentation before collecting. Look for rules about automated access, redistribution, authentication, rate limits and personal data. Google describes robots.txt as a way for site owners to tell crawlers which URLs they may request. Its instructions are scoped to the host, protocol and port where the file is served; a different subdomain can have different rules.

Google’s documentation also states that robots instructions cannot enforce crawler behavior, and a blocked URL can still appear in search results. Therefore, robots.txt is an access signal, not a security control or legal clearance. Google does not support a crawl-delay field in its interpretation, and other crawlers may interpret fields differently. Follow applicable disallow rules and any documented host guidance, but do not infer that compliance resolves copyright, privacy, contract or database-rights questions.

Fetch the file for the exact origin you plan to contact:

from urllib.parse import urlparse
import requests

url = "https://example.org/research/page"
parts = urlparse(url)
robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt"
r = requests.get(robots_url, timeout=20)
print(r.status_code)
print(r.text[:2000])

Also review the robots.txt specification and the site’s own instructions. Keep a dated copy of the file and terms page in your research records because they can change.

4. Plan a restrained collection

Identify the collector, target URLs, fields, schedule and stop conditions. Use a URL queue rather than an open-ended crawl. Restrict requests to pages needed for the question, cache responses, use a descriptive user agent with a contact address where appropriate, and stop when the planned sample is complete.

Do not invent a universal delay number. Use the host’s documented limit or a conservative, documented rate that does not interfere with normal service. Add retries only for temporary failures, with exponential backoff and a maximum attempt count. Avoid parallelism until you understand the site’s limits.

A minimal collector that extracts a title and canonical link might look like this:

import csv
import time
from datetime import datetime, timezone
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

URLS = [
    "https://example.org/page-a",
    "https://example.org/page-b",
]
HEADERS = {"User-Agent": "ResearchCollector/1.0 (contact: researcher@example.edu)"}

rows = []
with requests.Session() as session:
    session.headers.update(HEADERS)
    for url in URLS:
        collected_at = datetime.now(timezone.utc).isoformat()
        try:
            response = session.get(url, timeout=30)
            response.raise_for_status()
            soup = BeautifulSoup(response.text, "html.parser")
            canonical = soup.find("link", rel="canonical")
            rows.append({
                "source_url": url,
                "collected_at": collected_at,
                "title": soup.title.get_text(" ", strip=True) if soup.title else "",
                "canonical_url": urljoin(url, canonical["href"]) if canonical and canonical.get("href") else "",
                "status": response.status_code,
            })
        except requests.RequestException as exc:
            rows.append({"source_url": url, "collected_at": collected_at, "title": "", "canonical_url": "", "status": f"error: {exc}"})
        time.sleep(2)  # replace with the host's documented guidance

with open("research_extract.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["source_url", "collected_at", "title", "canonical_url", "status"])
    writer.writeheader()
    writer.writerows(rows)

This example intentionally uses a fixed URL list, records failures and timestamps, and leaves the rate decision visible. Adapt selectors to the target site’s markup; do not assume a CSS selector is stable over time.

Some values are present in the initial HTML; others appear after JavaScript runs. First inspect the response and browser developer tools to determine whether the data is embedded in JSON, available through an authorized endpoint or rendered only in the browser. An API or downloadable source is usually more stable than reverse-engineering private requests.

If browser automation is necessary, capture only the page state and fields required by your schema. Handle cookie-consent controls according to the site’s choices and your research authorization. Do not bypass authentication, paywalls, bot checks or technical access controls. A blocked or incomplete page should be recorded as missing rather than silently treated as a valid observation.

6. Protect people and third-party rights

Publicly visible does not automatically mean unrestricted to collect, republish or profile. Decide whether personal information is necessary. If it is not necessary, do not collect it; if it is necessary, define access controls, retention, deletion and a lawful basis before collection. Remove direct identifiers from working files when possible and restrict raw pages to the people who need them.

Check institutional review requirements before collecting information about people, especially sensitive attributes, vulnerable groups or material that could create risk if combined. The research framework by Brown and coauthors treats legal, ethical, institutional and scientific questions as connected decisions for U.S.-based social science research; it does not decide the outcome for a particular project. Common Crawl’s terms likewise place responsibility on users to respect privacy and others’ rights.

7. Validate extracted data

Validation is part of collection, not a final cosmetic step. Compare a sample of rows with the source pages by hand. Check that numbers use the expected currency and decimal convention, dates include a timezone or stated interpretation, and missing values are distinguishable from zero or an empty string.

  • Measure selector success: how many in-scope pages produced each required field?
  • Inspect outliers and sudden distribution changes.
  • Keep the original value beside any normalized value.
  • Record redirects, HTTP status, page type and parser version.
  • Re-run a small fixture set after changing extraction code.

Dynamic pages, redesigns, localization and A/B tests can create systematic missingness. Report which pages could not be collected and why. Never fill a missing observation from an unsupported assumption.

8. Preserve provenance and reproducibility

For every record, retain the source URL, collection timestamp, collector version, extraction method, transformations and validation status. Store a hash of downloaded content when permitted, or preserve a short source snapshot and a pointer to the original. Keep a README describing scope, exclusions, software versions, request policy and known limitations.

When sharing results, publish your schema and code where rights permit, but do not redistribute substantial copyrighted text or personal data merely to make a project convenient. If raw data cannot be shared, provide aggregated outputs, field-level examples and instructions for an authorized researcher to reproduce the process.

9. Troubleshooting common failures

Symptom Likely cause Fix
403 or 429 responses Access policy, authentication or excessive rate Stop, read the terms and API limits, reduce requests, and use an authorized endpoint. Do not rotate identities to evade controls.
HTML contains no target field Content is JavaScript-rendered or selector changed Inspect initial HTML and documented APIs; update a versioned selector only after confirming the page structure.
Repeated timeouts Slow origin, oversized pages or network instability Use a reasonable timeout, bounded retries with backoff and caching; record failures instead of retrying indefinitely.
Wrong language, currency or date Locale defaults or ambiguous parsing Set an authorized locale where supported and store raw values with an explicit normalization rule.
Duplicate records Tracking parameters, redirects or syndicated pages Canonicalize URLs, remove known tracking parameters and retain the original URL for audit.
Consent dialog obscures content Page requires an interaction before displaying fields Use the site’s documented consent flow or an authorized API; record the resulting page state and do not bypass controls.

10. Performance, reliability and cost

For a small study, the main cost is researcher time. A narrow URL list, cached responses and a simple parser are often more reliable than a large crawler. For larger collections, estimate total pages, response size, storage, parsing time and re-fetch frequency before starting. Queue work, checkpoint progress and make jobs restartable so a network failure does not repeat successful requests.

A clean capture can preserve the page state needed for visual research while removing common overlays.
A clean capture can preserve the page state needed for visual research while removing common overlays.

Reliability improves when you separate fetching from parsing: save permitted responses, then run the parser offline. This lets you debug selectors without contacting the site again. Track status codes, latency, retries and field-level missingness. A successful HTTP response is not proof that the page contains valid research data.

Cost also includes API fees, storage, review and compliance work. An archive or official dataset can be cheaper in requests and easier to reproduce, while direct collection may provide fresher fields. Choose based on authorization, coverage, freshness, provenance, privacy exposure, source burden and reproducibility rather than on request volume alone.

Or skip the browser setup

When your research needs clean page images or PDFs for visual review, ScreenshotNeo provides a website screenshot API and MCP server. One GET request returns a PNG, JPEG, WebP or PDF. Cookie and consent banners, newsletter popups and chat widgets are removed before the shot; each step can be turned off. Bot checks, blank pages, timeouts, failed loads and cache hits are not billed, and response headers identify the page verdict and billing status.

See the ScreenshotNeo API documentation for all options. The basic call is:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Options cover full-page capture with lazy images, CSS element selection, dark mode, device presets, retina scale, PDF paper and page settings, custom CSS and JavaScript, click and wait conditions, blocked resources, headers, cookies, user agents, timezone, geolocation, transparent backgrounds, resizing, caching, signed links, async webhooks, bulk capture and usage reporting. Its MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.

Plans include 1,000 screenshots per month free with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

FAQ

Is scraping the same as downloading a dataset?

No. Scraping extracts from pages you request; a dataset or archive is already collected and may have different coverage, terms and provenance.

Does a public page mean I can republish its contents?

No. Public visibility does not settle copyright, privacy, contract or database-rights questions. Check the applicable law, terms and your purpose.

Should I obey robots.txt?

Treat it as the host’s crawler instruction and honor disallow rules. It is not a lock, a complete technical standard or legal permission.

How do I prove a result came from a particular page?

Keep the URL, timestamp, source snapshot or permitted hash, extraction version, transformations and validation notes with each record.

What if the site changes while my study is running?

Version your parser, monitor field-level missingness, save failures and report the collection window and redesign effects in the methods section.