Alternative Data for Finance: Web Scraping for Investment Research

Direct answer: Web scraping can collect useful alternative data for investment research, such as product availability, prices, reviews, hiring activity, store locations, shipping indicators, and changes in company narratives. It is a collection technique, not proof that a signal is accurate, representative, permitted, or profitable. A defensible process starts with a specific investment question, checks terms and privacy constraints, records provenance, validates against independent sources and primary disclosures, and tests for bias and leakage before a signal influences a live decision.

This guide shows a practical workflow, a runnable Python collector, validation controls, operating considerations, and a browser-based capture option for dynamic pages.

What alternative data means in finance

Alternative data sits outside traditional company filings, audited financial statements, and standard market-data feeds. CFA Institute groups examples into individual data (social media, blogs, product reviews, web-search trends, and cellphone-location data), business data (card transactions, store visits, and bills of lading), and satellite data (agriculture, rig activity, traffic, shipping, and mining).

Web scraping is one way to collect a subset of these observations from public pages. A page being publicly reachable does not establish that its contents are accurate, complete, licensed for your use, or representative of the market. Treat every scraped field as an observation with a source, timestamp, extraction method, and uncertainty.

Start with a falsifiable investment question

Do not begin by collecting everything a website exposes. Define the decision the data might inform.

1. Question: For example, “Are stockouts at this retailer increasing before the next reported quarter?”
2. Universe: Name the companies, products, locations, or securities included.
3. Horizon: Specify whether the signal is intended for days, weeks, or a reporting cycle.
4. Hypothesis: State what observation should change your view and what would falsify it.
5. Decision rule: Define how the measurement could affect a watchlist, forecast, or position size.

This prevents a common failure mode: discovering an attractive correlation after repeatedly trying sources and transformations.

Choose sources and design collection responsibly

Source assessment checklist

• Who owns the source and how is the information produced?
• Is an official API available? Prefer it when it provides the needed fields.
• What is the update cadence, historical depth, and revision behavior?
• Can you distinguish a real change from a layout, catalog, or policy change?
• How much of the relevant universe is covered, and which entities are missing?
• Could the collection process expose personal or sensitive information?
• Do the terms of service, robots.txt, authentication requirements, and local laws permit the intended access and use?

Limit request rate and frequency, identify your client where appropriate, cache unchanged pages, and avoid creating undue server load. Exclude personal information unless you have a documented lawful basis and governance process. A regulator may use automated collection of public websites for market monitoring, but that does not provide a blanket permission for every private research use.

A runnable Python collection example

The example below collects product availability and displayed price from a set of public, server-rendered pages. It stores raw HTML hashes, timestamps, parser versions, and extraction errors so another analyst can reproduce the run. Replace the example URLs and CSS selectors with sources you have reviewed.

from __future__ import annotations

import csv
import hashlib
import time
from datetime import datetime, timezone
from pathlib import Path

import requests
from bs4 import BeautifulSoup

URLS = [
    "https://example.com/product-a",
    "https://example.com/product-b",
]
PRICE_SELECTOR = ".price"
AVAILABILITY_SELECTOR = ".availability"
OUT = Path("observations.csv")
USER_AGENT = "research-collector/1.0 (contact: analyst@example.org)"

session = requests.Session()
session.headers.update({"User-Agent": USER_AGENT})
rows = []

for url in URLS:
    collected_at = datetime.now(timezone.utc).isoformat()
    row = {
        "url": url,
        "collected_at": collected_at,
        "parser_version": "1.0",
        "http_status": "",
        "html_sha256": "",
        "price_text": "",
        "availability_text": "",
        "error": "",
    }
    try:
        response = session.get(url, timeout=30)
        row["http_status"] = response.status_code
        response.raise_for_status()
        row["html_sha256"] = hashlib.sha256(response.content).hexdigest()
        soup = BeautifulSoup(response.text, "html.parser")
        price = soup.select_one(PRICE_SELECTOR)
        availability = soup.select_one(AVAILABILITY_SELECTOR)
        row["price_text"] = price.get_text(" ", strip=True) if price else ""
        row["availability_text"] = (
            availability.get_text(" ", strip=True) if availability else ""
        )
    except requests.RequestException as exc:
        row["error"] = f"request: {exc}"
    except Exception as exc:
        row["error"] = f"parse: {exc}"
    rows.append(row)
    time.sleep(2)  # keep request frequency conservative

with OUT.open("w", newline="", encoding="utf-8") as handle:
    writer = csv.DictWriter(handle, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

print(f"Wrote {len(rows)} observations to {OUT}")

Install dependencies with python -m pip install requests beautifulsoup4. Keep the raw response or an access-controlled archive when policy allows; otherwise retain a cryptographic hash and enough metadata to explain what was observed.

Dynamic pages: use a browser only when necessary

Requests and HTML parsing cannot see content rendered after JavaScript runs. Use a browser automation tool when the data genuinely requires it, and keep the same governance controls. The following Playwright example waits for a selector, extracts text, and closes the browser.

import asyncio
from playwright.async_api import async_playwright

async def collect(url: str) -> dict:
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(headless=True)
        page = await browser.new_page()
        try:
            response = await page.goto(url, wait_until="networkidle", timeout=60_000)
            await page.wait_for_selector(".price", timeout=15_000)
            return {
                "url": url,
                "status": response.status if response else None,
                "price": await page.locator(".price").inner_text(),
                "availability": await page.locator(".availability").inner_text(),
            }
        finally:
            await browser.close()

print(asyncio.run(collect("https://example.com/product-a")))

Browser runs are more expensive and fragile. Set explicit timeouts, capture console and network errors, pin the browser version in production, and record whether the page was blocked, incomplete, or changed layout.

Validate before using a signal

1. Independent comparison: Compare the scraped measure with filings, company disclosures, market data, or a second unrelated source.
2. Missingness review: Measure which pages, products, dates, and entities are absent. Missing data is often systematic.
3. Revision tracking: Store observation time and detect retroactive edits. A page seen today may not reflect what was visible before an announcement.
4. Bot and access artifacts: Separate normal pages from CAPTCHA responses, login pages, rate-limit pages, and empty templates.
5. Selection and survivorship tests: Check whether only popular products, successful stores, surviving companies, or easily indexed pages enter the sample.
6. Parser-change tests: Alert when selector counts, field distributions, or HTML hashes change abruptly.
7. Leakage controls: Use only information available at the timestamp assigned to each decision. Do not backfill historical observations from pages that were edited later.
8. Paper evaluation: Run a historical or paper-trading study with a predeclared methodology. Do not describe an unrun test as evidence of performance.

Document the source, assumptions, caveats, transformations, and uncertainty in the research note. CFA Standard V(A) requires reasonable care, judgment, and inquiries into the sources and accuracy of data used in analysis. Independence, attribution, and conflict controls also apply when scraped data informs published research.

What to store for reproducibility

Field	Purpose
Source URL and owner	Shows provenance and supports permission review.
Observed timestamp and timezone	Prevents look-ahead and clarifies update latency.
Raw capture or content hash	Allows later verification without silently rewriting history.
Parser and dependency versions	Explains changes caused by code or browser upgrades.
HTTP status and verdict	Separates valid observations from blocks and failures.
Transformations and units	Makes normalization and currency conversion auditable.
Exceptions and missing fields	Prevents silent data loss.

Or skip the browser setup

ScreenshotNeo provides a website screenshot API when your research needs a stable visual record of a page or a browser-rendered source. The API accepts one GET request and returns PNG, JPEG, WebP, or PDF. See the ScreenshotNeo documentation for the complete parameter list.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

For investment research, the useful controls include full-page capture with lazy images loaded, element capture by CSS selector, custom CSS and JavaScript, click actions, waits for a selector, delay, or network idle, custom headers and cookies, user-agent, authorization, timezone, geolocation, request blocking, caching with a chosen TTL, image resizing, PDF page ranges, asynchronous jobs with signed webhooks, bulk capture for up to 100 URLs per call, and a usage API. Store the returned image or PDF with its URL, timestamp, parameters, and verdict.

Cookie banners, newsletter popups, and chat widgets are removed before the shot. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers identify the page verdict and whether it was billed. An MCP server provides take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients. One thousand screenshots per month are free with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account to begin.

Performance, reliability, and cost

• Throughput: Prefer batching, caching, and incremental collection over repeatedly downloading unchanged pages.
• Latency: Browser rendering, network-idle waits, PDFs, and full-page lazy loading take longer than a static HTTP request. Use asynchronous jobs for large collections.
• Reliability: Retry transient network failures with backoff, but do not retry CAPTCHA or permission failures aggressively. Keep idempotent job identifiers and record every attempt.
• Cost: Account for bandwidth, browser CPU, proxy or data-vendor fees, storage, engineering time, and compliance review. A cheap feed can be expensive if it produces untraceable or unusable observations.
• Coverage: A source that is available to many market participants may generate correlated strategies and herding risk. Ask who else can access it and how quickly.

Troubleshooting common failures

Symptom	Likely cause	Fix
HTTP 403 or 429	Access policy or request rate limit	Review terms and robots.txt, slow down, cache results, and use an official API where available.
Empty fields	JavaScript rendering or changed selectors	Inspect the response, switch to browser rendering when justified, and add parser-change alerts.
CAPTCHA or login HTML	Bot detection or authentication wall	Classify the observation as unavailable; do not treat the interstitial as page data.
Prices change unexpectedly	Currency, locale, personalization, or promotion	Set locale and timezone, record currency, and compare repeated observations.
Historical backtest looks too good	Look-ahead, survivorship, or revision leakage	Rebuild point-in-time data and freeze the collection and transformation rules before testing.
Screenshot contains a popup	Consent or widget loaded before capture	Use a removal option or selector hide rule, then verify the page verdict and stored artifact.

Short FAQ

Is web-scraped alternative data legal?

There is no universal yes or no. The answer depends on the website, terms, access method, data fields, jurisdiction, and intended use. Review permission, privacy, and market-conduct requirements with qualified counsel for a real deployment.

Should scraped data replace filings?

No. Use it as a complementary input and reconcile it with primary disclosures and market data.

How often should I collect?

Match cadence to the hypothesis and source update rate. More frequent requests do not create more information when the page has not changed.

What makes a signal investable?

Clear provenance, stable definitions, representative coverage, point-in-time availability, independent validation, leakage controls, and a documented decision rule.

Can I publish research based on scraped pages?

Yes only with appropriate permission, attribution, methodology, conflict controls, and disclosures. CFA standards prohibit misrepresentation and require independence and objectivity.