How to Create and Deploy a Stock Data Scraper
Build a reliable stock data scraper with Python, Alpha Vantage or SEC EDGAR, validation, storage, scheduling, deployment, and monitoring.

Direct answer: create a stock data scraper as a small, replaceable pipeline with five parts: an authorized provider adapter, raw-response capture, normalization and validation, durable storage, and an idempotent scheduled job. For price history, Alpha Vantage provides symbol-based daily, weekly, monthly, and intraday time-series endpoints. For filings and company facts, the SEC provides submissions and extracted XBRL data through REST APIs on data.sec.gov. Store the original response before parsing, keep provider code behind an interface, and make every run restartable.
This design prevents a provider change from forcing a rewrite of your database or scheduler. It also makes the scraper auditable: you can identify which provider response produced each row, whether prices were adjusted, when the data was retrieved, and whether a later parser change altered the result.
1. Define the data contract before writing code
A scraper is a production data product only when its output has an explicit contract. Write these decisions down in a configuration file or README:
- Identifiers: ticker symbols for market prices, or SEC CIKs and filing types for filings.
- Interval: daily, weekly, monthly, or an intraday interval supported by your provider.
- Timezone: the timezone used for storage and reporting. Convert provider timestamps consistently.
- Adjustment policy: raw prices, adjusted close, or a separate corporate-action series.
- History: initial lookback and the amount of data refreshed on each run.
- Freshness: acceptable delay after a market session. Alpha Vantage documents end-of-day updates for its default quote endpoint; real-time or 15-minute delayed US quotes may require premium access.
- Retention: how long raw payloads and normalized rows must remain available.
- Rights: whether your provider agreement permits internal use, display, resale, or redistribution.
Alpha Vantage’s daily time-series documentation describes open, high, low, close, and volume fields and a full option covering more than 25 years of history. It also documents adjusted-close and split/dividend data, symbol parameters, API keys, and JSON or CSV output in its official documentation. Treat the documented interval and entitlement as part of your product requirements, not an implementation detail.
2. Choose an authorized source
| Use case | Source | Typical key | Important checks |
|---|---|---|---|
| OHLCV price history | Alpha Vantage time-series APIs | Symbol and API key | Interval, adjusted semantics, request limits, delayed versus real-time entitlement |
| Company submissions | SEC EDGAR APIs | CIK and filing parameters | Filing type, accession number, user-agent requirements, retrieval frequency |
| Extracted financial facts | SEC XBRL data on data.sec.gov |
CIK and taxonomy concepts | Units, fiscal periods, restatements, accession metadata |
The SEC developer resources state that company submissions and extracted XBRL data are available as JSON REST APIs on data.sec.gov. The EDGAR HTTPS file system and RSS feeds are useful when your workflow needs filing documents or search-style discovery. Keep a separate SEC adapter because filings and OHLCV bars have different identifiers, timestamps, and validation rules.

3. Create a provider adapter in Python
Keep the rest of your application independent of Alpha Vantage’s response shape. The adapter below requests daily data, preserves the raw payload, and returns stable records. Put the API key in an environment variable rather than source control.
import csv
import hashlib
import io
import json
import os
import sqlite3
import time
from datetime import datetime, timezone
from pathlib import Path
import requests
BASE_URL = "https://www.alphavantage.co/query"
RAW_DIR = Path("raw")
DB_PATH = "prices.db"
def fetch_prices(symbol: str, outputsize: str = "full") -> tuple[dict, str]:
key = os.environ["ALPHAVANTAGE_API_KEY"]
params = {
"function": "TIME_SERIES_DAILY_ADJUSTED",
"symbol": symbol,
"outputsize": outputsize,
"apikey": key,
}
response = requests.get(BASE_URL, params=params, timeout=30)
response.raise_for_status()
payload = response.json()
retrieved_at = datetime.now(timezone.utc).isoformat()
raw_bytes = json.dumps(payload, sort_keys=True).encode("utf-8")
checksum = hashlib.sha256(raw_bytes).hexdigest()
RAW_DIR.mkdir(exist_ok=True)
(RAW_DIR / f"{symbol}-{checksum}.json").write_bytes(raw_bytes)
return payload, retrieved_at
def normalize_daily(symbol: str, payload: dict, retrieved_at: str):
series = payload.get("Time Series (Daily)")
if not series:
note = payload.get("Note") or payload.get("Information") or payload
raise RuntimeError(f"Provider returned no daily series: {note}")
rows = []
for date_text, values in series.items():
row = {
"provider": "alpha_vantage",
"symbol": symbol.upper(),
"interval": "1d",
"timestamp": f"{date_text}T00:00:00+00:00",
"open": float(values["1. open"]),
"high": float(values["2. high"]),
"low": float(values["3. low"]),
"close": float(values["4. close"]),
"adjusted_close": float(values["5. adjusted close"]),
"volume": int(values["6. volume"]),
"adjustment_state": "adjusted_close_available",
"retrieved_at": retrieved_at,
}
if row["volume"] < 0 or row["high"] < row["low"]:
raise ValueError(f"Invalid OHLCV row: {row}")
rows.append(row)
return rows
def save_rows(rows):
with sqlite3.connect(DB_PATH) as db:
db.execute("""CREATE TABLE IF NOT EXISTS prices (
provider TEXT, symbol TEXT, interval TEXT, timestamp TEXT,
open REAL, high REAL, low REAL, close REAL,
adjusted_close REAL, volume INTEGER, adjustment_state TEXT,
retrieved_at TEXT,
PRIMARY KEY(provider, symbol, interval, timestamp, adjustment_state)
)""")
db.executemany("""INSERT INTO prices VALUES
(:provider, :symbol, :interval, :timestamp, :open, :high,
:low, :close, :adjusted_close, :volume, :adjustment_state,
:retrieved_at)
ON CONFLICT(provider, symbol, interval, timestamp, adjustment_state)
DO UPDATE SET open=excluded.open, high=excluded.high,
low=excluded.low, close=excluded.close,
adjusted_close=excluded.adjusted_close, volume=excluded.volume,
retrieved_at=excluded.retrieved_at""", rows)
db.commit()
def run(symbol: str):
payload, retrieved_at = fetch_prices(symbol)
rows = normalize_daily(symbol, payload, retrieved_at)
save_rows(rows)
print(json.dumps({"symbol": symbol, "rows": len(rows), "retrieved_at": retrieved_at}))
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("symbol")
args = parser.parse_args()
run(args.symbol)
Run it with:
python -m venv .venv
. .venv/bin/activate
pip install requests
export ALPHAVANTAGE_API_KEY='your-key'
python scraper.py IBM
The primary key makes reruns upserts instead of duplicate inserts. The raw directory gives you a replayable source for parser upgrades and audits.
4. Normalize, validate, and preserve provenance
Normalized rows should have stable names regardless of provider. A practical minimum is:
provider,symbol,interval, and UTCtimestamp.open,high,low,close, andvolume.adjustment_state, because raw and adjusted prices answer different questions.retrieved_at, request parameters, provider status, request ID, code version, and raw-payload checksum.
Reject or quarantine malformed records. Validate that numeric fields parse, volume is nonnegative, high is at least low, timestamps are in the expected range, and the uniqueness key is respected. Do not silently convert missing values to zero. If a provider changes a field name, an explicit failure is safer than a plausible but incorrect series.
5. Store data for both queries and replay
Use a query-friendly table for cleaned prices and immutable object storage or a raw table for original responses. SQLite or Postgres is sufficient for a small project. Larger histories can be partitioned by provider and date in object storage or an analytical database.
For SEC data, retain accession numbers, filing dates, form types, company identifiers, units, fiscal periods, and the original JSON. XBRL facts can be restated, so the retrieval timestamp and accession metadata matter when explaining why a value changed.
6. Add checkpoints, backfills, and rate-aware retries
Use two commands: an incremental refresh and a bounded backfill. The incremental job reads the last successful timestamp per symbol and requests only the required window. The backfill takes explicit start and end dates, runs at lower concurrency, and records progress after each symbol.
import random
import time
def with_backoff(operation, attempts=5):
for attempt in range(attempts):
try:
return operation()
except Exception:
if attempt == attempts - 1:
raise
delay = min(60, 2 ** attempt) + random.random()
time.sleep(delay)
Respect the provider’s documented limits. A retry should handle transient network failures and selected server errors, not authentication errors or a response that says you have exceeded a quota. Log the response status and provider message before retrying. Keep concurrency bounded, and use a request queue when the symbol list grows.
7. Schedule and deploy the scraper
Package a locked Python environment or container. Supply secrets through the host’s secret manager. Run after the relevant market session when end-of-day data is sufficient. A simple cron entry is adequate for a small installation:

15 23 * * 1-5 cd /srv/stock-scraper && .venv/bin/python scraper.py IBM >> logs/ibm.log 2>&1
For a container deployment, keep the image reproducible and mount durable storage for the database and raw payloads:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY scraper.py .
ENTRYPOINT ["python", "scraper.py"]
Managed schedulers and workers add retry policies, secret handling, logs, and alerts, but they do not remove the need for an idempotent job. Record the code version and dependency lockfile with each run. Alert on repeated request failures, empty responses, stale timestamps, unexpected row counts, duplicate rates, and schema changes.
8. Operate and review the pipeline
- Freshness: compare the newest provider timestamp with your contract.
- Completeness: track expected trading days and gaps.
- Quality: count rejected rows, nulls, impossible OHLC relationships, and negative volume.
- Reliability: measure failed requests, retry counts, checkpoint age, and last successful run.
- Change control: reconcile a sample of symbols after parser or dependency upgrades.
- Rights: re-check API terms, rate limits, entitlements, and redistribution permissions before publishing data.
Alpha Vantage notes that real-time and 15-minute delayed US market data is regulated by exchanges, FINRA, and the SEC, and that commercial users should contact sales. Design your user-facing freshness label and licensing terms around the entitlement you actually have.
9. Troubleshooting common failures
| Symptom | Likely cause | Fix |
|---|---|---|
| HTTP 401 or an authentication message | Missing, invalid, or revoked API key | Check the environment variable, account status, and that the key is not committed to source control. |
| HTTP 429, a “Note,” or a quota message | Rate or plan limit exceeded | Reduce concurrency, add bounded backoff, cache unchanged requests, and confirm the plan’s limits. |
| Empty time series | Invalid symbol, unsupported interval, market holiday, or provider response shape changed | Log the complete response, validate the symbol, and fail the run instead of writing empty data. |
| Prices appear to jump after a split | Raw and adjusted series were mixed | Store adjustment state explicitly and select one policy for each downstream calculation. |
| Duplicate rows after reruns | No deterministic uniqueness key | Use an upsert keyed by provider, symbol, interval, timestamp, and adjustment state. |
| Job reports success but data is stale | Scheduler ran while the provider returned an old or delayed series | Check the newest data timestamp and alert when it exceeds the freshness contract. |
| SEC values do not match a previous run | Restatement, amended filing, or unit/fiscal-period mismatch | Retain accession metadata and raw JSON; compare form, period, unit, and retrieval time. |
10. Performance, reliability, and cost decisions
Optimize the expensive boundaries first. Cache raw responses by symbol, interval, parameters, and retrieval window. Batch symbols only when the provider supports it; otherwise use a bounded worker pool. Parse and validate incrementally for large payloads, and partition storage by provider and date. Keep backfills separate from daily refreshes so a historical repair cannot starve current data.
Your operating cost includes provider plans, compute, storage, logging, alerting, and engineering time spent handling entitlement changes. A free or low-cost API plan can be appropriate for a personal end-of-day dashboard while being unsuitable for a commercial real-time product. The acceptable delay, history depth, and redistribution rights should be written into the product requirements before launch.
Or skip the browser setup
A stock data pipeline often needs screenshots for audit evidence, portfolio reports, or visual monitoring dashboards. If you need a clean image or PDF of a URL, ScreenshotNeo provides a website screenshot API and MCP server. One GET request returns PNG, JPEG, WebP, or PDF. Before capture it accepts cookie and consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be disabled. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the result with X-Page-Verdict and X-Billed headers. Its MCP server exposes take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.
See the ScreenshotNeo API documentation for all options, including full-page capture, device presets, retina scale, custom CSS and JavaScript, selectors, waits, blocking rules, headers, cookies, geolocation, caching, signed links, asynchronous jobs, bulk capture, and usage reporting.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
There are 1,000 screenshots per month free with no card. Paid plans start at $5 for 3,000 shots, and every feature is included on every plan. Create a free ScreenshotNeo account.
FAQ
How do I scrape stock prices legally?
Use an authorized provider, follow its rate limits and terms, and verify whether your plan permits commercial display or redistribution. Entitlement is part of the data contract.
Should I store adjusted or raw prices?
Store the adjustment state and preserve both when the provider supplies them. Adjusted prices are useful for return calculations; raw prices may be required for event analysis and reconciliation.
How often should a scraper run?
Schedule it according to the provider’s freshness and your product requirement. End-of-day data can run after the market session; intraday products need a documented interval and entitlement.
Is SQLite enough?
SQLite works for a small symbol set and one writer. Move to Postgres or partitioned object storage when concurrent jobs, history size, or team access requires it.
How do I add another provider?
Implement the same fetch_prices contract, map its fields into the normalized schema, and retain provider metadata. The scheduler and downstream queries should remain unchanged.


