ScreenshotNeo

BlogHow-to

How to Create a Zillow Scraper in Python

Learn the authorized way to collect real-estate data with Python, Playwright, and Beautiful Soup while avoiding blocked requests and policy violations.

By the ScreenshotNeo team30 September 20269 min read

How to Create a Zillow Scraper in Python

Short answer: do not build a scraper that sends unauthorized automated queries to Zillow. Zillow’s consumer Terms of Use prohibit screen and database scraping, spiders, robots, crawlers, CAPTCHA bypass, and other automated activity intended to obtain information from its Services. Zillow’s Public Records Data Terms also prohibit robots, spiders, scrapers, and similar tools. For recurring or commercial work, start with an approved Zillow API or a licensed real-estate data feed, confirm the current terms, and then build a pipeline that can be audited and maintained.

This guide shows the Python architecture for an authorized source. The same design works with a licensed Zillow endpoint, your own property site, a partner feed, or a test page that you control. It covers HTTP requests, Playwright browser automation, Beautiful Soup parsing, normalization, validation, retries, observability, storage, and failure handling. A 403 or CAPTCHA is a stop condition that requires an authorization review; it is never a reason to rotate identities or evade controls.

1. Confirm authorization before writing code

Record these details before collecting a single record:

  • The exact endpoint, feed, or page you are permitted to access.
  • The terms or license version, retrieval date, geography, intended purpose, and rate limits.
  • Whether storage, internal sharing, display, or redistribution is allowed.
  • Which credential was issued, who owns it, and how it will be rotated.

Zillow’s consumer Terms of Use prohibit automated queries, including scraping and CAPTCHA bypass. Zillow’s Data & APIs terms describe access for preapproved licensees and limit API users to the components for which they have approval. API terms can also restrict bulk access and retention, so read the agreement for your product rather than assuming that a public page is an API.

2. Choose the right collection method

Option Best use Strengths Constraints
Approved Zillow API or licensed feed Recurring, commercial, production data Documented fields, clearer authorization, more stable integration Approval, credential, display, retention, and product restrictions
Playwright in an authorized workflow JavaScript-rendered pages where automation is permitted Real Chromium, Firefox, or WebKit; request lifecycle events; sync and async APIs Heavier operations, browser version drift, markup changes
HTTP client plus Beautiful Soup Authorized static HTML or XML Lightweight, fast, easy to test Does not execute client-side JavaScript; markup can change

Playwright’s Python documentation covers installation, browser launch, and synchronous or asynchronous APIs. Beautiful Soup’s documentation describes APIs for navigating, searching, and modifying HTML and XML parse trees.

3. Install a reproducible Python environment

python -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4 lxml playwright pydantic
playwright install chromium

Pin versions in a lock file for production. Keep API keys in environment variables or a secret manager, never in source control or logs.

4. Build a layered, policy-aware pipeline

Separate transport, parsing, normalization, validation, and storage. This lets you replace a feed without rewriting business logic and makes a schema or selector change visible in one place.

A maintainable collector separates transport, parsing, normalization, validation, and storage.
A maintainable collector separates transport, parsing, normalization, validation, and storage.
from __future__ import annotations

import json
import logging
import os
import re
import time
from dataclasses import asdict, dataclass
from decimal import Decimal, InvalidOperation
from typing import Any

import requests
from bs4 import BeautifulSoup

logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("authorized_collector")

@dataclass
class PropertyRecord:
    listing_id: str
    address: str | None
    price: Decimal | None
    beds: Decimal | None
    baths: Decimal | None
    square_feet: int | None
    source_url: str
    retrieved_at: str
    parser_version: str = "1"

def fetch(url: str, *, headers: dict[str, str] | None = None) -> requests.Response:
    """Fetch an endpoint that your agreement explicitly permits."""
    response = requests.get(
        url,
        headers=headers or {},
        timeout=(10, 60),
    )
    log.info(json.dumps({
        "event": "response",
        "url": response.url,
        "status": response.status_code,
        "content_type": response.headers.get("content-type"),
    }))
    if response.status_code in (401, 403, 429):
        raise RuntimeError(
            f"access signal {response.status_code}; stop and review authorization and limits"
        )
    response.raise_for_status()
    return response

def money(value: str | None) -> Decimal | None:
    if not value:
        return None
    cleaned = re.sub(r"[^0-9.]", "", value)
    try:
        return Decimal(cleaned) if cleaned else None
    except InvalidOperation:
        return None

def number(value: str | None) -> Decimal | None:
    if not value:
        return None
    match = re.search(r"[0-9]+(?:\.[0-9]+)?", value.replace(",", ""))
    return Decimal(match.group()) if match else None

def parse_html(html: str, source_url: str, retrieved_at: str) -> list[PropertyRecord]:
    soup = BeautifulSoup(html, "lxml")
    records: list[PropertyRecord] = []
    for card in soup.select("[data-listing-id]"):
        listing_id = card.get("data-listing-id", "").strip()
        if not listing_id:
            continue
        records.append(PropertyRecord(
            listing_id=listing_id,
            address=(card.select_one("[data-address]") or {}).get_text(" ", strip=True) or None,
            price=money((card.select_one("[data-price]") or {}).get_text(" ", strip=True)),
            beds=number((card.select_one("[data-beds]") or {}).get_text(" ", strip=True)),
            baths=number((card.select_one("[data-baths]") or {}).get_text(" ", strip=True)),
            square_feet=int(number((card.select_one("[data-sqft]") or {}).get_text(" ", strip=True)) or 0) or None,
            source_url=source_url,
            retrieved_at=retrieved_at,
        ))
    return records

def validate(records: list[PropertyRecord]) -> list[PropertyRecord]:
    valid: list[PropertyRecord] = []
    seen: set[str] = set()
    for item in records:
        if item.listing_id in seen:
            log.warning("duplicate listing_id=%s", item.listing_id)
            continue
        if item.price is not None and item.price < 0:
            log.warning("negative price listing_id=%s", item.listing_id)
            continue
        if item.square_feet is not None and item.square_feet <= 0:
            log.warning("invalid square footage listing_id=%s", item.listing_id)
            continue
        seen.add(item.listing_id)
        valid.append(item)
    return valid

if __name__ == "__main__":
    permitted_url = os.environ["AUTHORIZED_URL"]
    response = fetch(permitted_url)
    now = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
    records = validate(parse_html(response.text, response.url, now))
    print(json.dumps([asdict(r) for r in records], default=str, indent=2))

The selectors in this example are intentionally generic. Replace them with documented fields from your authorized source. Prefer stable semantic attributes, JSON, or a licensed schema over positional selectors such as div:nth-child(4). Preserve raw values when the license allows it, because they help diagnose parser changes.

5. Parse a documented JSON response first

If your approved API returns JSON, skip HTML parsing. Validate the response shape and map fields explicitly:

def parse_json(payload: dict[str, Any], source_url: str, retrieved_at: str) -> list[PropertyRecord]:
    output = []
    for raw in payload.get("listings", []):
        output.append(PropertyRecord(
            listing_id=str(raw["id"]),
            address=raw.get("address"),
            price=money(str(raw["price"])) if raw.get("price") is not None else None,
            beds=Decimal(str(raw["beds"])) if raw.get("beds") is not None else None,
            baths=Decimal(str(raw["baths"])) if raw.get("baths") is not None else None,
            square_feet=int(raw["square_feet"]) if raw.get("square_feet") else None,
            source_url=source_url,
            retrieved_at=retrieved_at,
        ))
    return output

Keep a versioned schema. Typical fields include listing ID, price, beds, baths, square footage, address, coordinates, and timestamps. Store both the source timestamp and retrieval timestamp when permitted.

6. Use Playwright only when an authorized page needs a browser

import asyncio
from playwright.async_api import async_playwright

async def render_authorized(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        events = []
        page.on("response", lambda r: events.append({"url": r.url, "status": r.status}))
        await page.goto(url, wait_until="domcontentloaded", timeout=60_000)
        await page.wait_for_load_state("networkidle", timeout=60_000)
        html = await page.content()
        await browser.close()
        return html

# html = asyncio.run(render_authorized(os.environ["AUTHORIZED_URL"]))

Playwright documents request, response, requestfinished, and requestfailed events. Capture status, final URL, redirects, and failed requests for diagnostics. Do not intercept traffic to discover undocumented endpoints or to defeat access controls.

7. Retries, rate limits, and stop conditions

  • Follow the documented request rate and concurrency limits.
  • Use a small, bounded retry count with exponential backoff only for transient network failures or explicitly retryable server responses.
  • Do not retry 401, 403, CAPTCHA, robots denial, or a policy warning. Escalate for authorization review.
  • Respect Retry-After for 429 responses when your agreement permits another attempt.
  • Use idempotent job IDs so a timeout does not create duplicate records.
import random

def get_with_backoff(url: str, attempts: int = 3) -> requests.Response:
    for attempt in range(attempts):
        try:
            return fetch(url)
        except (requests.Timeout, requests.ConnectionError):
            if attempt == attempts - 1:
                raise
            delay = (2 ** attempt) + random.random()
            time.sleep(delay)
    raise AssertionError("unreachable")

8. Storage, freshness, and redistribution

Store source URL, retrieval time, parser version, authorization context, and failure reason beside each batch. Deduplicate on the source listing ID and keep a change history if your license allows retention. Validate timestamp freshness before publishing. Follow the feed’s display, attribution, retention, and redistribution rules; Zillow API terms can require transactional presentation and prohibit bulk access or retained copies.

9. Performance and cost

  • Prefer JSON or static HTTP responses: they use less CPU and memory than a browser.
  • Reuse a requests.Session for approved calls and set connection and read timeouts.
  • Run a limited number of authorized browser contexts; each Chromium process consumes substantially more memory than an HTTP request.
  • Cache only when the license permits it, and assign an explicit TTL so stale listings are not mistaken for current data.
  • Measure response latency, parse time, records per batch, error rate, duplicate rate, and freshness lag.

API or feed pricing, quotas, and permitted retention are contract-specific. Do not estimate Zillow access cost from page traffic. Budget for browser CPU, storage, monitoring, and schema maintenance when a browser workflow is approved.

10. Troubleshooting

Symptom Likely cause Fix
401 Missing or invalid credential Check the environment variable and credential scope; ask the provider to reissue access.
403 or CAPTCHA Unauthorized automation or policy control Stop. Review the agreement and use an approved API or feed.
429 Rate limit exceeded Reduce concurrency, honor Retry-After, and request a documented quota.
Empty HTML Content is client-rendered or response is an interstitial Inspect status and content type; use Playwright only if authorized.
Parser returns zero records Markup or schema changed Save a permitted fixture, compare selectors, update the parser version, and add validation alerts.
Browser timeout Slow dependency, blocked resource, or too-short timeout Capture request failures, wait for a documented readiness signal, and keep a bounded timeout.
Duplicate records Pagination overlap or retries without idempotency Deduplicate on listing ID and persist job state.
Wrong price or area Locale formatting or unit conversion Normalize decimal separators and units explicitly; retain the raw value where allowed.
ScreenshotNeo removes common consent banners, popups, and chat widgets before capture.
ScreenshotNeo removes common consent banners, popups, and chat widgets before capture.

Or skip the browser setup

If your goal is a clean image of an authorized page for QA, documentation, or an internal workflow, ScreenshotNeo provides a single screenshot request. Its consent step accepts cookie banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status.

Read the ScreenshotNeo API documentation for all options. This call targets a page you are authorized to capture:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/authorized-page -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://example.com/authorized-page"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://example.com/authorized-page' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

You can also choose full-page capture with lazy images loaded, an element selector, dark mode, device presets or a custom viewport, retina scale, custom CSS and JavaScript, clicks, waits, blocked resource types, headers, cookies, user agent, authorization, timezone, geolocation, transparency, resizing, caching TTL, signed image links, asynchronous jobs with signed webhooks, bulk capture of up to 100 URLs per call, PDF output, usage reporting, and an OpenAPI specification. ScreenshotNeo also has an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.

There are 1,000 free screenshots each month with no card. Paid plans start at $5 for 3,000 shots, and every feature is available on every plan. Create a free ScreenshotNeo account.

FAQ

Can I use Beautiful Soup directly on Zillow?

Only when the source and use are authorized. Beautiful Soup parses HTML and XML; it does not grant permission or execute JavaScript.

When should I choose Playwright?

Use it for an authorized page whose records appear only after browser JavaScript runs. Prefer a documented API or feed when one is available.

Should I rotate proxies if Zillow blocks my script?

No. A block, CAPTCHA, or 403 is a signal to stop and review authorization, limits, and the approved access path.

How do I keep a scraper from silently breaking?

Version the parser, retain permitted fixtures, validate required fields and ranges, log provenance, alert on zero records or schema changes, and monitor freshness.

Is an API key safe in a Python file?

No. Load it from an environment variable or secret manager, restrict its scope, and prevent it from appearing in logs or client-side code.