How to Automate Real Estate Data Extraction: Scheduled Property Listings

Automate property-listing updates by requesting authorized data from the relevant MLS or provider, then scheduling incremental pulls against its API. For U.S. MLS data, the common standards-based path is the RESO Web API: it uses REST, OData V4, JSON, OAuth, and metadata. RESO defines standards; it does not provide listing data, credentials, or data rights. Access and permitted use come from the MLS or provider. [RESO Web API; RESO FAQ]

The reliable pattern is: authorize access, inspect that provider’s metadata, do a bounded initial backfill, and then poll for records whose modification timestamp exceeds a saved watermark. Paginate every result, upsert by a stable provider listing key, retain the raw payload for debugging, and periodically reconcile a wider window. Set the schedule according to provider limits and your freshness requirement.

1. Get an authorized data source

Start with the local MLS or an approved listing-data provider. Ask for the API endpoint, OAuth registration details, permitted geographic scope, rate limits, pagination rules, replication guidance, and the agreement that governs your use. A RESO certification is about conformance to standards; it is not a credential or a license to display listings.

RESO explicitly says it does not provide MLS data, property records, or access to other organizations’ APIs. Access is market-specific. In the United States, RESO’s certification page reported 484 functioning MLS systems and at least 90% industry adoption of certified Web API services as of September 28, 2026. These figures describe the ecosystem, not universal availability to every developer. [RESO FAQ; RESO certification]

For Canada, REALTOR.ca DDF offers authenticated RESO/OData access to Property and related resources; brokerage-owner permissions control access. Zillow documents APIs for property details, postings, valuation, mortgage, reviews, and directories, but developer approval and branding/display terms apply. Confirm current terms with each provider before building against it. [REALTOR.ca DDF; Zillow APIs]

2. Inspect the API and its metadata

RESO Web API is RESTful, uses OData V4 conventions, returns JSON, and supports OAuth authentication. Its metadata describes the resources and fields exposed by that service. The RESO Data Dictionary standardizes common concepts such as Property, Member, Office, and Media, but local providers can differ in which fields they expose and how they populate them. Read the actual endpoint metadata before designing your schema. [RESO Web API; RESO Data Dictionary]

Use metadata to answer practical questions: which listing key is stable, which modification timestamp is filterable, what status values occur, how media is represented, and which fields are required or restricted? Do not assume that a field name means every MLS supplies the same data. Keep provider-specific values in the raw record and map only understood fields into your normalized model.

3. Design the scheduled sync

Initial backfill

Run a bounded backfill by geography, status, or date window rather than requesting an unbounded dataset in one operation. Persist every page as it is processed. Respect any provider-defined maximum page size and request limits. Save the provider’s stable listing identifier and modification timestamp.

A scheduled sync filters by modification time, processes every page, and upserts records before advancing its watermark.

Incremental polling

After the backfill, query records changed since the last successful watermark. RESO describes replication as ongoing requests for the most recent changes and supports replication and queue-oriented capabilities. Use the provider’s documented replication pattern where available. [RESO Web API; RESO FAQ]

Use a small overlap window to reduce the chance that clock skew, delayed indexing, or a failure near the boundary loses an update. For example, if a run completed at 10:00, the next query might begin at 09:55, subject to provider limits. Deduplicate by listing key and modification timestamp. Advance the high-water mark only after all pages have been processed and writes have succeeded.

Storage and change handling

Store the raw provider payload with retrieval time and source identifier for audit and replay.
Upsert normalized records by the provider’s stable listing key, not by address alone.
Track status transitions, including active, pending, sold, and withdrawn where supplied.
Track media changes separately; URLs or media ordering may change even when core property fields do not.
Process deletions or removals only according to the provider’s explicit feed semantics. A missing record in one incremental response is not by itself proof of deletion.
Put malformed or unmappable records in a dead-letter queue with the raw payload and error so they can be corrected and replayed.

4. Runnable scheduling example in Python

This example shows the orchestration pattern around an authorized OData endpoint. Replace the endpoint, token acquisition, resource, key, filter fields, and pagination handling with values from your provider’s documentation. It deliberately does not invent a universal MLS endpoint or OAuth token URL.

import os
import time
import requests
from datetime import datetime, timedelta, timezone

API_ROOT = os.environ["MLS_API_ROOT"].rstrip("/")
ACCESS_TOKEN = os.environ["MLS_ACCESS_TOKEN"]
# Persist this value in durable storage after a successful run.
watermark_text = os.environ.get("LAST_SUCCESSFUL_WATERMARK")

if watermark_text:
    watermark = datetime.fromisoformat(watermark_text.replace("Z", "+00:00"))
else:
    # Initial backfill should be bounded and configured for your provider.
    watermark = datetime.now(timezone.utc) - timedelta(days=7)

# Re-read a short overlap. Choose it based on provider behavior and limits.
query_start = watermark - timedelta(minutes=5)
query_start_text = query_start.isoformat().replace("+00:00", "Z")

session = requests.Session()
session.headers.update({
    "Authorization": f"Bearer {ACCESS_TOKEN}",
    "Accept": "application/json",
})

url = f"{API_ROOT}/Property"
params = {
    "$filter": f"ModificationTimestamp gt {query_start_text}",
    "$orderby": "ModificationTimestamp",
    "$top": "100",
}
rows_received = 0
new_watermark = watermark

while url:
    response = None
    for attempt in range(5):
        response = session.get(url, params=params, timeout=30)
        if response.status_code in (401, 403):
            raise RuntimeError(f"Authorization failed: {response.status_code}")
        if response.status_code not in (429, 500, 502, 503, 504):
            break
        if attempt == 4:
            response.raise_for_status()
        retry_after = response.headers.get("Retry-After")
        delay = float(retry_after) if retry_after else min(2 ** attempt, 30)
        time.sleep(delay)

    response.raise_for_status()
    page = response.json()
    records = page.get("value", [])

    for record in records:
        # Implement these with your durable database and provider's field names.
        save_raw_payload(record)
        upsert_listing(record)
        rows_received += 1
        changed = record.get("ModificationTimestamp")
        if changed:
            parsed = datetime.fromisoformat(changed.replace("Z", "+00:00"))
            new_watermark = max(new_watermark, parsed)

    # OData services commonly provide a next-page link. Follow the exact
    # pagination mechanism documented by your provider.
    url = page.get("@odata.nextLink")
    params = None  # nextLink may already contain its query parameters

# Commit only once every page and record has been processed successfully.
store_watermark(new_watermark.isoformat().replace("+00:00", "Z"))
print({"rows_received": rows_received, "watermark": new_watermark.isoformat()})

save_raw_payload, upsert_listing, and store_watermark are intentional database integration points. Implement them transactionally or with idempotent writes. Some providers use a different timestamp literal format, pagination link, or replication endpoint; follow the service metadata and documentation.

5. Schedule the job and make it observable

Run the sync from a scheduler such as your cloud platform’s scheduled-job service, a queue worker, or cron. A 5–15 minute interval can be a starting point only when the provider allows it and the freshness requirement justifies it. Provider limits and service-level expectations determine the actual interval.

For every run, record the start and finish times, source endpoint, filter window, page count, records received and upserted, rejected records, retries, API errors, and committed watermark. Alert when a run is late, repeatedly fails, returns zero records unexpectedly, or commits the same watermark several times. Keep enough logs to explain what happened without logging access tokens or sensitive credentials.

Use a wider periodic reconciliation window in addition to incremental polling. It can catch updates missed because of outages, delayed provider indexing, or a bug in your cursor logic. Reconciliation should remain bounded and follow provider retention and rate constraints.

6. Reliability, performance, and cost

Retries: Retry transient network failures, HTTP 429 responses, and selected 5xx responses with exponential backoff. Honor Retry-After when supplied. Do not repeatedly retry 401 or 403; stop and alert because credentials or permissions need attention.
Idempotency: Upserts keyed by provider listing ID make overlapping windows and retrying a page safe. Use a transaction or checkpoint design so a partially processed run does not advance the watermark.
Pagination: Process all pages; a successful first page is not a complete sync. Avoid excessive concurrency unless the provider explicitly permits it.
Efficiency: Request only needed fields when supported, keep incremental windows small, and separate media retrieval from listing metadata when the provider’s rules and API support that pattern.
Cost: Provider charges, storage, compute, and media transfer depend on your contract and workload. Estimate from request frequency, page size, listing volume, retention, and media handling. There is no universal RESO price.
Recovery: Archive raw payloads, retain run checkpoints, and make failed records replayable. Test recovery on a bounded date range before relying on it for production.

7. Permissions and display rules

Check the governing agreement before storing or showing any field or image. Confirm rules for internal analytics, public display, attribution, photo storage, historical retention, derivative fields, and redistribution to third parties. A standardized API does not grant those rights. The MLS/provider agreement controls permitted use. Zillow likewise conditions API use on approval and branding/display requirements. [RESO FAQ; Zillow developer terms]

RETS is a deprecated transport. For new integrations, use a current RESO Web API where available; RESO identifies Web API as the modern transport and says RETS is no longer supported by RESO. [RESO Web API; RESO FAQ]

8. Troubleshooting

Symptom	Likely cause	Fix
401 Unauthorized	Expired token, incorrect audience, or wrong authentication flow.	Refresh credentials using the provider’s documented OAuth flow; verify token scope and endpoint audience. Stop retries until corrected.
403 Forbidden	Account lacks the resource, geography, or field permission.	Check the agreement and provider authorization; request the needed entitlement from the MLS/provider.
400 on a filter	OData syntax, timestamp literal, or field name is unsupported.	Inspect service metadata and provider examples; validate the filter against that service’s exact field types.
Only the first page is stored	Pagination link or continuation token was ignored.	Follow the provider’s next-link/token mechanism until there are no more pages; log page count.
Duplicate rows	Overlapping windows are not deduplicated or writes use a non-stable key.	Upsert by the provider listing identifier and track modification time; make retries idempotent.
Missed updates	Watermark advanced before all pages finished, no overlap, or provider indexing is delayed.	Commit watermark only after success, add a permitted overlap, and run periodic reconciliation.
Unexpectedly empty run	Wrong time zone, an advanced cursor, changed permissions, or genuinely no updates.	Log the exact filter and watermark; compare with a bounded diagnostic query and alert on anomalous zero-result runs.
429 or repeated 5xx	Rate limit or provider service issue.	Back off, honor retry headers, reduce concurrency/frequency, and contact provider support if persistent.

A screenshot can document how a listing page looks, but it does not replace an authorized structured-data feed.

Or skip the browser setup

For a listing page you are authorized to capture, ScreenshotNeo provides a one-request screenshot API. It is useful for visual monitoring or a page snapshot alongside a licensed data feed; it does not replace MLS access, grant listing rights, or turn a screenshot into structured listing data. See the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

ScreenshotNeo accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server gives AI agents tools for screenshots, page info, and PDF capture. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. Every feature is on every plan. Visit ScreenshotNeo and sign up free for 1,000 screenshots a month, no card required.

9. Short FAQ

Does RESO give me access to listings?

No. RESO defines standards. Obtain credentials and permission from the relevant MLS or provider.

How often should the sync run?

Choose the interval from the provider’s limits and your freshness requirement. A 5–15 minute cadence is only an example when permitted.

Can I use a consumer listing website as the source?

Use an authorized API/feed for canonical listing data. Do not assume page access permits automated extraction, storage, or redistribution.

Does RESO certification mean every field is available?

No. Inspect the provider’s metadata and agreement; local field availability and rights vary.

Implementation checklist: confirm access and rights; inspect metadata; backfill within bounds; persist raw and normalized records; paginate and upsert; advance watermarks only after full success; monitor runs; reconcile periodically; and follow retention and display rules.