ScreenshotNeo
Sign in Get a free key

BlogEngineering

How to Build a Corporate Filings Monitoring Pipeline

Build a recoverable SEC EDGAR pipeline that discovers filings, stores them idempotently, retrieves and parses documents, and sends useful alerts.

By the ScreenshotNeo team4 October 20269 min read

A practical way to monitor U.S. public-company filings is to combine SEC EDGAR discovery with a durable, idempotent processing pipeline: identify issuers by CIK, discover new accessions through RSS or submissions data, persist each event before processing, retrieve the filing, parse the data your use case needs, and send an alert with a source link. Reconcile local state against official SEC data so missed polls and processing failures can be recovered.

The SEC provides REST APIs for company submissions and extracted XBRL data in JSON, RSS feeds for company and latest-filings searches, and bulk daily archives. These are public resources; a basic monitor does not inherently require a paid filing-data vendor. The SEC does not provide a universal end-to-end alert latency guarantee, so measure the path you deploy.

1. Define coverage and discovery

Start with a watch list of companies and the filing forms that matter. Resolve each issuer to its SEC Central Index Key (CIK), and keep the mapping in configuration or a database. Avoid relying on company names alone: names can be similar or change, while the CIK is the SEC identifier used to associate company data.

Choose a discovery method based on coverage and recovery needs:

Method Useful for Tradeoff
Company-search RSS A focused watch list Convenient targeted discovery; still reconcile against company submissions or filing indexes.
Latest-filings RSS/search Monitoring selected forms, companies, or broader activity Filter and persist observations; feed events do not establish a delivery-time guarantee.
Submissions REST data Issuer-specific history and reconciliation Poll conservatively and maintain a cursor or compare known accessions.
Daily archives Broad ingestion and backfills Process archive batches and track completion so a missed day can be replayed.

Official references: SEC developer resources and SEC RSS feeds.

2. Design the event record before writing a poller

Persist each discovered filing before fetching or parsing it. A recommended record includes:

  • Accession number, CIK, form type, filing date, and acceptance timestamp when available.
  • Original SEC source or filing-index URL and the discovery source.
  • First-seen timestamp from your monitor, separate from SEC filing timing.
  • Processing state, attempt count, last error, and timestamps for retrieval, parsing, and notification.

Put a uniqueness constraint on accession number, or another stable SEC filing identifier. RSS polling can surface the same item repeatedly; deduplication prevents duplicate work and duplicate alerts. Retaining the original reference makes reprocessing and audit easier. This schema is an engineering recommendation, not an SEC-mandated design.

3. A runnable polling and persistence example

The following Python example polls the submissions JSON for one CIK, inserts unseen accessions into SQLite, and prints newly observed filings. It is a minimal discovery stage, not a complete production parser or notification service. Set an identifying User-Agent, use a persistent database, and keep the polling interval conservative.

import json
import os
import sqlite3
import time
from datetime import datetime, timezone

import requests

CIK = os.environ["SEC_CIK"].strip().zfill(10)
USER_AGENT = os.environ["SEC_USER_AGENT"]  # e.g. "FilingMonitor ops@example.com"
DB_PATH = os.environ.get("DB_PATH", "filings.sqlite3")
POLL_SECONDS = int(os.environ.get("POLL_SECONDS", "300"))
URL = f"https://data.sec.gov/submissions/CIK{CIK}.json"

session = requests.Session()
session.headers.update({"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"})

with sqlite3.connect(DB_PATH) as db:
    db.execute("""CREATE TABLE IF NOT EXISTS filings (
        accession TEXT PRIMARY KEY,
        cik TEXT NOT NULL,
        form TEXT,
        filing_date TEXT,
        acceptance_datetime TEXT,
        source_url TEXT NOT NULL,
        first_seen_utc TEXT NOT NULL,
        status TEXT NOT NULL DEFAULT 'discovered'
    )""")
    db.commit()

while True:
    try:
        response = session.get(URL, timeout=(10, 30))
        if response.status_code == 429:
            # Back off instead of immediately retrying a rate-limited request.
            time.sleep(min(POLL_SECONDS * 2, 900))
            continue
        response.raise_for_status()
        payload = response.json()
        recent = payload["filings"]["recent"]
        rows = zip(
            recent["accessionNumber"], recent["form"], recent["filingDate"],
            recent.get("acceptanceDateTime", [None] * len(recent["accessionNumber"])),
            recent["primaryDocument"]
        )
        now = datetime.now(timezone.utc).isoformat()
        with sqlite3.connect(DB_PATH) as db:
            for accession, form, filing_date, accepted, primary_doc in rows:
                accession_compact = accession.replace("-", "")
                filing_url = (f"https://www.sec.gov/Archives/edgar/data/{int(CIK)}/"
                              f"{accession_compact}/{primary_doc}")
                result = db.execute(
                    "INSERT OR IGNORE INTO filings "
                    "(accession,cik,form,filing_date,acceptance_datetime,source_url,first_seen_utc) "
                    "VALUES (?,?,?,?,?,?,?)",
                    (accession, CIK, form, filing_date, accepted, filing_url, now)
                )
                if result.rowcount:
                    print(json.dumps({"event": "new_filing", "accession": accession,
                                      "cik": CIK, "form": form, "filing_date": filing_date,
                                      "acceptance_datetime": accepted, "url": filing_url,
                                      "first_seen_utc": now}))
        time.sleep(POLL_SECONDS)
    except requests.RequestException as exc:
        print(f"request failed: {exc}")
        time.sleep(min(POLL_SECONDS * 2, 900))

Run it with pip install requests, then set SEC_CIK (digits or zero-padded CIK), SEC_USER_AGENT (an identifying application name and contact), and optionally POLL_SECONDS and DB_PATH. The submissions API has a recent-filings section; for long outages, do not assume that a single recent window covers your entire gap. Reconcile against the company submissions history and use archives or filing indexes when needed.

4. Retrieve and parse for the actual use case

After discovery, retrieve the primary filing document and any exhibits your rules require. Filing types and content vary, so avoid assuming one parser handles every form. Version parsers, retain raw input or a durable filing reference, and record parse outcomes separately from discovery.

Use extracted XBRL company facts when you need standardized tagged values across filings. Use the filing itself for narrative disclosures, exhibits, and context. XBRL facts are not a substitute for reading the source when interpretation depends on the surrounding text. See the SEC API documentation and data resources.

5. Make retries safe and respect SEC access limits

The SEC fair-access guidance asks clients to make efficient requests and keep aggregate traffic at no more than 10 requests per second across all machines. Individual resources can also impose rate limits and return HTTP 429. Aggregate request budgets across workers; adding machines does not multiply the stated fair-access limit. Consult the SEC developer resources and EDGAR API Development Toolkit for current guidance.

  • Use bounded exponential backoff with jitter for transient errors and 429 responses; honor any server-provided retry guidance.
  • Use timeouts and cap retries. A request that hangs forever can stall a queue.
  • Use ETag, Last-Modified, and Cache-Control when the resource provides them, to avoid unnecessary refetches.
  • Separate discovery, retrieval, parsing, storage, and notification into retryable stages. A failure in email or chat delivery should not cause the filing to be ingested again.
  • Make each stage idempotent: repeat processing for an accession should update its state rather than create another filing or alert.

Keep the pipeline state machine simple, for example discovered → retrieved → parsed → notified, with explicit retryable and terminal error states. A dead-letter queue or error table should preserve the accession and failure reason for investigation and replay.

6. Send alerts that are useful and precise

Filter by issuer, form, and business rules before notifying. A concise alert should include the form, company name and CIK, accession number, filing link, SEC filing date or acceptance time when available, and the monitor’s first-seen time. Label extracted fields and generated summaries as derived data and link to the original filing.

Do not describe first observation as the official filing time or status. SEC filer guidance says an official filing requires an acceptance message that includes a filing date. Store acceptance status and monitor-observed time separately. See Determine the Status of My Filing.

7. Reconcile coverage and measure detection lag

On a schedule, compare stored accessions with official submissions data or filing indexes. Reconciliation catches feed gaps, poller downtime, parser failures, and delayed notifications. Provide an operator view for the last successful poll, oldest unprocessed filing, error counts, and notification backlog.

Measure detection lag as the difference between an SEC-provided filing timestamp and your first-seen timestamp, while documenting which discovery path and clocks are involved. The reviewed SEC sources publish no universal end-to-end monitoring latency benchmark. Set an internal objective only after measuring your chosen method in production; do not promise “real time” without evidence.

8. Operational timing, performance, and cost

Poll frequency, number of issuers, and reconciliation strategy determine request volume. For a watch list of N issuers polled every T seconds, the rough baseline is N/T requests per second before retries and retrievals; batch or stagger work where the chosen endpoints allow it. Keep the total under the SEC’s stated 10 requests per second guidance, and leave headroom for retries and backfills.

SEC filer submission hours are 6 a.m. to 10 p.m. Eastern Time on weekdays except federal holidays; out-of-hours submissions are processed the next business day. This is context about filer-side processing, not a polling or alerting SLA. See Submit Filings.

The SEC’s public APIs and RSS feeds mean a basic U.S. EDGAR monitor need not pay a data vendor. Your infrastructure costs depend on retention, parsing compute, queueing, and notification volume. Preserve raw references and derived records according to your operational and compliance needs.

Or skip the browser setup

Corporate filing records come from SEC EDGAR, so a browser screenshot is not a substitute for the submissions API, RSS, or filing documents. ScreenshotNeo can help when your workflow also needs a clean visual capture of a public filing page or other web page. One GET request returns an image or PDF; see the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://www.sec.gov/Archives/edgar/data/320193/000032019324000069/aapl-20230930.htm -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://www.sec.gov/Archives/edgar/data/320193/000032019324000069/aapl-20230930.htm"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://www.sec.gov/Archives/edgar/data/320193/000032019324000069/aapl-20230930.htm' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`ScreenshotNeo request failed: ${res.status}`);
const fs = await import('node:fs/promises');
await fs.writeFile('shot.webp', Buffer.from(await res.arrayBuffer()));

ScreenshotNeo removes cookie banners, popups, and chat widgets before the capture. Bot checks, blank pages, and failed loads are never billed. Its MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. See ScreenshotNeo and the API docs. Sign up for 1,000 free screenshots a month, with no card.

Troubleshooting

Symptom Likely cause Fix
HTTP 429 Aggregate request rate or resource-specific limit exceeded. Back off, reduce poll frequency, stagger workers, and honor cache headers. Do not retry in a tight loop.
Same filing appears more than once Feed or poll observations are repeated and the store has no stable uniqueness key. Deduplicate on accession number and make alert delivery idempotent.
Missing filings after downtime The recent submissions window or feed poll did not cover the full outage. Run reconciliation against submissions history or filing indexes; use archives for broad backfill.
Filing discovered but no alert Retrieval, parsing, filter, or notification stage failed independently. Inspect per-stage status and retry only the failed stage; expose a notification backlog.
Parser output is incomplete Form layouts, exhibits, or inline data differ from parser assumptions. Version parsers by form or document type, retain source references, and route unknown formats for review.
Alert timestamp differs from filing date SEC filing timing and local first-seen timing represent different events. Show both fields with explicit labels; do not treat first-seen time as official filing status.

FAQ

How do I get alerts when a company files with the SEC?

Resolve the company to a CIK, discover filings using its RSS feed or submissions data, persist new accessions, apply your form rules, and send an alert linked to the filing. Reconcile periodically to recover missed observations.

Can I monitor companies outside the United States with this pipeline?

This design covers filings available through U.S. SEC EDGAR. Other countries and regulators have separate filing systems and data interfaces.

Does an SEC RSS feed guarantee immediate delivery?

No delivery-latency guarantee is established by the cited SEC materials. Measure your own detection lag and reconcile with submissions data.

Do I need to buy a filing-data subscription?

Not for a basic monitor using the SEC’s public APIs, RSS feeds, and archives. A vendor may be useful for requirements beyond this pipeline, but the SEC sources are sufficient to build the core flow.

What should I alert on first?

Begin with a small set of forms and issuers, then expand after verifying deduplication, recovery, and notification behavior. A narrower rule set makes missed or noisy alerts easier to diagnose.

Keep reading