ScreenshotNeo

BlogHow-to

How to Scrape Reddit with Python: API Access, Pagination, Rate Limits, and Compliance

Learn how to collect Reddit data with Python using OAuth, pagination, rate-limit handling, deletion workflows, and compliant storage.

By the ScreenshotNeo team30 September 202610 min read

How to Scrape Reddit with Python: API Access, Pagination, Rate Limits, and Compliance

Direct answer: Use Reddit’s authenticated Data API with OAuth, a registered application, and a unique descriptive User-Agent. Fetch subreddit listings page by page with the after cursor, honor the rate-limit headers, store only the fields your purpose requires, and remove deleted content from your dataset. Do not treat HTML scraping, undocumented JSON endpoints, proxy rotation, CAPTCHA bypass, or robots.txt as permission to collect Reddit data.

This guide shows a complete Python collector, equivalent cURL and Node.js requests, pagination that can resume after interruption, rate-limit handling, deletion and retention jobs, troubleshooting, and production design choices. It also explains when PRAW is useful and when direct HTTP gives you better control.

1. Decide what you are collecting

Write down the purpose before creating credentials. A small script that reads a few public posts has a different scope from a long-running subreddit monitor, moderation helper, or academic dataset.

  • Choose specific subreddits, listing types, time ranges, and fields.
  • Avoid collecting author names, profile links, or other identifiers unless they are necessary for the stated purpose.
  • Set a retention period and define how deletion requests will be applied to raw and derived data.
  • Separate raw Reddit content from aggregates such as counts, averages, or topic labels.

Reddit’s Data API Terms restrict uses such as exceeding limits, masking OAuth identity, unauthorized commercial monetization, retaining content beyond the approved use case, and training machine-learning or AI models on User Content without permission from applicable rightsholders. Commercial work, research beyond available limits, or another use that is not expressly permitted may require a separate agreement with Reddit.

2. Use an authorized access path

Register an application in Reddit’s developer settings and obtain OAuth credentials. Your client must send an OAuth access token and a unique, descriptive User-Agent, for example linux:com.example.reddit-collector:v1.0 (by /u/your_reddit_name). Do not rotate identities to evade limits or pretend to be a browser.

The official API reference documents listing endpoints and parameters such as after, before, limit, count, and show: Reddit API documentation. Reddit Help states that robots.txt is for search engines, not Data API users; a robots.txt rule is therefore not an API authorization.

Academic researchers should use Reddit for Researchers, which Reddit identifies as its official and authorized research route. If your use is commercial or exceeds normal access, contact Reddit about the agreement required for that purpose.

3. Create OAuth credentials

  1. Create a Reddit account dedicated to the collector if your organization needs separate ownership and audit trails.
  2. Open the developer application page and create a script application.
  3. Record the client ID and client secret in a secret manager or environment variables. Never commit them to source control.
  4. Choose the narrowest scopes needed. Read-only collection normally uses the read scope.
  5. Request a token from Reddit’s OAuth endpoint with HTTP Basic authentication.

For a script application, the token request looks like this:

curl -u "$REDDIT_CLIENT_ID:$REDDIT_CLIENT_SECRET" \
  -A "linux:com.example.reddit-collector:v1.0 (by /u/your_reddit_name)" \
  -d grant_type=password \
  -d username="$REDDIT_USERNAME" \
  -d password="$REDDIT_PASSWORD" \
  -d scope=read \
  https://www.reddit.com/api/v1/access_token

Use the grant type appropriate for your application and Reddit’s current authentication guidance. Keep the returned access token in memory or a protected secret store and refresh it before expiry.

4. A complete Python subreddit collector

The following example reads new posts from a subreddit, follows listing cursors, records rate-limit headers, retries transient failures, and writes newline-delimited JSON. Install the only dependency with python -m pip install requests.

import json
import os
import time
from pathlib import Path

import requests

CLIENT_ID = os.environ["REDDIT_CLIENT_ID"]
CLIENT_SECRET = os.environ["REDDIT_CLIENT_SECRET"]
USERNAME = os.environ["REDDIT_USERNAME"]
PASSWORD = os.environ["REDDIT_PASSWORD"]
SUBREDDIT = os.getenv("REDDIT_SUBREDDIT", "python")
USER_AGENT = "linux:com.example.reddit-collector:v1.0 (by /u/your_reddit_name)"
OUT = Path("reddit-posts.jsonl")

session = requests.Session()
session.headers.update({"User-Agent": USER_AGENT})

def get_token():
    response = session.post(
        "https://www.reddit.com/api/v1/access_token",
        auth=(CLIENT_ID, CLIENT_SECRET),
        data={
            "grant_type": "password",
            "username": USERNAME,
            "password": PASSWORD,
            "scope": "read",
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()["access_token"]

def fetch_listing(token, after=None):
    params = {"limit": 100, "raw_json": 1}
    if after:
        params["after"] = after
    response = session.get(
        f"https://oauth.reddit.com/r/{SUBREDDIT}/new",
        headers={"Authorization": f"Bearer {token}"},
        params=params,
        timeout=30,
    )
    if response.status_code in (429, 500, 502, 503, 504):
        reset = response.headers.get("X-Ratelimit-Reset")
        delay = min(int(reset) if reset and reset.isdigit() else 30, 300)
        time.sleep(max(delay, 1))
        response.raise_for_status()
    response.raise_for_status()
    return response

def main():
    token = get_token()
    after = None
    pages = 0
    with OUT.open("a", encoding="utf-8") as output:
        while pages < 10:
            response = fetch_listing(token, after)
            payload = response.json()["data"]
            children = payload.get("children", [])
            if not children:
                break
            for child in children:
                post = child["data"]
                record = {
                    "id": post.get("id"),
                    "retrieved_at": time.time(),
                    "subreddit": post.get("subreddit"),
                    "title": post.get("title"),
                    "selftext": post.get("selftext"),
                    "score": post.get("score"),
                    "num_comments": post.get("num_comments"),
                    "created_utc": post.get("created_utc"),
                    "permalink": post.get("permalink"),
                    "author": post.get("author"),
                }
                output.write(json.dumps(record, ensure_ascii=False) + "\n")
            output.flush()
            remaining = response.headers.get("X-Ratelimit-Remaining")
            reset = response.headers.get("X-Ratelimit-Reset")
            print(f"page={pages + 1} items={len(children)} remaining={remaining} reset={reset}")
            after = payload.get("after")
            pages += 1
            if not after:
                break
            time.sleep(1)

if __name__ == "__main__":
    main()

Set the environment variables before running it:

export REDDIT_CLIENT_ID='your-client-id'
export REDDIT_CLIENT_SECRET='your-client-secret'
export REDDIT_USERNAME='your-account'
export REDDIT_PASSWORD='your-password'
export REDDIT_SUBREDDIT='python'
python collect_reddit.py

The script saves the last cursor only in memory. For a scheduled job, persist after after each successful page so a crash can resume without restarting. Use a unique key such as the Reddit post ID to make writes idempotent.

5. Equivalent requests with cURL and Node.js

Get an access token with cURL

curl -u "$REDDIT_CLIENT_ID:$REDDIT_CLIENT_SECRET" \
  -A "linux:com.example.reddit-collector:v1.0 (by /u/your_reddit_name)" \
  -d grant_type=password \
  -d username="$REDDIT_USERNAME" \
  -d password="$REDDIT_PASSWORD" \
  -d scope=read \
  https://www.reddit.com/api/v1/access_token

Fetch one listing page with cURL

curl -H "Authorization: Bearer $REDDIT_ACCESS_TOKEN" \
  -A "linux:com.example.reddit-collector:v1.0 (by /u/your_reddit_name)" \
  "https://oauth.reddit.com/r/python/new?limit=100&raw_json=1"

Fetch pages with Node.js

const userAgent = 'node:com.example.reddit-collector:v1.0 (by /u/your_reddit_name)';
const token = process.env.REDDIT_ACCESS_TOKEN;
let after;

for (let page = 0; page < 10; page++) {
  const url = new URL('https://oauth.reddit.com/r/python/new');
  url.searchParams.set('limit', '100');
  url.searchParams.set('raw_json', '1');
  if (after) url.searchParams.set('after', after);

  const response = await fetch(url, {
    headers: { Authorization: `Bearer ${token}`, 'User-Agent': userAgent }
  });
  if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
  const listing = await response.json();
  const data = listing.data;
  for (const child of data.children) {
    console.log(JSON.stringify(child.data));
  }
  after = data.after;
  if (!after) break;
  await new Promise(resolve => setTimeout(resolve, 1000));
}

6. Pagination and listing options

Listing endpoints return a wrapper containing data.children and a cursor in data.after. Send that value as the next request's after. Stop when it is null or when your scope is complete. The before cursor supports walking backward. limit controls the page size up to the endpoint's allowed maximum; count reports how many items have already been walked and can help with ranking or time-window logic. Some endpoints support show=all to include filtered entries, subject to the endpoint and your authorization.

For incremental monitoring, save the newest observed ID and timestamp, then request new pages until you reach the saved boundary. Expect posts to move or scores to change between runs; IDs and retrieval timestamps are more reliable than list position.

7. Rate limits, retries, and reliability

Reddit Help currently lists 100 queries per minute per OAuth client for eligible free access, averaged over a ten-minute window. Treat that as a current policy figure, not a permanent guarantee. Read X-Ratelimit-Used, X-Ratelimit-Remaining, and X-Ratelimit-Reset on every response and slow down before remaining capacity reaches zero.

  • Retry 429 and temporary 5xx responses with exponential backoff and jitter.
  • Honor the reset value instead of sending a burst immediately after a failure.
  • Do not retry authentication errors, malformed parameters, or permission errors without fixing the request.
  • Use request timeouts and a bounded retry count.
  • Persist cursors and write each record atomically so a process restart does not duplicate data.
  • Log status code, endpoint, cursor, request ID if supplied, and rate-limit headers, while redacting tokens and secrets.

8. Deletion, retention, and data minimization

Reddit requires removal of deleted posts, comments, and account-linked identifiers. Reddit Help recommends routinely deleting stored user data and content within 48 hours to support compliance. Keep a deletion job separate from ingestion:

  1. Store Reddit IDs and retrieval timestamps for every record.
  2. Recheck records on a schedule appropriate to your approved use.
  3. When content is deleted or no longer permitted, remove the raw text, author identifier, and cached copies.
  4. Recompute derived aggregates when deletion changes their inputs.
  5. Record the deletion event without retaining the deleted content.

Collect only fields needed for the purpose. A sentiment dashboard may need text and timestamp but not an author's name. Encrypt data at rest, restrict access, and document who can export it.

9. Direct HTTP versus PRAW

PRAW provides Python objects and convenient iterators. It is a good fit for prototypes and scripts where readable Reddit-specific abstractions matter. Direct HTTP requests are preferable when you need exact control over OAuth exchanges, pagination cursors, headers, retry policy, raw payload logging, or a small dependency surface.

Concern Direct HTTP PRAW
Pagination Explicit cursor handling Convenient iterators, less visible transport
Headers and observability Full control and easy logging Wrapper-managed defaults
Retries Your policy and backoff Library behavior plus your handling
Testing Mock ordinary HTTP responses Mock Reddit objects or library calls
Maintenance Track API changes yourself Track PRAW and Reddit compatibility

Check PRAW's current authentication support before deploying; older manuals may describe flows that have changed.

10. Common errors and fixes

401 Unauthorized

The token is missing, expired, or sent to the wrong host. Request a fresh token, send Authorization: Bearer ..., and use oauth.reddit.com for authenticated API calls.

403 Forbidden

The account lacks the required scope, the subreddit is private, or Reddit blocked the request. Verify authorization and access instead of trying a different User-Agent.

429 Too Many Requests

Your client exceeded its current allowance. Stop sending requests, read X-Ratelimit-Reset, then resume with backoff. Multiple processes sharing one client ID can exhaust the same budget.

Empty pages or a null cursor

The listing is exhausted, filtered, or your boundary is too narrow. Inspect the raw response and stop when after is null.

Invalid or generic User-Agent

Reddit expects a unique descriptive value. Include your application name, version, platform, and Reddit username; never spoof another client.

Duplicate records after a restart

Persist the cursor only after writing a page successfully and enforce a unique constraint on the Reddit ID.

Deleted content remains in exports

Your deletion process covers the primary table but not caches, search indexes, backups, or derived files. Trace every copy and apply the same deletion workflow.

11. Performance and cost planning

The main constraints are API allowance, response size, storage, and deletion work. Request the largest practical page size, select only needed fields in your own storage, and avoid repeatedly downloading unchanged history. Queue work by subreddit and share one rate-limit-aware scheduler per OAuth client. Compress JSONL or use columnar storage for analytics, but ensure deletion can remove individual records from every format.

Reddit's API access policy and commercial terms determine your operational cost and permitted volume. Do not assume that proxy services, account pools, or HTML scraping reduce cost compliantly; they can violate policy and create unreliable data.

12. Or skip the browser setup

If your goal is to archive how a Reddit page looks rather than collect structured posts and comments, ScreenshotNeo provides a direct screenshot API. See the ScreenshotNeo API documentation for all options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://www.reddit.com/r/python -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://www.reddit.com/r/python"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://www.reddit.com/r/python' }); const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie banners, popups, and chat widgets are removed before the shot. Bot checks, blank pages, and failed loads are never billed, and response headers identify the page verdict and billing result. Its MCP server lets AI agents take screenshots, inspect page information, and capture PDFs. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

13. FAQ

Can I scrape Reddit without the API?

Unauthenticated HTML collection and undocumented endpoints are not the compliant route described by Reddit's current guidance. Use OAuth and the Data API, or obtain the agreement required for your use.

Is robots.txt permission?

No. Reddit Help says robots.txt is for search engines, not Data API users.

What is the current Reddit rate limit?

Reddit Help currently lists 100 queries per minute per OAuth client for eligible free access, averaged over ten minutes. Always follow the headers returned to your client.

Should I store deleted post text for audit purposes?

No. Keep a minimal deletion event or ID if your policy requires an audit trail, but remove the deleted content and account-linked identifiers from stored datasets.

When should I choose Reddit for Researchers?

Use that program for academic research access. Reddit identifies it as the official and authorized research route for researchers.

Can I train an AI model on collected Reddit posts?

Reddit's Data API Terms restrict using User Content to train machine-learning or AI models without express permission from applicable rightsholders. Obtain the required permission before building such a dataset.