How to Rotate Proxies in Python with Health Scoring

To rotate proxies in Python with health scoring, keep a pool of proxy records, select only eligible records, send each request with a finite timeout, classify the outcome, and update each proxy’s score and cooldown. The score is application policy: there is no universal proxy-health formula. The example below uses recent successful probes, latency, observation age, and consecutive transport failures, with bounded retries so a request cannot loop forever.

Use proxies only for workloads and destinations you are authorized to access. Rotation does not grant permission to bypass a site’s access rules, rate limits, or terms. A destination’s 403 or 429 response can reflect destination policy rather than a broken proxy.

1. Choose a Python proxy interface

Use the HTTP client your application already depends on when possible. Requests accepts a proxies mapping per request; urllib3 provides ProxyManager; Python’s standard library provides ProxyHandler. The scoring and rotation policy lives in your application, independently of those clients.

Approach	Useful when	Proxy configuration
Requests	You already use Requests and want a concise per-request mapping.	`{"http": ..., "https": ...}`
urllib3	You need explicit pool, timeout, retry, or concurrency controls.	A `ProxyManager` per proxy endpoint.
`urllib.request`	You want the standard library or need to account for environment proxy settings.	`ProxyHandler` with explicit mappings.

Requests documents proxy mappings, request timeouts, and TLS verification in its [API reference](https://requests.readthedocs.io/en/latest/api/). urllib3 documents proxy managers, CONNECT, timeouts, retries, SOCKS support, and certificate verification in its [advanced usage guide](https://urllib3.readthedocs.io/en/stable/advanced-usage.html). Its [connection pool reference](https://urllib3.readthedocs.io/en/stable/reference/urllib3.connectionpool.html) describes pool blocking. Python documents ProxyHandler and environment-derived proxy configuration in [`urllib.request`](https://docs.python.org/3/library/urllib.request.html).

2. Install dependencies and configure endpoints

This runnable Requests example targets an endpoint you control or have permission to probe. Install Requests with python -m pip install requests. Replace the example proxy URLs with endpoints supplied through a secret manager or environment variables. Credentials embedded in proxy URLs are convenient but must never be printed or committed.

import os

PROXIES = [
    {"id": "proxy-a", "url": os.environ["PROXY_A_URL"]},
    {"id": "proxy-b", "url": os.environ["PROXY_B_URL"]},
    {"id": "proxy-c", "url": os.environ["PROXY_C_URL"]},
]

# Example format: http://user:password@proxy.example:8080
# HTTPS destinations can use an HTTP proxy via CONNECT.
# Use https:// for a TLS connection to an HTTPS proxy endpoint when supported.

Requests expects scheme-specific entries in its mapping. For an HTTPS target, set the https mapping. An HTTP proxy commonly tunnels HTTPS traffic with CONNECT; an HTTPS proxy first establishes TLS with the proxy itself. Confirm the scheme and capabilities with your proxy operator. urllib3 also supports SOCKS proxies when its SOCKS extra and PySocks dependency are installed.

3. Define health observations and an illustrative score

Keep the underlying observations as well as the summary score. A recent success ratio answers a different question from latency, and a transport timeout is different from a destination rejection. The following policy keeps at most 20 recent probe results. It combines recent success rate and a latency score, reduces scores for stale observations and failure streaks, and quarantines repeated transport failures temporarily.

A proxy rotation loop selects an eligible endpoint, observes the result, and updates health state.

The weights and thresholds are examples to tune for your workload, not a standard or benchmark. Probe an authorized endpoint with a finite timeout. Treat status codes such as 403 and 429 as destination outcomes; they do not by themselves prove the proxy transport is dead.

from collections import deque
from dataclasses import dataclass, field
from datetime import datetime, timezone, timedelta
import time

@dataclass
class ProxyState:
    id: str
    url: str
    observations: deque = field(default_factory=lambda: deque(maxlen=20))
    consecutive_transport_failures: int = 0
    last_success_at: float | None = None
    cooldown_until: float = 0.0

    def score(self, now: float | None = None) -> float:
        now = time.time() if now is None else now
        if not self.observations:
            return 50.0  # Unknown: eligible for an initial probe, not proven healthy.
        success_ratio = sum(ok for ok, _latency, _kind in self.observations) / len(self.observations)
        successful_latencies = [lat for ok, lat, _kind in self.observations if ok and lat is not None]
        # Illustrative: full latency credit at or below 0.5s, tapering to zero at 3s.
        mean_latency = (sum(successful_latencies) / len(successful_latencies)
                        if successful_latencies else 3.0)
        latency_score = max(0.0, min(1.0, (3.0 - mean_latency) / 2.5))
        raw = 70 * success_ratio + 30 * latency_score
        if self.last_success_at is not None:
            age_hours = max(0.0, (now - self.last_success_at) / 3600)
            recency_factor = max(0.5, 1.0 - age_hours / 48)
            raw *= recency_factor
        raw -= min(40, 10 * self.consecutive_transport_failures)
        return round(max(0.0, min(100.0, raw)), 1)

    def eligible(self, now: float | None = None) -> bool:
        now = time.time() if now is None else now
        return now >= self.cooldown_until

    def record(self, ok: bool, latency: float | None, kind: str) -> None:
        now = time.time()
        self.observations.append((ok, latency, kind))
        if ok:
            self.last_success_at = now
            self.consecutive_transport_failures = 0
            self.cooldown_until = 0.0
        elif kind == "transport":
            self.consecutive_transport_failures += 1
            # Exponential quarantine, capped at five minutes.
            delay = min(300, 5 * (2 ** (self.consecutive_transport_failures - 1)))
            self.cooldown_until = now + delay

This example’s score is a ranking hint rather than a promise that the next request will work. Unknown proxies begin at a neutral score so they can receive a probe. A production policy may separate a scheduled health probe from user traffic, use a longer observation window, track different destinations, or make stale proxies ineligible until rechecked. Retain failure categories for diagnosis instead of collapsing every failure into a single Boolean.

4. Select, request, classify, and rotate

The next code uses Requests, round-robin ordering among eligible proxies, and one retry after a transport failure. It does not retry a destination policy response, and the example sends a GET, which is ordinarily safe to repeat when the endpoint’s semantics are read-only. For writes or other operations with side effects, retry only when the operation is explicitly safe to replay, such as when the server supports an idempotency key.

import requests
import time
from requests.exceptions import ProxyError, ConnectTimeout, ReadTimeout, SSLError, RequestException

TARGET = "https://your-authorized-health-endpoint.example/health"
TIMEOUT = (3.0, 8.0)  # connect timeout, read timeout

states = [ProxyState(item["id"], item["url"]) for item in PROXIES]
next_index = 0

def choose_proxy():
    global next_index
    eligible = [p for p in states if p.eligible()]
    if not eligible:
        # Wait only within a caller-owned deadline in production; never spin.
        raise RuntimeError("No proxy is currently eligible")
    # Round-robin is simple and avoids a single proxy taking every request.
    eligible.sort(key=lambda p: (-p.score(), p.id))
    chosen = eligible[next_index % len(eligible)]
    next_index += 1
    return chosen

def get_with_rotation(url: str, max_attempts: int = 3):
    attempts = min(max(1, max_attempts), len(states))
    tried = set()
    last_error = None
    for _ in range(attempts):
        candidates = [p for p in states if p.eligible() and p.id not in tried]
        if not candidates:
            break
        proxy = max(candidates, key=lambda p: (p.score(), p.id))
        tried.add(proxy.id)
        started = time.monotonic()
        try:
            response = requests.get(
                url,
                proxies={"http": proxy.url, "https": proxy.url},
                timeout=TIMEOUT,
                verify=True,
            )
            latency = time.monotonic() - started
            # A response means the proxy transported the request. Keep status
            # as a separate observation; 403/429 are destination policy signals.
            transport_ok = True
            proxy.record(transport_ok, latency, "response")
            return response
        except (ProxyError, ConnectTimeout, ReadTimeout) as exc:
            proxy.record(False, None, "transport")
            last_error = exc
        except SSLError as exc:
            # Certificate failures are security/configuration failures. Do not
            # turn off verification; surface the issue for investigation.
            proxy.record(False, None, "tls")
            raise
        except RequestException as exc:
            proxy.record(False, None, "request")
            last_error = exc
    if last_error:
        raise last_error
    raise RuntimeError("No eligible proxy remained within the retry limit")

if __name__ == "__main__":
    response = get_with_rotation(TARGET)
    print("HTTP status:", response.status_code)
    print("Response bytes:", len(response.content))
    print("Scores:", {p.id: p.score() for p in states})

In a multi-threaded or multi-process service, protect selection and state updates with a lock or move them into a shared coordinator. This in-memory demonstration is for one process. The selector chooses the highest score and uses an advancing index in its tie ordering; if you prefer even distribution, implement strict round-robin among eligible records or weighted random choice. A proxy pool is application state, not the HTTP connection pool.

5. Probe health separately from business requests

A background probe can refresh stale measurements before user traffic needs a proxy. Use a health URL you control, expect a specific response, and keep probe frequency modest. If a target endpoint rate limits probes, use a separate authorized health service or lower the frequency. A successful TCP connection alone is weaker evidence than an application-level response, but the probe must not become a load source.

Set a connect timeout and a read timeout so a dead endpoint cannot block the worker indefinitely.
Record status, elapsed time, timestamp, and failure category for every probe.
Count a response as transport success separately from whether its HTTP status is acceptable for your application.
Quarantine repeated transport failures with a bounded cooldown, then allow a limited recheck.
Expire or discount old observations so yesterday’s result does not dominate today’s selection.

A 403, 429, or other status should be interpreted against the endpoint’s documented behavior. Do not respond to a destination rate limit by increasing rotation or retry volume. Honor the destination’s policy and reduce or stop requests as required.

6. urllib3 and standard-library variants

urllib3 with ProxyManager

urllib3’s manager has a request style similar to a regular pool manager. Construct a manager for the selected proxy and set explicit timeouts. The following compact example shows a single request; wrap this in the same selection, observation, and bounded retry policy shown above.

import urllib3

proxy_url = "http://user:password@proxy.example:8080"
http = urllib3.ProxyManager(
    proxy_url,
    num_pools=1,
    maxsize=10,
    block=True,
    cert_reqs="CERT_REQUIRED",
)
response = http.request(
    "GET",
    "https://your-authorized-health-endpoint.example/health",
    timeout=urllib3.Timeout(connect=3.0, read=8.0),
    retries=False,  # own retry budget is managed by the rotation layer
)
print(response.status, len(response.data))

Keep certificate verification enabled and configure an appropriate CA bundle if your environment requires one. urllib3 supports retries at request or pool level; if you use its retry facilities, set the allowed methods, status handling, and total budget deliberately. Avoid stacking a hidden library retry loop on top of the rotation loop, which can multiply attempts. block=True with a bounded maxsize caps concurrent connections in that pool and can help avoid flooding a host. Pool limits and proxy selection solve different problems.

Python standard library with ProxyHandler

from urllib.request import ProxyHandler, build_opener

proxy_url = "http://user:password@proxy.example:8080"
opener = build_opener(ProxyHandler({
    "http": proxy_url,
    "https": proxy_url,
}))
with opener.open(
    "https://your-authorized-health-endpoint.example/health",
    timeout=8,
) as response:
    body = response.read()
    print(response.status, len(body))

By default, ProxyHandler can use proxy settings from the environment, such as http_proxy. Pass an empty mapping to disable autodetected proxies, or an explicit mapping to make routing predictable. This matters when a script behaves differently in a shell, container, or server because inherited environment variables changed.

7. Security, performance, and cost controls

TLS: Keep HTTPS certificate verification enabled. Proxying changes the route; it does not make certificate checks unnecessary. Investigate certificate-chain or proxy configuration errors instead of setting verify=False.
Credentials: Store proxy credentials in environment-backed secrets or a secret manager, redact URLs from logs, and rotate credentials under your organization’s policy.
Timeout budget: Choose connect and read timeouts from the caller’s overall deadline. With up to N sequential attempts, worst-case waiting can approach N times the per-attempt timeout, plus backoff. Stop when the overall deadline is reached.
Concurrency: Bound worker count and per-host connection pools. A large proxy list does not control aggregate request rate. Apply the destination’s documented rate limits.
Retries: Retry only transient transport failures and only safe operations. Do not retry indefinitely, and avoid retrying TLS verification failures as if they were ordinary outages.
Observability: Track attempts, latency distributions, categorized errors, cooldowns, score changes, and the age of the latest successful observation. Do not put raw credentials in labels or telemetry.
Cost: Proxy costs depend on your provider’s plan and metering model; the scoring design itself adds application compute and probe traffic. Measure probe volume and account for requests that consume provider bandwidth or per-request quotas.

Proxy routing and TLS certificate verification are separate concerns.

8. Troubleshooting

Symptom	Likely cause	Fix
`ProxyError` or connection refused	Wrong endpoint, port, scheme, credentials, or proxy unavailable.	Check the endpoint with the provider, verify the scheme, and record a transport failure with cooldown.
Connect or read timeout	Proxy unreachable, destination slow, or timeout too strict.	Separate connect/read limits, inspect latency history, and size both to the caller deadline.
403 or 429 response	Destination policy, authorization, or rate limiting.	Do not classify it automatically as a dead proxy. Follow destination rules and lower or stop request volume.
TLS certificate error	Invalid interception setup, missing CA, hostname mismatch, or wrong proxy scheme.	Verify proxy configuration and trusted CA setup. Keep verification enabled.
Unexpected direct connection	Wrong mapping key, scheme mismatch, or client using environment settings differently.	Set both relevant scheme mappings explicitly and inspect environment proxy variables.
All proxies are cooling down	Failures exceeded the policy threshold or cooldowns outlast the request deadline.	Return a controlled unavailable result, wait within an explicit deadline, then run a limited authorized recheck.
Score looks healthy but requests fail	Probe endpoint differs from the real destination, observations are stale, or destination outcomes were conflated.	Track age and destination context; keep transport and HTTP policy results separate.
Unexpectedly high request volume	Retries are nested across the client, rotation layer, and caller.	Set one total attempt budget and disable or account for lower-level retries.

9. Screenshot API option for browser captures

If your authorized workload is specifically website screenshot capture, you can use a screenshot API instead of maintaining browser infrastructure. [ScreenshotNeo](https://screenshotneo.com) is a website screenshot API and MCP server from Yorker Media. Its screenshot endpoint does not require you to configure proxy rotation in your Python client; it is not a general-purpose proxy pool.

Or skip the browser setup

One GET request returns an image or PDF. See the [ScreenshotNeo API documentation](https://screenshotneo.com/docs/) for configuration and options.

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before capture; bot checks, blank pages, failed loads, timeouts, and cache hits are never billed. Its MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 screenshots. Sign up for [1,000 free screenshots a month](https://screenshotneo.com/account/sign-up/) with no card.

10. Frequently asked questions

What is a good proxy health score?

There is no universal score. Use a score only as an explainable selection aid, preserve raw observations, and tune weights against authorized workload outcomes.

Should every proxy be tested before every request?

Usually not. A bounded background probe or a request outcome can refresh state. Per-request probes add latency and traffic; choose freshness requirements that fit your service.

Can I use one score for every destination?

Only if the same destinations and network paths are representative. A proxy may reach one authorized service successfully while another has different latency or policy behavior.

Does proxy rotation make a request anonymous?

No. It changes a network route. It does not guarantee anonymity, authorization, or compliance with a destination’s rules.