Retry Failed Requests in Python

To retry failed HTTP requests in Python, configure an urllib3.util.Retry policy on a Requests HTTPAdapter, mount it on both HTTP schemes, and set a finite timeout on each request. This example retries selected transient connection failures and responses, honors Retry-After, and uses capped exponential backoff with jitter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=4,
    connect=4,
    read=2,
    status=3,
    backoff_factor=0.5,
    backoff_jitter=0.2,
    backoff_max=30,
    status_forcelist=(429, 500, 502, 503, 504),
    allowed_methods=frozenset({"GET", "HEAD", "OPTIONS"}),
    respect_retry_after_header=True,
)

session = requests.Session()
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

response = session.get(
    "https://api.example.com/data",
    timeout=(3.05, 15),
)
response.raise_for_status()
print(response.json())

Install the dependencies with python -m pip install requests urllib3. The retry options and their behavior are described in the Requests advanced usage guide, Requests API reference, and urllib3 Retry reference.

1. What gets retried, and how many times?

Requests does not retry failed connections by default. Its adapter accepts an urllib3 Retry object for fine-grained retry behavior. Mounting the adapter on a Session applies the policy to subsequent calls made through that session. Mount both prefixes so HTTP and HTTPS requests receive the configuration.

The example sets a total retry budget of four retries, in addition to the initial attempt. In practice, the separate limits also matter: connect controls connection-related retries, read limits read-error retries, and status limits retries for selected response codes. The total limit places an overall bound on retries. Keep these numbers finite: retries consume time and server capacity, and an unbounded loop can tie up a worker indefinitely.

Setting	Purpose	Practical guidance
total	Overall retry ceiling	Choose a small finite limit based on the caller’s deadline.
connect	Connection establishment failures	Retrying may help with transient network problems.
read	Errors while reading a response	Use care: the server may have processed the request already.
status	Responses matching status_forcelist	Set the list deliberately and honor server guidance.
allowed_methods	Methods eligible for retries	Start with safe-to-repeat methods such as GET and HEAD.

2. Configure status codes and HTTP method safety

status_forcelist names the response statuses that may trigger a retry, but it does not mean every request receiving one of those statuses will be retried. The request method must also pass the allowed_methods policy. A common transient set is 429 (rate limited) and selected 5xx server errors, such as 500, 502, 503, and 504. Follow the specific API’s documented behavior; a 400 or 404 usually needs a request or application fix, not another identical attempt.

GET, HEAD, and OPTIONS are commonly safe to repeat. urllib3’s default allowed set also includes PUT, DELETE, and TRACE, which HTTP defines as idempotent methods. Idempotency depends on the operation’s real semantics, however. Do not add POST merely because it failed: a server might have completed the operation even if the client did not receive the response, so a retry could create a duplicate. Retry POST only when the endpoint explicitly supports safe replay, typically through an idempotency key or equivalent contract.

3. Backoff, jitter, and Retry-After

Immediate retries can send a burst of requests to a service that is already struggling. Exponential backoff spaces attempts out: urllib3 computes a delay from backoff_factor * 2**previous_retries, with its backoff behavior beginning after the initial retry. With a factor of 0.5, delays grow as retries accumulate. backoff_max caps the computed wait. urllib3’s default backoff factor is zero, so set one when delayed retries are appropriate.

backoff_jitter adds a random amount to the delay. This reduces the chance that many clients that failed together will all retry at the same moment. Jitter is useful in distributed systems, but it does not replace a cap or finite retry count.

When respect_retry_after_header=True, urllib3 can honor the server’s Retry-After response header for supported retry cases, including rate limits and service unavailable responses. The server’s wait instruction should take precedence over guessing a shorter delay. Account for these sleeps in your caller’s overall deadline; a retry policy can otherwise wait longer than the user-facing operation can tolerate.

4. Timeouts are separate from retries

A retry count does not set a timeout. Always pass one on every network call. Requests accepts a single timeout value or a pair in the form (connect_timeout, read_timeout). In the example, connection establishment can wait up to 3.05 seconds and the read timeout is 15 seconds.

The read timeout is not necessarily a total deadline for receiving the complete response. It measures the time allowed between socket reads. A server that keeps sending data slowly may therefore take longer overall than the configured read timeout suggests. For a strict end-to-end deadline, track elapsed time in the calling code and stop initiating attempts when the remaining budget is exhausted. Remember that retry backoff adds additional elapsed time.

5. Make it reusable and inspect failures

Build a configured session once and reuse it for calls to the same service. A reusable session can also reuse connections. Keep the policy near the client setup so different call sites do not accidentally use different retry rules.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry


def build_session() -> requests.Session:
    retry = Retry(
        total=4,
        connect=4,
        read=2,
        status=3,
        backoff_factor=0.5,
        backoff_jitter=0.2,
        backoff_max=30,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET", "HEAD", "OPTIONS"}),
        respect_retry_after_header=True,
    )
    session = requests.Session()
    adapter = HTTPAdapter(max_retries=retry)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session


session = build_session()
try:
    response = session.get(
        "https://api.example.com/data",
        timeout=(3.05, 15),
    )
    response.raise_for_status()
    payload = response.json()
except requests.exceptions.Timeout:
    # The configured retry budget or timeout was exhausted.
    raise
except requests.exceptions.ConnectionError:
    # DNS, connection, or other transport failure remained after retries.
    raise
except requests.exceptions.HTTPError:
    # A final HTTP response was unsuccessful; inspect its status and body.
    raise

Call raise_for_status() if your code should treat final 4xx and 5xx responses as exceptions. Otherwise, Requests returns the final response and your code must inspect its status. Log the operation identifier, final status or exception type, and elapsed time. Avoid logging access tokens, authorization headers, cookies, or sensitive response bodies. For precise attempt-level telemetry, use application logging or instrumentation around the client; a final exception alone may not tell an operator which intermediate responses occurred.

6. Choose the right retry approach

Approach	Good fit	Trade-off
Requests with urllib3 Retry	Existing Requests client needing method, status, redirect, and Retry-After controls	Retries happen in the transport adapter; application logic still handles final failure.
urllib3 directly	Code already using urllib3 or requiring pool-level configuration	You manage a lower-level HTTP client interface.
Tenacity	Retrying a broader operation involving HTTP plus parsing, queues, or other I/O	You must still decide which HTTP failures and methods are safe to repeat.
Manual loop	Unusual policy depending on response content or a shared deadline	You must implement attempt bounds, backoff, jitter, and correct exception handling.

urllib3’s Retry is usually the simplest fit when the retry decision is about HTTP transport and status. Tenacity provides decorator-based retries and wait strategies for general Python functions; it does not automatically understand whether a particular HTTP operation is safe to replay. See the Tenacity documentation for its retry and wait options.

7. Troubleshooting common retry problems

Symptom	Likely cause	Fix
Request fails on the first connection error	The call uses a plain Requests request, not the configured session, or the adapter was not mounted for that scheme.	Use session.get(...) and mount adapters on both http:// and https://.
429 or 503 is returned without another attempt	The status is absent from status_forcelist, the method is excluded, or the retry budget is spent.	Check the status list, allowed method, and configured limits. Honor the endpoint’s retry contract.
POST happens twice	The method was allowed without an idempotency design, or the application manually repeated it after an ambiguous failure.	Remove POST from automatic retries unless the API supports idempotency keys or safe replay.
Retries happen immediately	backoff_factor is zero or too low.	Set a deliberate factor and consider jitter and a cap.
The process waits much longer than expected	Timeouts, multiple attempts, backoff, or Retry-After waits all contribute to elapsed time.	Choose limits around an overall caller deadline; do not interpret read timeout as a total deadline.
TypeError: unexpected keyword argument 'backoff_jitter'	The installed urllib3 version does not support that option.	Upgrade urllib3 to a compatible version or omit jitter and implement an appropriate alternative.
HTTP error is not raised	Requests returns unsuccessful responses unless the caller checks them.	Call response.raise_for_status() or explicitly inspect response.status_code.
Retry code behaves differently after a dependency update	urllib3 option names and behavior depend on the installed version.	Pin and review compatible Requests/urllib3 versions; consult the installed version’s reference.

8. Performance, reliability, and cost

Retries can improve the chance of completing an operation through a brief network interruption, but every extra attempt adds latency and load. A request that succeeds on its fourth attempt takes longer and consumes more client and server resources than a successful first attempt. During an outage, retries from many clients can amplify traffic. Bound attempts, use backoff with jitter, obey Retry-After, and avoid retrying permanent client errors.

Set the policy according to the operation’s latency budget and service contract. For interactive requests, a small retry budget may be appropriate; background jobs may allow longer waits, provided the job has a deadline and does not duplicate side effects. Repeated failures should become visible through metrics and logs rather than being hidden by retries. Track final failures and latency, and distinguish transport errors from unsuccessful HTTP responses.

Retries have no direct per-attempt fee in the client library, but they can consume compute, bandwidth, API quota, and server capacity. For metered APIs, check whether unsuccessful or repeated requests count toward the provider’s limits or billing. A retry policy cannot make an unavailable service reliable on its own; it only handles some transient failures.

9. Screenshot workflows without browser setup

If your Python request is part of generating website screenshots, retries solve only transport failures. You still need a browser capture service or a browser you operate. ScreenshotNeo is a website screenshot API and MCP server. It accepts one GET request with a URL and returns an image or PDF; see the ScreenshotNeo API documentation for its request options.

Or skip the browser setup

This is a direct call you can run with Requests:

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

Or with cURL:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Or with Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before the shot. Bot checks, blank pages, and failed loads are never billed. Its MCP server gives AI agents screenshot tools, and 1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000. The API also supports PNG, JPEG, WebP, and PDF, with options such as full-page capture, element selection, device presets, custom CSS and JavaScript, request blocking, and async jobs. For details, use the docs. Sign up for 1,000 free screenshots a month, with no card required.

10. FAQ

Does Requests retry failed requests automatically?

No. Configure an adapter with urllib3 Retry, or write an explicit policy at the application layer.

Should I retry HTTP 429?

Often, if the API allows it. Respect its Retry-After header and keep a finite attempt and time budget.

Should I retry every 5xx response?

No universal list fits every API. Choose transient statuses according to the service contract and leave permanent or non-retryable failures for application handling.

Can I retry a POST request?

Only when repeating the operation is safe by contract, such as when the endpoint supports an idempotency key. A connection or read error does not prove the server failed to act.

What is the difference between timeout and retry?

A timeout limits waiting for a connection or response activity on one attempt. A retry allows another attempt after a qualifying failure. Configure both and account for their combined elapsed time.