How to Rotate Proxies in Scrapy Spiders
Learn how to assign, rotate, validate, and troubleshoot proxies in Scrapy with request metadata and custom downloader middleware.

Direct answer: Scrapy rotates proxies when your code chooses a different proxy URL for each request and stores it in Request.meta['proxy']. Scrapy’s built-in HttpProxyMiddleware reads that value and applies the proxy. Scrapy does not provide a complete proxy-pool scheduler or health-scoring system by default, so you must decide where proxy selection, retries, failure handling, and endpoint retirement belong.
The simplest implementation selects an endpoint in the spider:
yield scrapy.Request(
url,
meta={"proxy": selected_proxy_url},
callback=self.parse,
)
For a small spider, selecting from a list in the spider is enough. For several spiders, centralize the policy in custom downloader middleware. In both cases, use proxy URLs that you are authorized to use and verify that their scheme is supported by your configured download handler.
How Scrapy proxy rotation works
Scrapy’s HttpProxyMiddleware is enabled by default. It sets the HTTP proxy for a request by using the request’s proxy metadata value. It also reads the http_proxy, https_proxy, and no_proxy environment variables. A per-request value takes precedence over the HTTP(S) proxy environment variables and ignores no_proxy. The official documentation describes the mechanism in Scrapy’s HttpProxyMiddleware reference.
A proxy value can look like either of these:
http://proxy.example.net:8080
http://username:password@proxy.example.net:8080
Rotation is therefore an endpoint-selection policy:
- Keep a pool of proxy endpoint URLs.
- Select one when constructing a request, or in downloader middleware.
- Put it in
request.meta['proxy']. - Record enough information to understand which endpoint handled a request.
- Decide how retries, redirects, timeouts, and failed endpoints should be handled.
Scrapy’s documentation establishes how to pass the endpoint. It does not establish that an endpoint is alive, anonymous, suitable for a target, or permitted for your use.
Option 1: rotate in the spider
Spider-level selection is the clearest approach when one spider owns the policy. The following example cycles through a list, assigns a proxy to every request, and keeps the pool in an environment variable so credentials do not need to be committed to source control.

import os
from itertools import cycle
import scrapy
class RotatingQuotesSpider(scrapy.Spider):
name = "rotating_quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
"https://quotes.toscrape.com/page/3/",
]
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
raw = os.environ.get("SCRAPY_PROXIES", "")
self.proxies = [item.strip() for item in raw.split(",") if item.strip()]
if not self.proxies:
raise ValueError("Set SCRAPY_PROXIES to one or more proxy URLs")
self.proxy_cycle = cycle(self.proxies)
def start_requests(self):
for url in self.start_urls:
proxy = next(self.proxy_cycle)
yield scrapy.Request(
url,
meta={"proxy": proxy},
callback=self.parse,
errback=self.errback_request,
dont_filter=True,
)
def parse(self, response):
yield {
"url": response.url,
"status": response.status,
"quotes": response.css(".quote .text::text").getall(),
}
def errback_request(self, failure):
request = failure.request
self.logger.warning(
"Request failed url=%s proxy=%s error=%r",
request.url,
request.meta.get("proxy"),
failure.value,
)
Run it with:
export SCRAPY_PROXIES='http://proxy-a.example:8080,http://user:pass@proxy-b.example:8080'
scrapy crawl rotating_quotes -O quotes.json
itertools.cycle gives deterministic round-robin selection. That is useful for a predictable test, but it is not a health-aware pool. A dead endpoint will continue to receive traffic until you remove it or add failure handling.
Random selection
Random selection avoids a strict sequence, but it can select the same endpoint repeatedly and makes failures harder to reproduce.
import random
proxy = random.choice(self.proxies)
yield scrapy.Request(url, meta={"proxy": proxy}, callback=self.parse)
Use a seeded random generator when you need reproducible runs. Do not treat random choice as anonymity, geographic diversity, or a guarantee that a target will accept the request.
Option 2: centralize rotation in downloader middleware
Downloader middleware is better when several spiders should share one policy. Middleware can assign a proxy after a request is created, so spider code remains unaware of the pool.

# myproject/middlewares.py
import os
from itertools import cycle
class ProxyRotationMiddleware:
def __init__(self):
raw = os.environ.get("SCRAPY_PROXIES", "")
self.proxies = [item.strip() for item in raw.split(",") if item.strip()]
if not self.proxies:
raise ValueError("SCRAPY_PROXIES is empty")
self.proxy_cycle = cycle(self.proxies)
@classmethod
def from_crawler(cls, crawler):
return cls()
def process_request(self, request, spider):
# Respect an explicit proxy selected by the spider.
if request.meta.get("proxy"):
return None
request.meta["proxy"] = next(self.proxy_cycle)
return None
Register it in settings.py:
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.ProxyRotationMiddleware": 540,
}
Scrapy combines your setting with the base middleware configuration. Middleware order matters: lower numbers are closer to the engine, while higher numbers are closer to the downloader. Check your chosen order against HttpProxyMiddleware, retry middleware, redirect middleware, cookies, authentication, and any third-party handlers. A custom policy that runs at an unexpected point can overwrite a deliberate per-request value or make retry behavior difficult to reason about.
Do not rotate accidentally on retries
A retry can either reuse the original endpoint or receive a new endpoint, depending on where selection occurs and how you implement retry handling. Decide explicitly. Reusing the endpoint helps isolate target-side failures; changing it can help when the endpoint itself is unavailable. Scrapy does not prescribe a universal pool-health algorithm.
If you need a new proxy on retry, store a retry count in metadata and select deliberately:
def process_request(self, request, spider):
retry_count = request.meta.get("proxy_retry_count", 0)
if retry_count == 0 and request.meta.get("proxy"):
return None
request.meta["proxy"] = self.choose_proxy(exclude=request.meta.get("proxy"))
return None
The surrounding retry implementation must also increment the count. Keep this policy separate from assumptions about status codes unless you have established that those responses indicate an endpoint problem in your environment.
Proxy schemes, handlers, and destination protocols
Proxy metadata is not guaranteed to work with every download handler. Scrapy documents these compatibility constraints:
| Configuration | What to check |
|---|---|
| HTTP proxy URL | Use a proxy URL such as http://host:port and test it with the configured handler. |
| HTTPS proxy URL | The built-in HTTP/1.1 handler supports HTTPS proxy URLs only for HTTP destinations. |
| SOCKS URL | Supported by HttpxDownloadHandler; other built-in handlers do not support SOCKS URLs. |
| H2DownloadHandler | The documented proxy metadata mechanism is unsupported. |
These constraints mean a proxy that works for one destination or handler may fail for another. Test the exact combination deployed in production rather than validating only the proxy endpoint with a separate client.
Environment variables and precedence
For a process-wide default, set:
export http_proxy='http://proxy.example:8080'
export https_proxy='http://proxy.example:8080'
export no_proxy='localhost,127.0.0.1'
Scrapy’s middleware observes these variables. A request-level meta['proxy'] value wins over the HTTP(S) proxy environment variables and ignores no_proxy. This is useful for intentional per-request routing, but it can surprise operators who expect local hosts to bypass a proxy. If bypass behavior matters, implement it in your selection policy instead of assuming no_proxy will override explicit metadata.
Credentials, logging, and request metadata
Proxy credentials embedded in a URL can leak through logs, exception messages, debug extensions, or exported request metadata. Keep them in environment variables or a secret manager, and redact proxy URLs before logging.
from urllib.parse import urlsplit, urlunsplit
def redact_proxy(proxy_url):
parts = urlsplit(proxy_url)
host = parts.hostname or ""
if parts.port:
host = f"{host}:{parts.port}"
return urlunsplit((parts.scheme, host, parts.path, parts.query, parts.fragment))
Never print the raw value of request.meta['proxy'] in production logs when it contains a username or password.
Testing a rotating spider
Start with a small, permitted target and a short proxy list. Verify:
- Each request receives the expected endpoint.
- The selected URL uses a supported scheme.
- Authentication is not exposed in logs.
- Failures identify the request and a redacted endpoint.
- Retries and redirects follow your intended selection policy.
- Timeouts are bounded so one endpoint cannot stall the crawl indefinitely.
For a simple connectivity check, you can use cURL with the same proxy URL:
curl --proxy "http://proxy.example:8080" --max-time 30 https://example.com/
Node.js can make a direct request through an HTTP proxy with an agent supplied by a proxy-agent package. The exact agent package and handler must match your deployment; Scrapy’s own compatibility rules still apply to the Scrapy process.
import { request } from 'node:https';
request('https://example.com/', { timeout: 30000 }, response => {
console.log(response.statusCode);
response.resume();
}).on('error', console.error).end();
This Node example is a connectivity check, not a replacement for Scrapy’s middleware configuration.
Common errors and fixes
“The proxy is ignored”
Cause: The request does not contain meta['proxy'], the custom middleware did not load, or another component overwrote the metadata.
Fix: Log the redacted value immediately before download, confirm the middleware path in DOWNLOADER_MIDDLEWARES, and inspect middleware order. Remember that environment variables apply defaults; they do not prove that your rotation code is running.
Proxy authentication failures
Cause: Incorrect credentials, URL encoding problems, or a provider that expects a different authentication method.
Fix: URL-encode reserved characters in usernames and passwords, validate the same URL outside Scrapy, and keep credentials out of logs.
HTTPS requests fail through an HTTP proxy
Cause: Proxy URL and destination protocol compatibility differs by handler. Scrapy documents specific limitations for HTTPS proxy URLs and HTTP destinations.
Fix: Confirm the destination protocol, configured download handler, and proxy scheme. Test the exact combination rather than changing only the URL string.
SOCKS proxy errors
Cause: SOCKS URLs are not supported by every built-in handler.
Fix: Use the handler documented to support SOCKS, such as HttpxDownloadHandler, or use a compatible HTTP proxy.
Requests hang
Cause: An endpoint is slow or unavailable, and the timeout is too high or absent.
Fix: Set bounded download and connection timeouts, record elapsed time per endpoint, and define an operational rule for temporarily removing consistently failing endpoints.
Redirects use an unexpected endpoint
Cause: Redirect middleware creates or modifies a follow-up request, or your custom middleware selects again.
Fix: Trace the metadata across the redirect chain and choose whether redirects should retain the original proxy. Make the rule explicit in middleware.
Retries amplify traffic
Cause: A retry policy can multiply requests when several endpoints fail.
Fix: Bound retry counts, distinguish endpoint failures from target responses, and measure attempts separately from successful pages. Scrapy does not provide a universal proxy-health threshold.
Performance, reliability, and cost considerations
Rotation adds connection setup, DNS, authentication, and possible TLS overhead when endpoints are not reused. Measure throughput and latency with the same handler, concurrency, timeout, and proxy mix used in production. A larger pool is not automatically faster: slow endpoints can reduce effective concurrency, and excessive parallelism can violate a site’s access rules.
Keep endpoint selection cheap. A cycle or indexed list is constant-time. If selection consults a remote pool service, protect the spider from that service becoming a single point of failure. Cache configuration at startup where appropriate, and make pool updates observable.
Track at least request count, response status, timeout count, retry count, elapsed time, and failures grouped by a redacted endpoint identifier. These measurements let you distinguish target behavior from endpoint health without exposing credentials.
Proxy service pricing, availability, geographic coverage, and usage terms vary. The Scrapy documentation does not establish a provider comparison or current commercial terms, so select a service based on your authorization, target requirements, protocol compatibility, and operational controls.
Or skip the browser setup
Scrapy is the right tool when you need a programmable crawl and proxy policy. If your actual requirement is a clean image or PDF of a page, ScreenshotNeo removes browser setup and exposes a single capture endpoint. It accepts the cookie or consent banner like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and responses identify the result with X-Page-Verdict and X-Billed headers. It also provides an MCP server with take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.
See the ScreenshotNeo API documentation for the complete option list, including full-page capture, CSS-selector element capture, dark mode, device presets, custom viewports, retina scale, PDF paper sizes and margins, custom CSS and JavaScript, clicks, waits, request blocking, headers, cookies, user agents, timezone, geolocation, transparent backgrounds, resizing, caching, signed links, asynchronous jobs, webhooks, bulk capture, usage data, and the OpenAPI specification.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
There are 1,000 screenshots per month free with no card. Paid plans start at $5 for 3,000 shots, and every feature is available on every plan. Create a free ScreenshotNeo account to try the capture API.
FAQ
Does Scrapy rotate proxies automatically?
No. Its built-in middleware applies a proxy value, while your spider or custom middleware chooses that value.
Can I set a proxy for only one request?
Yes. Pass the endpoint in that request’s meta={"proxy": "http://host:port"}.
Should I rotate on every request?
That is a policy choice. Consider session requirements, endpoint reliability, target rules, connection reuse, and retry behavior before choosing per-request rotation.
Why does no_proxy not bypass my selected proxy?
An explicit per-request proxy takes precedence and ignores no_proxy. Apply bypass logic before setting the metadata value.
Can I use SOCKS proxies?
Only with a compatible handler. Scrapy documents SOCKS support for HttpxDownloadHandler; other built-in handlers do not support SOCKS URLs.
Where should proxy health logic live?
Put it in custom downloader middleware when several spiders need the same policy. Keep the rules for retries, timeouts, redirects, and endpoint retirement explicit and observable.


