o2coEdge Proxy
Chrome extensionSign inGet 1 GB free
Use casesE-commerce pricingSERP & rank trackingAd verificationAI training dataTravel & faresBrand monitoringSiteHow it worksPricingLearnAboutContactChrome extensionSign inGet 1 GB free
Tutorials

How to Use Rotating Proxies With Scrapy

8 min read · Updated September 28, 2026

Scrapy is built to crawl fast, which is exactly what gets a single IP blocked. The fix is to send that traffic through a rotating proxy. With o2co you don't maintain an IP list or rotation code — you point Scrapy at one endpoint and the gateway picks the exit. This guide covers the wiring, auth, sane concurrency, and what rotation on the endpoint actually does, so you can pace the crawl to match.

The endpoint

Everything routes through a single proxy endpoint with username/password auth. It speaks HTTP and SOCKS5 on the same port; Scrapy's built-in proxy middleware speaks HTTP, so that is the form used throughout:

# the four values on your dashboard:
host: HOST
port: PORT
user: USERNAME
pass: PASSWORD

# as one proxy URL:
http://USERNAME:PASSWORD@HOST:PORT

One address, one credential pair, in front of a pool of residential and datacenter exits — you don't manage a list of IPs. If rotating proxies are new to you, the rotating proxy primer explains what the endpoint is doing on your behalf.

What Scrapy supports out of the box

Scrapy's core ships an HttpProxyMiddleware, enabled by default. It reads the standard http_proxy / https_proxy environment variables and honors a per-request request.meta["proxy"] value. It also handles auth: give it a proxy URL with USERNAME:PASSWORD@ inline and it strips the credentials out and sends them as a Proxy-Authorization header on every request — for HTTPS targets, on the CONNECT that opens the tunnel.

Nothing to install and no sidecar: because the endpoint accepts HTTP proxy clients, Scrapy's native middleware talks to it directly. Everything below is configuration, not code you have to maintain.

Option A — environment variables

The most portable approach: export the proxy URL, credentials included, and run the spider. HttpProxyMiddleware picks it up with nothing to configure in settings.py:

export http_proxy=http://USERNAME:PASSWORD@HOST:PORT
export https_proxy=http://USERNAME:PASSWORD@HOST:PORT
scrapy crawl myspider

The credentials never need to be base64-encoded by hand — the middleware builds the Proxy-Authorization header from the URL, so Scrapy never has to be taught about the proxy's auth itself.

Option B — a custom downloader middleware

If you'd rather keep the proxy out of the environment — only some spiders should use it, or you want per-domain logic later — set it per request in a downloader middleware. A per-request meta["proxy"] always wins over the environment variables:

# middlewares.py
class ProxyMiddleware:
    PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

    def process_request(self, request, spider):
        # HttpProxyMiddleware runs after this and turns the inline
        # credentials into the Proxy-Authorization header.
        request.meta["proxy"] = self.PROXY
# settings.py
DOWNLOADER_MIDDLEWARES = {
    # runs before the built-in HttpProxyMiddleware (priority 750)
    "myproject.middlewares.ProxyMiddleware": 350,
}

Leave HttpProxyMiddleware enabled at its default position — it runs after yours, and it is what strips the credentials out of the URL and attaches the header, and it does so per request, so different requests can carry different usernames (that is how geo-targeting works below).

How rotation works on the endpoint

With a classic IP list, you'd write code to cycle through proxies. Skip that here — the gateway picks the exit for each connection, based on your username and your account's balancer mode:

  • Default (plain username): connections from one source machine share an exit while they keep arriving, and after about a minute idle the next one may get a new exit. A burst of requests never splits across addresses mid-flow — but a crawler that never pauses stays on the same exit.
  • Rotate (USERNAME-rotate): every new connection may leave from a different exit — consecutive connections never share one when at least two are available. This is the mode for a crawl, at the same per-GB rate.
  • Sticky mode (set on your account by the operator): one machine holds one exit for a fixed window, 10 minutes by default.
  • Some exits in the pool are residential pools that hand out a new IP per connection on their side, so the IP a target sees can change even while the exit stays the same.
# rotate: a new exit per connection
request.meta["proxy"] = "http://USERNAME-rotate:PASSWORD@HOST:PORT"

Rotation is per connection, not per request, and Scrapy keeps connections alive and reuses them, so several requests to one site can ride the same connection and the same exit. If you need a new IP on every request, send Connection: close (for example in DEFAULT_REQUEST_HEADERS) so each request opens a new connection. There is no named-session parameter; for a logged-in or multi-step flow, use the plain username so it stays on one exit. Verify what you actually get (below) rather than assuming. More on why hopping mid-flow backfires in how websites detect proxies.

Concurrency and AUTOTHROTTLE

On the plain username a busy crawler can sit on one exit for a long stretch, and even with -rotate a reused connection keeps its exit, so pacing per target domain is what keeps each address usable. Tune Scrapy to behave:

# settings.py
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 1.0

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

DOWNLOAD_TIMEOUT = 30          # default is 180 s
RETRY_ENABLED = True
RETRY_TIMES = 3                # 429 and 5xx are in RETRY_HTTP_CODES by default
# many blocks return 200 with a challenge body, not a 4xx: check the
# page in your callback and use get_retry_request() to re-queue it

AUTOTHROTTLE adapts the delay to the target's response latency, which naturally backs off when a site is straining — a much better signal than a fixed delay. Keep per-domain concurrency modest. Proxies hide your IP; they don't hide inhuman request rates. The broader checklist lives in proxies for web scraping.

Geo-targeting with the username suffix

To make a connection egress through a specific country, append -country-XX to the username, where XX is a lowercase ISO-3166 alpha-2 code (-country-DE fails auth). It combines with -rotate in either order. For example, USERNAME-country-de egresses through Germany:

# German egress: change only the username
request.meta["proxy"] = "http://USERNAME-country-de:PASSWORD@HOST:PORT"

# German egress, a new exit per connection
request.meta["proxy"] = "http://USERNAME-country-de-rotate:PASSWORD@HOST:PORT"

If you crawl several markets, pick the suffix per request in your middleware — a dict from target domain to country is usually enough — and set meta["proxy"] accordingly. The suffix selects geography; exit selection within that country works as described above. A country your plan doesn't include, or one with no healthy exit at the moment, fails the request instead of exiting somewhere else — so log those errors per country.

Verify it's working

Before a real crawl, confirm requests actually egress through the proxy, and see how often the IP changes for your account:

scrapy shell "https://api.ipify.org?format=json"
# in the shell:
>>> fetch("https://api.ipify.org?format=json")
>>> response.text   # the egress IP, not your own

# fetch again right away, then again after a minute or two idle,
# and compare the IPs. With USERNAME-rotate plus a Connection: close
# header, each fetch opens a new connection and may show a new IP

Billing note

Traffic is billed per GB, in both directions, whichever exit a request lands on: from a plan's monthly quota, then from a data balance that never expires (see pay-as-you-go vs subscription). Block pages and retries move bytes too. Scrapy's HTTP cache (HTTPCACHE_ENABLED) keeps the bill down while you develop — don't re-fetch what you've already pulled.

Recap

  • One proxy endpoint — host and port from your dashboard — with user/pass auth, as http://USERNAME:PASSWORD@HOST:PORT.
  • Scrapy's native HttpProxyMiddleware handles it — credentials in the URL, nothing to install.
  • Don't rotate yourself: the gateway picks the exit per connection. The plain username keeps a busy machine on one exit; USERNAME-rotate gives each new connection a different one.
  • Use AUTOTHROTTLE, a DOWNLOAD_TIMEOUT, and modest per-domain concurrency so you don't hammer the exit you're on.
  • Geo via the -country-XX username suffix, which combines with -rotate.

Set that up once and Scrapy does what it's good at — crawling — while o2co supplies the exits. Get 1 GB free and point a test spider at it, or compare plans on the pricing page. Rendering JavaScript-heavy pages? See Playwright with a proxy.