Stop writing null prices over good records.
Soft blocks on the large marketplaces can arrive as HTTP 200 with the price block stripped, and your pipeline stores it as fact.
Example Brand Wireless Earbuds, Active Noise Cancelling, 36h Case
Enter the characters you see below
MXQ7FNExample Brand Wireless Earbuds, Active Noise Cancelling, 36h Case
Enter the characters you see below
MXQ7FNSample product page, illustrative timings. The block-page wording is the real robot check.
A null price overwrites a record that was right
A hard block you can at least detect. The dangerous response is HTTP 200 with the page shell intact and the price block simply absent — a null overwrites a good record, and your dashboard reports a competitor delisting that never happened. Your own retries then multiply both the pressure and the bill.
Judged on the ASN, not on your stack→Residential and mobile egress
ASN reputation can be scored before the path is ever parsed. A datacenter range another tenant already burned on this host can be rejected on arrival, whatever your client stack looks like.
The pool aggregates residential, mobile and datacenter exits from 17+ upstream providers, so the request leaves from an address that belongs on a consumer route rather than a hosting range.
One health score for every host→Reputation scored per target
Conventional pools grade an exit with one target-agnostic number. An IP clean on a Shopify storefront can be burnt on a large marketplace because another tenant ran heavy volume against the same ASINs from it.
The edge tracks how each egress has been landing against the marketplace you pull, separately from the next host and from a given Shopify store. An exit cold on one still carries your pulls on the others.
Rotation resets the flow you built→Sticky mode, keyed on your source IP
The postcode you set on the site, the cart behind it and the ordinary session cookies that carry both are bound to the address that earned them. Per-connection rotation makes you a new visitor on every hop.
Ask the operator to put the account in sticky mode and every connection from that machine leaves through one exit for the window — 10 minutes by default, longer on request. A second machine gets its own exit.
You are billed for block pages→Retries happen before your socket
A block page costs the same bytes as a product page, and the retry logic sits in your code where a status line is all you can see. Your bill scales with how badly the pool is doing.
When an attempt fails at the transport, the request is re-run through another exit at the edge. Your client sees one response, and the failed attempts are absorbed rather than billed.
Before you point a crawler at it.
"99% success rate" is a marketing number. What does it mean at the socket?
It means what reaches your code. Attempts that fail inside the edge are re-run through another exit and absorbed — not returned to you, not billed. It is not a claim that a given ASIN exists, is in stock, or that the target will never change its markup. It is a claim about who eats the failed attempts.
My problem is HTTP 200 with the price block missing, not hard blocks. Does an internal retry help when the status code is fine?
Not by itself, and we will not pretend otherwise. The edge acts on what it can see — connection failures, TLS errors, block status codes, known challenge responses — and a stripped page served as a clean 200 is none of those. What per-target scoring does is keep you off the exits that host is already treating badly. Keep a field-presence check in front of the sink: no price node, no write.
Rotation breaks a multi-step pull. Can I pin one exit per ASIN?
Not per ASIN — there is no session-id parameter, so no username suffix names a pin. The username carries only the country (-country-xx) and -rotate, which does the opposite: every new connection may take a different exit. Leave -rotate off and a machine that keeps sending keeps its exit; after about a minute with no new connections, the next one may get a fresh exit. For a longer hold, ask the operator for sticky mode, keyed on your source IP: every connection from one machine leaves through the same exit until it has been idle for the window, 10 minutes by default. Several pinned chains at once means several machines.
Do I have to adopt an SDK or rewrite the crawler?
No SDK. One endpoint with a username and password, spoken as an HTTP proxy or SOCKS5 on the same port: curl -x, requests with the proxy URL, Scrapy through its proxy middleware, Go through a net/http transport, Playwright and Puppeteer with the credentials in the launch options. Nothing to install, nothing to port.
We have had abuse complaints land on us before. Where is the line?
Public pages, at a rate the target tolerates, for legitimate commercial purposes — pricing, assortment, MAP, competitive research. The AUP forbids aggressive crawling that trips anti-bot systems and generates abuse reports, and there is nothing here for login walls or authenticated flows. Absorbing failed attempts lowers your retry pressure on the target.
One credential, every job.
All six →Run your ASIN basket on the free 1 GB
One endpoint, HTTP or SOCKS5, no SDK, no card for the first GB. Point Scrapy at the endpoint on your dashboard, run a basket you know is in stock, and diff the null-price rate against your current pool.
Get 1 GB free