A block page is not training data.
Challenge interstitials arrive as well-formed HTML, so they parse, chunk and embed like any real document.
- /tech/2026/jul/28/chip-supply
- /tech/2026/jul/27/grid-storage
- /tech/2026/jul/27/rail-signalling
- /tech/2026/jul/26/port-automation
- /tech/2026/jul/26/water-sensors
- /tech/2026/jul/25/fibre-rollout
- /tech/2026/jul/25/battery-recycling
- /tech/2026/jul/24/satellite-imagery
3 of 8 “documents” are the challenge page. Every one is well-formed HTML, so the extractor chunks it and the embedder stores it.
- /tech/2026/jul/28/chip-supply
- /tech/2026/jul/27/grid-storage
- /tech/2026/jul/27/rail-signalling
- /tech/2026/jul/26/port-automation
- /tech/2026/jul/26/water-sensors
- /tech/2026/jul/25/fibre-rollout
- /tech/2026/jul/25/battery-recycling
- /tech/2026/jul/24/satellite-imagery
8 of 8 parsed as documents. Nothing to quarantine, nothing to re-crawl.
Sample crawl output, illustrative. The interstitial text is the real one, and it is well-formed HTML.
The block page comes back out of the retriever
Your fetcher says 200, so the block page is parsed, chunked and embedded like any other document, and later it is retrieved as context for an answer. Near-duplicate detection does not reliably catch them: when a block page carries a unique incident id, it can survive MinHash as its own record. And you pay for all of it.
A block page arrives as HTTP 200→Stumbles are re-run, not returned
Managed challenge and access-denied bodies can arrive as 200 text/html, with a content-type your fetcher accepts and a body your extractor will return prose from. Nothing in the transport marks it.
An attempt that fails at the transport — reset, TLS error, block status, known challenge response — is re-run through another exit before your code sees it, and absorbed rather than billed.
The exit is drawn blind→Exits with memory of that host
A rotating pool hands you a random exit with no memory of the host. Hosting ranges can carry blanket rules, and velocity can be scored across a whole subnet rather than per address, so your worker fanout is itself the signature.
Per-target scoring picks an exit with a record of landing on that specific host. The memory sits at the edge, so a hundred parallel fetchers stop rediscovering the same burned addresses.
Rotation drops the session mid-render→One chain, one machine, one exit
Session cookies are bound to the address that earned them. Per-connection rotation scatters one render across dozens of exits, so every subresource arrives as a new visitor.
Ask the operator to put your account in sticky mode and every connection from one source machine leaves through the same exit for the window — 10 minutes by default. Two pinned chains means two machines.
You are billed for discovery→You pay for bytes you keep
Every blocked attempt comes back at full page weight, and then every retry does too. Discovery cost scales with how badly the pool is doing, not with how much text you kept.
Absorbed attempts never hit the invoice, so the billed axis is content. Metered plans bill a flat rate per gigabyte, and traffic never expires between training runs.
Before you point a crawler at it.
At corpus scale a per-GB rate is not survivable. We pull terabytes.
Correct, and nobody should push an entire pretraining crawl through a paid edge. Run your own fetchers on the open long tail and route only the hosts that defeat them through here. Metered plans bill a flat rate per gigabyte, traffic never expires, absorbed attempts are not billed, and for real volume you can open a ticket for a quote.
Do you help us ignore robots.txt or a site's AI-crawler policy?
No. You set your user agent and run your own robots and rate logic; nothing here bypasses a site's stated crawler policy, and the acceptable use policy forbids crawling hard enough to trip anti-bot systems and generate abuse reports. Do not spoof another operator's crawler user agent either — the ones that matter publish a way for a site to check whether the request really came from them, and a spoof that fails that check is worse than an honest agent string. What the edge fixes is collateral damage to a polite crawler.
How do I know a 200 is real content and not a challenge page?
You do not get that from the edge. It acts on what the transport shows — connection failures, TLS errors, block status codes, known challenge responses — and a challenge dressed as a clean 200 is invisible to it. Keep your own validators in front of the sink: a minimum extracted-text length per host, an expected selector or JSON-LD @type, and a hash set of known block-page bodies.
Will one exit hold a full headless render, subresources and all?
Usually, yes: by default a machine's connections share an exit while they keep coming, so one render's subresources land together. Leave -rotate off the username for rendering; it gives every new connection a different exit. For a guarantee across a longer chain, ask for sticky mode — an account mode, not a per-request flag; there is no session-id parameter on the username. Every connection from one source machine then leaves through the same exit until it has been idle for 10 minutes (the default). One machine drives one pinned chain at a time.
We run thousands of concurrent fetchers. Which plan shape fits?
Metered volume, because the billing axis is bytes rather than connections. The unlimited-bandwidth plans price on concurrency rather than bytes and suit a steady narrow pipeline such as an agent browsing on request, not a wide crawl. Either way it is a plain proxy endpoint — HTTP or SOCKS5 — with no SDK.
One credential, every job.
All six →Point one shard of your frontier at it for a day
Take the shard of your frontier your own fetchers keep losing and run a day of it. Count usable documents per gigabyte, not fetch success. The first gigabyte is free.
Get 1 GB free