Web Scraping Proxy for Public Data

Crawling public pages at any volume needs more than one exit. The question is how to spread requests so success stays flat as the job grows.

How Blocking Actually Arrives

Rarely as a 403. The usual sequence is slower responses, then subtly altered markup — a missing price, a truncated listing, a placeholder where the data was — and only then a hard block. Pipelines that alert on HTTP status miss the middle stage entirely and quietly collect wrong data for hours. Watch for changes in response shape, not just in status codes.

Rotation That Does Not Break Pagination

Rotate per session rather than per request. A crawl that walks pages two through nine needs the same exit for all of them or the site loses the thread and returns inconsistent results. Set the exit country where the page is localised, since prices and availability commonly differ by region. Back off at the first sign of altered markup rather than waiting for an error, and resume on a different address.

Tuning For Cost

Block images, fonts and analytics unless you need them; on a metered plan that is usually the difference between comfortable and expensive. Cache anything requested more than once. Track gigabytes per successfully parsed page — when that number starts climbing, a target has usually begun serving something different rather than your code getting worse.

Respecting Limits On Purpose

Beyond the legal considerations, aggressive crawling is self-defeating: it shortens the life of every exit you own and trains the target to filter harder. Read the robots file, keep request rates in a range the site can absorb, and slow down when response times climb rather than when errors start. Pipelines run this way stay productive for months; the ones that push until something breaks spend most of their time being rebuilt.