I've been playing around with most of the browser using agents and noticed that the cost of the residential proxies used by the agent for stealth is often as high as the token cost.
Residential proxies were designed for massively parallel and short-lived scraping tasks (load a web page, scrape its content and close the session, repeat for each of the 10k pages you want to scrape and for every single day).
However, most browser agents relying on those proxy services have a completely different access pattern. They need long-lived and sticky sessions that might last several minutes (and which might load hundreds of MB of content along the way).
This left me wondering whether residential proxies (and their per GB pricing) is actually the best infrastructure to build browser using agents today. I guess once agents effectively become first class citizens of the web the need for stealth infrastructure will not be as acute. In the meantime, does anybody know of a better alternative?
Residential proxies were designed for massively parallel and short-lived scraping tasks (load a web page, scrape its content and close the session, repeat for each of the 10k pages you want to scrape and for every single day).
However, most browser agents relying on those proxy services have a completely different access pattern. They need long-lived and sticky sessions that might last several minutes (and which might load hundreds of MB of content along the way).
This left me wondering whether residential proxies (and their per GB pricing) is actually the best infrastructure to build browser using agents today. I guess once agents effectively become first class citizens of the web the need for stealth infrastructure will not be as acute. In the meantime, does anybody know of a better alternative?