A lot of the discussion around the /crawl endpoint seems to miss a key detail in the docs. The crawler explicitly identifies itself as a bot, respects robots.txt, and does not bypass CAPTCHAs, WAF rules, or Cloudflare Bot Management.
So technically it’s a nice managed crawling system, but in practice it only works on sites that already allow bots to crawl them. For many real-world data extraction use cases, the problem isn’t crawling infrastructure, it’s dealing with sites that actively block bots. In those cases you still need traditional scraping approaches.
Interesting take on it. Some people probably wouldn't like to be called soft but there is likely some truth to it.
I feel it really comes down to priorities.
Scraping has always been a means to a end for most companies. Get data and then use it for something valuable. Before getting the data was easy, but now it is getting increasingly harder.
I think the key here is highlighting the fact that the time of cheap/easy/low skilled access to web data is ending. Companies either need to skill up on understanding how to bypass anti-bots or pay someone else to do it for them and they focus on the data.
One of the main ideas, we explored here is how scraping has shifted from being mainly a technical challenge to an economic one:
- Infrastructure and proxies have gotten cheaper, but anti-bot defenses have evolved fast.
- Because of that, the real cost of scraping is now the cost per successful result, and spikes of 5x–20x can happen when defenses tighten.
- The bottleneck today isn’t just “can you scrape it?”, it’s whether you can do it profitably and efficiently.
I’d love to hear how folks here are dealing with rising scraping costs or what strategies have worked when data value doesn’t obviously outweigh defense costs.
So technically it’s a nice managed crawling system, but in practice it only works on sites that already allow bots to crawl them. For many real-world data extraction use cases, the problem isn’t crawling infrastructure, it’s dealing with sites that actively block bots. In those cases you still need traditional scraping approaches.