The Chrome extension approach may shift some (most?) of the risk to the end user, since technically they are now the one scraping. Theoretically getdex would be relatively better off in this arrangement, while putting their customers into a legal gray area.
Currently, yes. However it's quite early and I'm exploring whether this is the best option.
The reason I was posting here on HN is to see if anyone more technical may be interested in this as well. E.g. to leverage an API like this with AI agents or otherwise.
When building a SaaS with a Go backend, it's nice to be able to have the option of the agents and workflows being in the same process. And being confident in the ability of that to scale well.
While it's true that Go lacks good ML libraries, for some this isn't too consequential if your app is primarily using Anthropic or OpenAI and a database that offers semantic or hybrid search for RAG. The ML is done elsewhere. Plus it could be that you can leverage MCP servers and at that point you're language agnostic.
Regarding the concurrency model approach with Go and agents, I initially baked a message based approach (a la the Actor model, with one goroutine per agent) into Dive Agents, but eventually found that this would be better implemented as another layer. So currently in Dive it's the user's choice on how to implement concurrency and whether to use messaging. But I anticipate building that back in as an optional layer.
Risor is an embedded scripting library and CLI for Go. This release adds new modules for web crawling, background scheduling, and more. Happy scripting.
Just a quick release announcement for Risor, an embedded scripting library and CLI for Go. Plenty of additions relating to web crawling, background scheduling, and more. Happy scripting.
I'm definitely with you that sites should be leveraging CDNs and similar. But I get that many don't want to do any work to support bots that they don't want to exist in the first place.
To me it seems like the companies actually doing the crawling have an incentive to leverage centralized caching. It makes their own crawling faster (since hitting the cache is much faster than using Playwright etc to load the page) and it reduces the impact on all these sites. Which would then also decrease the impact of this whole bot situation overall.
An aspect I find interesting is that these crawlers are all doing highly redundant work. As in, thousands of crawlers are running around the world, and each crawler may visit the same site and pages multiple times a week.
This seems like an opportunity for a company like Firecrawl, ScrapingBee, etc to offer built-in caching with TTLs so that redundant requests can hit the cache and not contribute to load on the actual site.
Even if each company that operates a crawler cached pages across multiple runs, I'd expect a large improvement in the situation.
For more dynamic pages, this obviously doesn't help. But a lot of the web's content is more static and is being crawled thousands of times.
I built something for my own company that crawls using Playwright and caches in S3/Postgres with a TTL for this purpose.
Does this make sense to anyone else? I'm not sure if I'm missing something that makes this harder than it seems on the surface. (Actual question!)