First off, I want to thank you and the other members of the CC Foundation, the CC data set is an incredible resource to everyone.
Much of the UA data, including CCBot, is from an upstream source[0]. I was torn on whether CCBot and other archival bots should be included in the configs, since these services are not AI bot scraping services. I've added an exclusion for CCBot[1] and the archival services from the recommended configs.
For those not using cloudflare but who have access to web server config files and want to block AI bots, I put together a set of prebuilt configs[0] (for Apache, Nginx, Lighttpd, and Caddy) that will block most AI bots from scraping contents. The configs are built on top of public data sources[1] with various adjustments.
Now that's the challenging part, especially since Mozilla needs to fund browser development. The initial differentiation for a Mozilla search would have been difficult, but at their peak they had 30%+ market share, and if the default search on Firefox was Mozilla Search then they might have been able to make it all work financially. DuckDuckGo makes over $100 million in revenue per year[1], if a Mozilla Search made as much money annually, then even though it's below their current $500 million search contract with Google, with some fiscal responsibility they probably would have had enough to support a search engine and browser development concurrently.
>The biggest issue is that an ancient version of Firefox is the only viable option for a web browser.
There was some work done on porting Palemoon, Otter Browser, and QtWebEngine to ArcaOS. A preview version of a QtWebEngine-based browser[0] was available for download and may work with newer websites (given QtWebEngine is based on Chromium), but I have not tested it myself.
Much of the UA data, including CCBot, is from an upstream source[0]. I was torn on whether CCBot and other archival bots should be included in the configs, since these services are not AI bot scraping services. I've added an exclusion for CCBot[1] and the archival services from the recommended configs.
[0] https://darkvisitors.com/agents/ccbot
[1] https://github.com/anthmn/ai-bot-blocker/commit/ae0c2c40fd08...