the most annoying thing about new laptops is how difficult is to find replacement batteries that can be trusted and work well. The battery situation is a downgrade compared to the previous pluggable ones.
in the perfect world we would have a very high trust internet in which everyone follows the rules, checks robots.txt, rate limits, etc. But we don't live in such a world. Getting angry at these ai bots is useless. People should start considering what and how they host their data. If you're worried about bandwidth costs you have many alternatives to host your data for free or at a very little cost. i.e: github/gitlab/cloudflare.
If you're worried about your data getting scraped and used then maybe you can consider putting it behind a login or do some proof of work/soft captcha. Yeah, this isn't perfect but it will keep most dumb bots away.
Some people are hosting their sites like we're still in 1995 and times have changed.
that's a good point, in search we have google as a monopoly and since a big percentage of sites only want to be crawled by them it reinforces the monopoly. So a lot of people complain about bots not following robots.txt but if you follow them to the letter it's impossible to make anything useful. Also AFAIK robots.txt doesn't have any legal standing
in Argentina some people call facebook marketplace the "deep web", not sure if it has that reputation in other countries. Lots of stolen things being sold and also scams
I'm using chrome on linux and noticed that this year cloudflare is very agressive in showing the "Verify you are a human" box. Now a lot of sites that use cloudflare show it and once you solve the challenge it shows it again after 30 minutes!
What are you protecting cloudflare?
Also they show those captchas when going to robots.txt... unbelievable.
After using the AI chatbots for some time, I think that they are not so useful for non programmers other than for doing small tools, that may be difficult to modify and polish by a non programmer.
But they still fail and have subtle errors too often so they are more useful for programmers which already know what the AI is doing and can spot mistakes.
fully agree here, using a browser for everything is the dumb way. You just usually use it to circumvent the blocking and then reuse the cookies to call the endpoints directly.
thanks, it doesn't seem like it's the direction it's going at the moment. If you look at the robots.txt of many websites, they are actually banning AI bots from crawling the site. To me it seems more likely that each site will have its own AI agent to perform operations but controlled by the site.
what do you think would be the use cases for this project? being lightweight is awesome but usually you need a real browser for most use cases. Testing sites and scraping for example. It may work for some scraping use cases but I think that if the site uses any kind of bot blocking this is not going to cut it.
I wanted to try a diffent approach which is to use chromium's network stack directly instead of patching curl to impersonate it. In this case you're using the real thing so it's a bit easier to maintain when there are changes in the fingerprint.
They use different SSL libraries/configuration. Chrome uses BoringSSL and other libraries may use OpenSSL or some other library.
Besides that the SSL library may be configured with different cipher suites and extensions. The solution these impersonators provide is to use the same SSL library and configuration as a real browser.