With the custom headers, you can actually trick a lot of sites with bot protection to let you load their sites (even big sites like youtube, which I have found success in)
I am quite aware, but I actually built most of the scraping logic a long time ago, before I even knew that playwright was a thing.
I am looking to refactor a lot of this, and switching over to playwright is a high priority, using something like camoufox for scraping, instead of just chromium.
Most of my work on this the past month has been simple additions that are nice to haves
Currently working on a PR to swap over