How do you differentiate background and worker processes? I’m currently running VMs for background processes processing pub sub topics. Is that not something that could go on cloud run?
In a previous company we were scraping customers web pages to load customer and product information because the number of systems and teams involved to connect to their CDP would take months to get up and running. A few problems we had to solve was the web pages changing underneath us and some pages not being formatted the same. We ended up having to detect those changes server-side to alert us to update the scraper. How do you all deal with these use-cases?
Also, we tried to leverage: https://www.diffbot.com/ but the lack-of-accuracy/lack-of-complete-data + cost never justified it's usage.