What’s the right solution? We were just trying to solve our own problem and it turned out interesting. If you know the not “nonsense” one that works please share.
Currently your prompts are processed by our server hosted at www.gopeek.ai. This was meant for velocity and early iteration while we're getting data models right.
We're already working on whitelists and blacklists for files, topics, memory exports, and can even work on self-hosted/locally hosted versions so please let us know what is a non-starter on this front.
Sensitive data is already wiped and never stored such as keys, tokens, and PII.
L8 might be apt, but honestly human brains are far more fluid and dynamic than human limb length. You can train your brain to do remarkable things it couldn't do before remarkably fast. You just can't train your limbs to be a foot longer.
Aka, I'd argue that elon musk drive gives you a greater chance to become an L8 engineer, where as elon musk drive may make you an NBA player, but at a significantly lower chance.
1) Public websites don’t require any more permission than taking photos of a public storefront. We abide by privacy laws and make sure we don’t overload website servers.
2) We aren’t complaining. We’re curious how others view this topic and space because it’s a contentious topic. We recognize that we might be able to address the larger issue of lack of compensation for websites being scraped by facilitating a win-win marketplace (only loser is proxy providers).
We built Harvest to reduce the pain of gathering web data by clicking through websites to copy data into excel sheets, databases, and CRMs. Something millions of people do everyday.
We recognize it as an unrewarding, tedious, and time-consuming thing humans have had to do until the latest abilities of browser agents.
As we built and learnt more about the industry we started to understand the underlying problems. For 99% of web sites web scraping isn’t the problem, the lack of compensation is.
We think there’s actually a better way to do this. If there’s enough demand, we can facilitate a rev share between agent scrapers and websites. Scrapers will pay less than what they pay for proxies and websites get a new revenue stream.
These are our thoughts at least so far. We aren’t ashamed of what we’ve built by any means in the way your comment implies lol. We want to see if we can benefit both parties in a win-win marketplace.
Yeah I mean the salad example is great because in that case it feels ridiculous to have to look at ads.
Maybe we don’t need to protect websites that rely on advertised content? Maybe it makes a better internet?
Based on what you’re saying here it sounds like good content will inevitably go behind paywalls in this case, and the number of paid subs people need to have will only continue to explode. Aka no free internet. Maybe that was bound to happen one way or another.
And all LLM providers will pay their data providers directly by API or something?
original internet (DARPA) was meant for communicating research and science. Maybe we return to that being the primary thing that will continue to be free?