To clarify the math. Netlify bills $50 for each 100GB over the Pro plan limit of 1TB. Which is the barrel I'm looking down just this month before others get the same idea. So yes, I'm squeezed on both side unless I put the work in to rehost.
I created and maintain ProtonDB, a popular Linux gaming resource. I don't do ads, just pay the bills from some Patreon donations.
It's a statically generated React site I deploy on Netlify. About ten days ago I started incurring 30GB of data per day from user agents indicating they're using Prerender. At this pace almost all of that will push me past the 1TB allotted for my plan, so I'm looking at an extra ~$500USD a month for the extra bandwdith boosters.
I'm gonna try the robots.txt options, but I'm doubtful this will be effective in the long run. Many other options aren't available if I want to continue using a SaaS like Netlify.
My initial thoughts are to either move to Cloudflare Pages/Workers where bandwidth is unlimited, or make an edge function that parses the user agent and hope it's effective enough. That'd be about $60 in edge function invocations.
I've got so many better things to do than play whack-a-mole on user agents and, when failing, pay this scraping ransom.
Can I just say fuck all y'all AI harvesters? This is a popular free service that helps get people off of their Microsoft dependency and live their lives on a libre operating system. You wanna leech on that? Fine, download the data dumps I already offer on an ODbL license instead of making me wonder why I fucking bother.
I felt a similar reaction to the tone of the article. Twice the author said 'You may as well not have bothered' if no one recognized the value of your output. And your response also frames it as 'What you've done has no value outside of yourself'.
I think what the post you're replying to is getting at is that by framing things this way we overvalue things which others approve of, and undervalue things we enjoy doing for the sake of doing.