Nowadays, somebody can just ask claude to build them a scraper/bot that hooks into a proxy network and all of a sudden they can easily send 20k+ reqs/min from hundreds or thousands of IPs cycling them as they get rate limited or banned. In my work, the scrapers have gotten way more aggressive in the last 2 years or so. Frankly, I'm happy there is a solution.
There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.
No tracking. No cookies. No behavioral targeting (targeting based on stuff you've previously done). Every website where our ads appear AND every advertiser is hand approved. No JS from advertisers: just a plain JPG/PNG and text.
We're small but on track to pay out $500k to publishers this year.
They absolutely do. Every sponsorship you see on a podcast or a youtube video or a streamer is a contextual ad. Many open source sponsorships are actually a form of marketing. You could argue that search ads are pretty contextual although there's more at work there. Every ad in a physical magazine is a contextual ad. Physical billboards take into account a lot of geographical context: the ads you see driving in LA are very different than the ones you see in the Bay Area. Ads on platforms like Amazon, HomeDepot, etc. are highly contextual and based on search terms.
Founder of EthicalAds here. In my view, this is only partially true and publishers (sites that show ads) have choices here but their power is dispersed. Advertisers will run advertising as long as it works and they will pay an amount commensurate with how well it works. If a publisher chooses to run ads without tracking, whether that's a network like ours or just buyout-the-site-this-month sponsorships, they have options as long as their audience generates value for advertisers.
That said, we 100% don't land some advertisers when they learn they can't run 3rd party tracking or even 3rd party verification.
My employer, Read the Docs, has a blog on the subject (https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...) of how we got pounded by these bots to the tune of thousands of dollars. To be fair though, the AI company that hit us the hardest did end up compensating us for our bandwidth bill.
We've done a few things since then:
- We already had very generous rate limiting rules by IP (~4 hits/second sustained) but some of the crawlers used thousands of IPs. Cloudflare has a list that they update of AI crawler bots (https://developers.cloudflare.com/bots/additional-configurat...). We're using this list to block these bots and any new bots that get added to the list.
- We have more aggressive rate limiting rules by ASN on common hosting providers (eg. AWS, GCP, Azure) which also hits a lot of these bots.
- We are considering using the AI crawler list to rate limit by user agent in addition to rate limiting by IP. This will allow well behaved AI crawlers while blocking the badly behaved ones. We aren't against the crawlers generally.
- We now have alert rules that alert us when we get a certain amount of traffic (~50k uncached reqs/min sustained). This is basically always some new bot cranked to the max and usually an AI crawler. We get this ~monthly or so and we just ban them.
Auto-scaling made our infra good enough where we don't even notice big traffic spikes. However, the downside of that is that the AI crawlers were hammering us without causing anything noticeable. Being smart with rate limiting helps a lot.
I'm not the poster you're responding to but I'm one of the founding team of EthicalAds. We're a small team, focused exclusively on marketing to devs, and really trying to show high-quality ads without tracking people (ads are contextually targeted).
You can get a feel for what you'll earn here[1]. Basically you earn 70% of the gross of what we charge advertisers (see advertiser pricing[2]). Keep in mind these are ad views which aren't quite the same as pageviews. They're a subset.
Ads are a straight-forward path to monetization but not always the best. If you can make a project work as SaaS or really make sponsorships work for you (this requires effort), those will definitely earn A LOT more money per pageview than ads. Ads require a lot of traffic to make them work well. Usually you want high tens to hundreds of thousands of pageviews per month.
From a what we look at for publishers (sites that show ads) perspective, we're usually looking for high quality dev-focused sites or projects that don't want to just show Google ads. Per ad, publishers will earn much more with us than Google display ads but if you want to stick 4-5 Google ads on your site, video ads, or the like we can't compete with that and we don't want our ads on sites that do that. Devs hate them. My email is in my bio if you want to discuss further.
My employer, Read the Docs is a heavy user of Cloudflare. It's actually hard to imagine serving as much traffic as we do as cheaply as we can without them.
That said, for publicly hosted open source documentation, we turn down the security settings almost all the way. Security level is set to "essentially off" (that's the actual setting name), no browser integrity check, TOR friendly (onion routing on), etc. We still have rate limits in place but they're pretty generous (~4 req/s sustained). For sites that don't require a login and don't accept inbound leads or something like that, that's probably around the right level. Our domains where doc authors manage their docs have higher security settings.
That said, being too generous can get you into trouble so I understand why people crank up the settings and just block some legitimate traffic. See our past post where AI scrapers scraped almost 100TB (https://news.ycombinator.com/item?id=41072549).
There's a few ways first party cookies can track you. Probably the biggest single way is Google Analytics which by default uses only first party cookies. Even without cookies at all, GA could track you across the web although first party cookies do make this a little easier and "better". However, first party cookies can help trackers in other ways like for CNAME cloaking[1] which basically makes a first-party cookie function similarly to a third-party one.
Disclosure: I work for a small privacy focused ad company.
I edited to make my comment more clear but polyfill.io sends dynamic polyfills based on what features the identified browser needs. Since it changes, the SRI hash would need to change so that part won't work.
SRI generally won't work here because the served polyfill JS (and therefore the SRI hash) depends on the user agent/headers sent by the user's browser. If the browser says it's ancient, the resulting polyfill will fill in a bunch of missing JS modules and be a lot of JS. If the browser identifies as modern, it should return nothing at all.
Edit: In summary, SRI won't work with a dynamic polyfill which is part of the point of polyfill.io. You could serve a static polyfill but that defeats some of the advantages of this service. With that said, this whole thread is about what can happen with untrusted third parties so...
Keychain Access has not been "fine". It's had multiple unaddressed data loss bugs. For example, Keychain lost all passwords from all Keychains after the Catalina update[1] and this wasn't fixed in the next 3 Catalina minor updates. Multiple users reported the issue to Apple and the response was crickets. Even if you restored the passwords, it helpfully deleted them all again. I switched to 1Password and declared Keychain Access a lost cause. I don't think I'll be giving them a second chance here.
I can't confirm if the price is the same or not, but their terms[1] specifically mention that the price can be different.
> Prices for Offerings you order for delivery or pickup through the Online Grocery Ordering Service may be higher than the prices for such Offerings in our physical stores.
Most outlets I've seen (eg. Target) are the same in that they just list a higher price on the website than it costs in-store and they're upfront about this. It takes me 20-25 minutes of in-store picking to shop including checkout for my weekly groceries. Even if that is done by a minimum wage worker (~$17/hr here) that's ~$6-8 of service on top of them bringing it out to the curb. In addition, eggs are usually specially packed in their own bag (frequently with a sticker on the bag labeled "eggs") when they're bought online and curbsided. It seems a bit naive to me to think that all this service would just be included/free.
While I agree with you to an extent, this is not a very good way to check actual in-store prices. Most grocery stores charge a different rate for products that are delivered or even curbsided than they do if you go to the store and buy it yourself. This is true even if you go directly vs. going through an intermediary like DoorDash.
I live in coastal CA and the cheapest eggs at my local market are $3.49/dozen. Trader Joes and Costco are closer to $2.20/dozen if you just want plain white eggs. The moment you go organic, the cheapest is Costco at about $4/dozen.
Edit: Just to be clear, at Costco you buy 2 dozen rather than 1 and I've divided the price to a single dozen.
Ya, other types of retargeting like this are also likely. The jump from visiting a website to an advertiser does physical mailings isn't a big one (political advertising uses this a lot). Long story short, she was probably retargeted based on her actions and probably not based on the insurer or provider doing anything illegal.
Edit: I don't want to sound like I'm blaming the victim here. That's not my intent. I just don't think blaming the insurer or provider is fair either. I dump the blame on the data broker/ad network and to a far lesser extent the advertiser.
Healthcare providers and insurers in the US are bound by HIPAA privacy rules, but data brokers (mentioned in the article) and the ads industry generally are not. For example, if she used an app in the doctor's waiting room that shared/sold location data to a data broker, they can use her location data for retargeting purposes. There have been many cases in the past where advertisers targeted users based on visiting medical or other sensitive locations.
As to how they mailed it to her and got her home address, a data broker who has location data can fairly easily determine a user's home address from that data. Many brokers and networks may also already have an association between a "pseudo-anonymous advertising ID" and real user with name and address. Not saying that location-based retargeting happened this time as the article doesn't give us enough to go off of and other types of retargeting are another possibility.
Overall, I think it's unlikely that the provider or insurer shared her data and other alternatives are more likely.
Disclosure: I work in the ads industry but on contextual targeting only. Some location-based retargeting is terrifying and will probably eventually be criminal. It's a bit of the wild west right now.
Zillow's estimates are definitely questionable and I don't want to be seen as defending their precision. However, unless you're referring to an actual house addition or something like earthquake retrofitting, most capital improvements are rounding errors to the value of a house in California. Land alone (vs. "improvements") for a house purchased recently is ~70% of the value and you can see this broken out on a property tax statement. Replacing knob and tube wiring or putting in AC is low tens of thousands of dollars at most and frequently those minor things aren't even reported to the assessor. Sure, an old house that hasn't been updated isn't going to sell for as much as one that has, but unless it's a major change, it isn't going to be hundreds of thousands in difference.
Because I can't query Zillow's data, I don't know any better way to do it than find a neighborhood where most of the houses are old and just see what their property taxes are one by one. In my neighborhood which was built in the 1910s and 1920s, there's quite a few examples of people paying sub $1k in property taxes per year. It isn't a majority or anywhere close to it but within a couple blocks of my house I found nearly 10 examples. I even found a 4-unit multifamily paying just under $1k total.
If you want a pure anecdote, a friend in a nearby house just had the knob and tube in their house replaced with modern electrical (~$35k). The electrician found gas lines in the 1st floor ceiling for indoor gas lighting. Although the lighting fixtures were gone, the gas lines were still hooked up to the main house gas line.
This isn't as crazy an example as you think. People and companies who bought properties before prop tax increases were capped in 1978 are paying almost nothing in property taxes.
I live in California but not in the Bay Area in a house Zillow estimates at $1.2M. I have a neighbor in an admittedly smaller house that Zillow estimates at $740k. In 2022, they paid $482 dollars in property taxes on an assessed value of $34.5k. Whenever they sell, the new owner will pay 20x in taxes. Sites like Zillow show all of this and I can find multiple examples within a couple blocks of my house.
I work on ads, but not for Google and FWIW, I've only been able to reproduce a few of these malvertising reports. However, I wouldn't be surprised if there were additional targeting parameters on these campaigns. Rather than targeting just anybody searching for VLC, Blender, or Audacity, these malvertisers want to target folks more likely to click a "download now" malvertisement. Maybe only target older users, non-developers, Windows users only, or a number of other facets that probably have a higher rate of installing malware. I have no knowledge if these folks are doing this, but that's what I'd do if I were a scummy advertiser shilling malware. If they can avoid wasting their ad budget on sophisticated users, I'm sure they will.
There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.