The Company Quietly Funneling Paywalled Articles to AI Developers(theatlantic.com)
theatlantic.com
The Company Quietly Funneling Paywalled Articles to AI Developers
https://www.theatlantic.com/technology/2025/11/common-crawl-ai-training-data/684567/
17 comments
I will add, the news websites didn't start paywalling just because of LLM scrapers. They started doing it after certain parts of the GDPR passed, because they could no longer sustain themselves as much from targeted advertising and data sales. News has supplemented itself with advertising for a long time, but targeted advertising has sadly become perceived as mandatory by advertising companies, even before Google's dominance.
So publishers are in a no-win situation: if they lock down their content completely (server-side paywall), they disappear from Google Search and lose traffic. If they keep the "leaky" paywall, their content gets hoovered up for free by Common Crawl to train models that will then compete directly against them. They're trapped
So basically Common Crawl is a data laundromat for Big Tech. They outsource their dirty and ethically questionable data collection to a "non-profit," and then act like they're just "researchers" using an "open" dataset. Those "donations" from OpenAI and Anthropic are just payment for plausible deniability
The Nonprofit Quietly Funneling Paywalled Articles to HN Readers
Except those who have to solve reCaptcha and leave it instantly.
Meanwhile, Common Crawl’s executive director, Rich Skrenta, has publicly made the case that AI models should be able to access anything on the internet. “The robots are people too,” he told me, and should therefore be allowed to “read the books” for free.
The shamelessness of the propaganda reaches new heights. The industry shills no longer even attempt to make arguments, they just rely on people repeating their slogans.
The shamelessness of the propaganda reaches new heights. The industry shills no longer even attempt to make arguments, they just rely on people repeating their slogans.
It's just an economic argument. Paying for content licenses would be a massive operational expense. It's much cheaper to hire a couple of lawyers to invent a pseudo-philosophical justification for why you shouldn't have to pay.
>“You shouldn’t have put your content on the internet if you didn’t want it to be on the internet,”
This is absolutely the correct and original take. This modern corporate bending over backwards to try to appease the lawyers and pretend the web isn't public is the new and weird take. Seriously, if it's not supposed to be public then don't put it in public.
When I send a HTTP request to a webserver on the public internet it is up to them to decide if they want to respond to that request. And it is 100% up to me what I do with the data in that request on my machine in private.
>Common Crawl doesn’t log in to the websites it scrapes, but its scraper is immune to some of the paywall mechanisms used by news publishers. For example, on many news websites, you can briefly see the full text of any article before your web browser executes the paywall code that checks whether you’re a subscriber and hides the content if you’re not.
This weasly idea above, that corporations get to decide how you display the HTML, is very, very dangerous to our society. It's as if visiting a website and downloading the publicly available contents is a nation setting up an embassy of "foreign soil" on your hardware that they control and you don't.
Their cultural expectation is that you cannot do what you want with that data. Modifying it or how it's displayed is, to them, is like walking into their business location and moving around the displays. So obviously the only legal interface is the one they provide "at their location" or via another incorporated entity they associate with. But of course they aren't at their location they're at my location on my property in my PC. But slowly this commercial norm is working it's way into leglistation to become our reality as web attestation.
What they see, and what they want, is a situation equal to you going to their business premise and sitting down at one of their machines. They want to own your computer in just the same way simply by you visiting a website. That shit's fucked.
I'll turn off CSS and JS if want to and read the text if I want to on my computer in my RAM. If you don't want me doing that don't respond to the HTTP request. And stop trying to characterize all interactions on the web as between corporations. There are more of us human people than corporate people. Our use cases matter.
Alex Reisner and The Atlantic should be ashamed of themselves. They obviously don't know what they're talking about and are just repeating a corporate PR line, or, at best, intentionally trying to create controversy out of nothing.
This is absolutely the correct and original take. This modern corporate bending over backwards to try to appease the lawyers and pretend the web isn't public is the new and weird take. Seriously, if it's not supposed to be public then don't put it in public.
When I send a HTTP request to a webserver on the public internet it is up to them to decide if they want to respond to that request. And it is 100% up to me what I do with the data in that request on my machine in private.
>Common Crawl doesn’t log in to the websites it scrapes, but its scraper is immune to some of the paywall mechanisms used by news publishers. For example, on many news websites, you can briefly see the full text of any article before your web browser executes the paywall code that checks whether you’re a subscriber and hides the content if you’re not.
This weasly idea above, that corporations get to decide how you display the HTML, is very, very dangerous to our society. It's as if visiting a website and downloading the publicly available contents is a nation setting up an embassy of "foreign soil" on your hardware that they control and you don't.
Their cultural expectation is that you cannot do what you want with that data. Modifying it or how it's displayed is, to them, is like walking into their business location and moving around the displays. So obviously the only legal interface is the one they provide "at their location" or via another incorporated entity they associate with. But of course they aren't at their location they're at my location on my property in my PC. But slowly this commercial norm is working it's way into leglistation to become our reality as web attestation.
What they see, and what they want, is a situation equal to you going to their business premise and sitting down at one of their machines. They want to own your computer in just the same way simply by you visiting a website. That shit's fucked.
I'll turn off CSS and JS if want to and read the text if I want to on my computer in my RAM. If you don't want me doing that don't respond to the HTTP request. And stop trying to characterize all interactions on the web as between corporations. There are more of us human people than corporate people. Our use cases matter.
Alex Reisner and The Atlantic should be ashamed of themselves. They obviously don't know what they're talking about and are just repeating a corporate PR line, or, at best, intentionally trying to create controversy out of nothing.
> When I send a HTTP request to a webserver on the public internet it is up to them to decide if they want to respond to that request. And it is 100% up to me what I do with the data in that request on my machine in private.
Sorry, no. That's a software engineer's fantasy. And it's not even applicable to this case: it's not like Common Crawl is is only uses the data on their own machine in private, they're distributing it. Copyright is a real thing, that grants real rights, whether you like that or not. It doesn't disappear because computers and you don't like it.
Sorry, no. That's a software engineer's fantasy. And it's not even applicable to this case: it's not like Common Crawl is is only uses the data on their own machine in private, they're distributing it. Copyright is a real thing, that grants real rights, whether you like that or not. It doesn't disappear because computers and you don't like it.
Publishing for money works by spreading the development costs over N customers. When N is large, you can spread the costs very thinly. That means better development and/or lower costs for each customer.
Your vision that each recipient has the right to break this compact just undermines the model. I really like that I can spend $20 at the theater to watch a movie that cost $100m to develop. And while I understand your point about HTTP, I would rather live in a world where costs can be easily shared fairly and efficiently. Yeah, some publishers will gouge me, but the market sorts that out.
Your vision that each recipient has the right to break this compact just undermines the model. I really like that I can spend $20 at the theater to watch a movie that cost $100m to develop. And while I understand your point about HTTP, I would rather live in a world where costs can be easily shared fairly and efficiently. Yeah, some publishers will gouge me, but the market sorts that out.
[deleted]
Your PC and phone are on the internet, should I be able to access them?
They aren't publishing content, are they?
It's an information system with access controls.
We already have laws against breaking into systems bypassing authorization systems. If they applied they'd be applied here. But they don't. Because in this situation we're talking about public websites and corporations being made you don't run their javascript.
As an aside, I do host a website from my desktop computer at home and you are free to access it! It's public. I intentionally put it on the public internet. I do not use a smart phone or the internet via phone.
As an aside, I do host a website from my desktop computer at home and you are free to access it! It's public. I intentionally put it on the public internet. I do not use a smart phone or the internet via phone.
I don't think the LLM companies are fully innocent here, to be fair.