Ask HN: How do I stop companies from scraping my site?
61 comments
Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1]
What is not then permitted is to give other people copies, or publish them on your website, or pretend it's your own work etc...
When it comes to LLMs or image generation models, they don't keep any copies and they don't generate any copies either, so they consider themselves to be well in the clear. [2]
If you want to stop people scraping your stuff anyway, you can always use robots.txt , or put things up behind a login-wall.
Do consider the morality of what you are doing though. Personally I feel that published data should be scrape-able where practical.
[1] https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Googl.... (you're even allowed to do this with physical books)
[2] https://www.uspto.gov/sites/default/files/documents/OpenAI_R... (With apologies for my crude summary of their actual arguments)
What is not then permitted is to give other people copies, or publish them on your website, or pretend it's your own work etc...
When it comes to LLMs or image generation models, they don't keep any copies and they don't generate any copies either, so they consider themselves to be well in the clear. [2]
If you want to stop people scraping your stuff anyway, you can always use robots.txt , or put things up behind a login-wall.
Do consider the morality of what you are doing though. Personally I feel that published data should be scrape-able where practical.
[1] https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Googl.... (you're even allowed to do this with physical books)
[2] https://www.uspto.gov/sites/default/files/documents/OpenAI_R... (With apologies for my crude summary of their actual arguments)
Publishing a derivative work of something you scraped is not permitted. The question is whether LLMs are derivative works, which has not been settled in courts yet.
Technically correct on the point that it has not been settled in courts yet.
For eg stable diffusion I actually dove a bit deeper on this:
The LAION dataset is -in itself- almost indisputably legal: It contains URLs of images, and tags describing those images[1]. To be sure: _It does not contain actual images_. People have made this mistake before, and LAION quite correctly reply that they don't have them. [2]
Stable diffusion's models are much smaller [3] than the LAION dataset which they use as their source. Again, the models do not contain images.
This is pretty suggestive that SD models should not be considered derivative works. Of course sometimes funny things happen in court, so we can't be sure until a court actually decides.
I don't currently run my own GPT or LLAMA, but those should have similar properties. (in fact, OpenAI appears to claim so.)
[1] https://laion.ai/blog/laion-5b/ (if you look for the files in any of the variations, you'll see it's in the order of terrabytes)
[2] https://www.vice.com/en/article/pkapb7/a-photographer-tried-...
[3] https://huggingface.co/stabilityai/stable-diffusion-xl-base-... : see under "files and versions", it's sd_xl_base_1.0.safetensors. File size is 6.9 GB
For eg stable diffusion I actually dove a bit deeper on this:
The LAION dataset is -in itself- almost indisputably legal: It contains URLs of images, and tags describing those images[1]. To be sure: _It does not contain actual images_. People have made this mistake before, and LAION quite correctly reply that they don't have them. [2]
Stable diffusion's models are much smaller [3] than the LAION dataset which they use as their source. Again, the models do not contain images.
This is pretty suggestive that SD models should not be considered derivative works. Of course sometimes funny things happen in court, so we can't be sure until a court actually decides.
I don't currently run my own GPT or LLAMA, but those should have similar properties. (in fact, OpenAI appears to claim so.)
[1] https://laion.ai/blog/laion-5b/ (if you look for the files in any of the variations, you'll see it's in the order of terrabytes)
[2] https://www.vice.com/en/article/pkapb7/a-photographer-tried-...
[3] https://huggingface.co/stabilityai/stable-diffusion-xl-base-... : see under "files and versions", it's sd_xl_base_1.0.safetensors. File size is 6.9 GB
Thanks for the links! To me (merely an amateur observer), it is not perfectly obvious that the LAION dataset is legal, because I don't think "we don't have your original images anymore, we only kept the results of applying an algorithm to them" is a universal defense. If the algorithm is "resize to 640×480" you are probably still infringing, but if it's "calculate md5 checksum" you're probably not. I don't know where on the spectrum LLM training will fall, legally!
Heh, it does in fact calculate the MD5 checksum!
In total, the fields in the LAION dataset are:
Note that LAION is a metadata dataset, it's not an image generation model itself. (But it has been used to train one)
In total, the fields in the LAION dataset are:
image_path
caption
NSFW
similarity
LICENSE
url
key
status
error_message
width
height
original width
original height
exif
md5
Note that LAION is a metadata dataset, it's not an image generation model itself. (But it has been used to train one)
Not sure that the size of the model is relevant.
If you take a PNG of Mickey Mouse, and create a similar vector graphic that is a fraction of the size, that would still be derivative.
If you take a PNG of Mickey Mouse, and create a similar vector graphic that is a fraction of the size, that would still be derivative.
I do not trust a court to weigh in properly on the deep technical nuances involved.
These kinds of things are not settled by arbitrary judges making off the cuff decisions.
They are settled by specialist technical judges with advice of dozens of specialists explaining the tech in detail.
Check out Google vs Oracle the lawsuit about whether an API is copyrightable for some examples of how the rulings are very much based on deep understanding.
The only downside is the process is slow and needs both a plaintiff and defendant willing to spend hundreds of thousands or millions of dollars to settle the case in court. It also takes many months of time.
They are settled by specialist technical judges with advice of dozens of specialists explaining the tech in detail.
Check out Google vs Oracle the lawsuit about whether an API is copyrightable for some examples of how the rulings are very much based on deep understanding.
The only downside is the process is slow and needs both a plaintiff and defendant willing to spend hundreds of thousands or millions of dollars to settle the case in court. It also takes many months of time.
last I heard about this ruling was that APIs are copyrightable, which is terrible for open source reimplementations of all stuff
More trying to focus on how the ruling wasn't naive as OP implies all rulings are by saying "I wouldn't trust a judge".
Alternatively they could mean "judges should stay away from X" but that is nonsensical given the courts being arbitrage...
Alternatively they could mean "judges should stay away from X" but that is nonsensical given the courts being arbitrage...
we're talking about a law (copyright), who else can weight in besides the courts?
or are you suggesting abolishing copyright? because I'm all for it
or are you suggesting abolishing copyright? because I'm all for it
This is an interesting topic that comes up time and time again. I'd like to understand, if we abolish copyright and I make a computer game, pour all my money and time in it, what's to stop EvilBigPublisher Co from simply taking it, rebranding it and killing my company with their might?
Or, if I came across a copy of Unreal Engine, what would stop me from using it and not paying Epic a dime?
Or, if I came across a copy of Unreal Engine, what would stop me from using it and not paying Epic a dime?
1. Nothing stops them from doing the above now.
2. Unreal lawyers costing you more in legal fees than you can afford
2. Unreal lawyers costing you more in legal fees than you can afford
Seeing as you are knowledgeable on the subject, may be you know if there is a licence, like creative commons, that will allow all the usual intelectual property uses except in AI training?
Or may be an old licence that is so strict that it will protect from AI shenanigans by default?
Or may be an old licence that is so strict that it will protect from AI shenanigans by default?
I sell apples at the market. How can I prevent people from buying my apples just to make pies or tarts and pretending those products are theirs?
I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
I grew the apple, I should be able to decide what people do with it. I have a sign that says the apples are only for eating, but people are ignoring it.
I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business.
For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatGPT can customize it's response based on my unique situation whereas the Medium article is fixed, but it's still unfair to the author of the Medium article that ChatGPT was trained on, because without their effort, none of this would have been possible. And yet, because I am no longer generating as revenue for Medium, the original authors are no longer being compensated for the value they provided me.
I personally believe that LLM hosts should be required to pay for training data. With RAG, I think it makes sense to charge per-query, but for base models, content authors would probably need to organize into larger groups to facilitate payment.
For example, I don't read Medium articles as often as I used to, in part because I can just ask my programming questions directly to ChatGPT. Granted, this is to be expected because ChatGPT can customize it's response based on my unique situation whereas the Medium article is fixed, but it's still unfair to the author of the Medium article that ChatGPT was trained on, because without their effort, none of this would have been possible. And yet, because I am no longer generating as revenue for Medium, the original authors are no longer being compensated for the value they provided me.
I personally believe that LLM hosts should be required to pay for training data. With RAG, I think it makes sense to charge per-query, but for base models, content authors would probably need to organize into larger groups to facilitate payment.
> I think this is a poor analogy. The better comparison would be if people were buying my apples and extracting the seeds to plant their own trees, then selling their own apples at a lower price than me and putting me out of business
Thereby increasing the amount of apples available to the population, lowering prices, making them available to more people. A net gain for society as a whole.
This happen all the time, you benefit from it all the time. You only have a problem with it because for once this system is not working in your direct favour.
Thereby increasing the amount of apples available to the population, lowering prices, making them available to more people. A net gain for society as a whole.
This happen all the time, you benefit from it all the time. You only have a problem with it because for once this system is not working in your direct favour.
It might also have a positive effect on artists themselves. See Jevon's Paradox on increasing efficiency of production increases demand.
> A net gain for society as a whole.
Perhaps at first. A race to the bottom usually results in everyone being worse-off though — see red delicious apples.
Perhaps at first. A race to the bottom usually results in everyone being worse-off though — see red delicious apples.
It turns out people want pretty things, not 'good' things. It is very difficult to prevent this from occurring without other nasty side effects.
[deleted]
Your apple analogy is completely legal and ethical. The alternative is Monsanto's attempt at DRM with their "terminator" seeds, which many found to be unethical.
Sadly, this is not even clear cut, though of course this is a particularly egregious example.
For instance depending on jurisdiction it might be illegal to sell seeds that can make plants that can produce viable seeds, unless you have a specific authorisation and depending on the species.
For instance depending on jurisdiction it might be illegal to sell seeds that can make plants that can produce viable seeds, unless you have a specific authorisation and depending on the species.
Legality has nothing to do with ethical. I by force could make it a law that you have to pay me to eat, most reasonable people would consider that an affront to ethical behavior.
OP said both.
Also, you know that morality is relative, and laws basically represent polities' morals, right ?
Also, you know that morality is relative, and laws basically represent polities' morals, right ?
The analogy seems fitting. The point is that the content was meant for direct rather than indirect consumption. My conclusion however is different--all are valid uses. Apples are what was sold, not a license for specific use of apples. It would be different if the content was posted on your own site with specific terms of service that prohibited specific uses. I don't think Reddit for example had anything like that in place when the data was scraped.
I sell apples at the market. Someone took my apple for free and made a pie out of it. They said it was not theft, because my market stall was in the public.
I sold an apple with DRM. One person bought my apple and gave it to a Chinese hacker, whom stripped the DRM and duplicated my apple for everyone, for free. They said it was not theft.
I grew an apple and put it on a cloud provider's server. I sold access to many duplicated apples, until a technology update forced me to change providers. I found out I could not take back my apples from the cloud providet without paying exorbitant fees. They said it was not theft.
I made an encrypted apple, I sold it on an app store. The app store banned me and my apple, the virus software flagged my apple, the TPM chip stopped my apple from being played, the government cracked open the encryption and duplicated my apple for national security purposes. They said it was not theft.
The only way to sell apples, is through perfectly-black, 'black boxes'. Putting apples on computers, gets them stolen. End of story.
It's been forty years now. There's nothing to be 'unsure about'.
'Benefit of the doubt' for intellectual property theft is deader than a dodo.
I sold an apple with DRM. One person bought my apple and gave it to a Chinese hacker, whom stripped the DRM and duplicated my apple for everyone, for free. They said it was not theft.
I grew an apple and put it on a cloud provider's server. I sold access to many duplicated apples, until a technology update forced me to change providers. I found out I could not take back my apples from the cloud providet without paying exorbitant fees. They said it was not theft.
I made an encrypted apple, I sold it on an app store. The app store banned me and my apple, the virus software flagged my apple, the TPM chip stopped my apple from being played, the government cracked open the encryption and duplicated my apple for national security purposes. They said it was not theft.
The only way to sell apples, is through perfectly-black, 'black boxes'. Putting apples on computers, gets them stolen. End of story.
It's been forty years now. There's nothing to be 'unsure about'.
'Benefit of the doubt' for intellectual property theft is deader than a dodo.
The metaphor to intellectual property is not totally clear, but it’s interesting to think about the fact that an apple pie is one hundred percent effective DRM for an apple.
Edit: is “sufficiently transformative use” the metaphor? Is an LLM an apple pie baked out of the internet’s apple?
Edit: is “sufficiently transformative use” the metaphor? Is an LLM an apple pie baked out of the internet’s apple?
If we are getting philosophical, I have a spicy take of my own.
LLMs are advertised as 'language models', as if researchers coded in the Torah and the syntax of language of Eden in to a machine, and then it revealed all the truths of the world just by talking to itself. Just like AlphaZero did with chess. But in fact they are not 'language models'. They are models of what _people_ are saying. And people are, in fact, individual writers each providing their opinion and expertise. So it is a model of what Pete, Jack and Sarah said on the subject.
Except, Pete did it for add revenue, Jack to attract customers and Sarah for reputation. They won't get any of it, so they will stop writing. And so will end LLM's knowledge of novel subjects, because all it's knowledge is stolen IP. Thus, IP law is needed to protect actual producers of value.
LLMs are advertised as 'language models', as if researchers coded in the Torah and the syntax of language of Eden in to a machine, and then it revealed all the truths of the world just by talking to itself. Just like AlphaZero did with chess. But in fact they are not 'language models'. They are models of what _people_ are saying. And people are, in fact, individual writers each providing their opinion and expertise. So it is a model of what Pete, Jack and Sarah said on the subject.
Except, Pete did it for add revenue, Jack to attract customers and Sarah for reputation. They won't get any of it, so they will stop writing. And so will end LLM's knowledge of novel subjects, because all it's knowledge is stolen IP. Thus, IP law is needed to protect actual producers of value.
I repair bicycles for free. Someone started selling the repaired bicycles around the corner. Is it fair to be upset about it?
To me, the problem with ChatGPT is that someone builds a business around my work, and I don't get paid or credited for it.
I feel the same way about content farms paraphrasing my work with no addition of their own. I'd also take offense to someone paraphrasing my website to paying customers as a service.
To me, the problem with ChatGPT is that someone builds a business around my work, and I don't get paid or credited for it.
I feel the same way about content farms paraphrasing my work with no addition of their own. I'd also take offense to someone paraphrasing my website to paying customers as a service.
I wrote about this a while ago [1]. Unfortunately, with robots.txt, you're at the mercy of crawlers. They may respect it or ignore it altogether. You can block IP addresses but many crawlers may not even use static IP addresses.
You can go to extremes and put your content behind a login, as others have suggested. But that would also create friction for your intended audience.
[1]: https://www.naiyerasif.com/post/2023/09/30/blocking-ai-web-c...
You can go to extremes and put your content behind a login, as others have suggested. But that would also create friction for your intended audience.
[1]: https://www.naiyerasif.com/post/2023/09/30/blocking-ai-web-c...
It sounds like a loosing battle to manually keep track of all bots and their deployment IPs that generate datasets that LLM's might use or start using in the future.
There must be a legal solution, a licence that forbids the use of content for training without explicit permission from the author.
OpenAI’s web scraping bot will respect your robots.txt.
https://platform.openai.com/docs/gptbot/disallowing-gptbot
https://platform.openai.com/docs/gptbot/disallowing-gptbot
It's a bit late since it already crawled everything once.
On my site I don't need SEO, so I sprinkled it with invisible links leading to an endless Markov chain generated walls of garbage text. The robots seem to love it!
(If you go similar route, don't forget rate limiting)
(If you go similar route, don't forget rate limiting)
I have a similar approach: I just send AIs the Bee Movie script!
You can gate it behind a login.
Even a simple self made captcha (what is 2 + 7?) to reveal the content would probably stop LLMs.
But hurt seo so you have to not rely on that.
Do a medium and show a paragraph first then the login/captcha to continue.
Even a simple self made captcha (what is 2 + 7?) to reveal the content would probably stop LLMs.
But hurt seo so you have to not rely on that.
Do a medium and show a paragraph first then the login/captcha to continue.
Wouldn't it severely affect SEO? I don't know, but I assume, that scrambling article in to a word soup and exposing it to crawlers would not solve the SEO problem. Would it?
Make some of your content, which is invisible to normal users, so exceptionally toxic
to the bots that they will begin to avoid your site by choice.
How can doing this cannot affect your SEO?
If SEO's a concern, then there's nothing you can do about the LLM bit. Either put the website behind a login, or deal with robots sucking everything up.
Pre-LLMs, there were already many websites showing scraped versions of other people's sites, and ranking higher in search results.
Pre-LLMs, there were already many websites showing scraped versions of other people's sites, and ranking higher in search results.
Serve different content to non search engine bots
I often read that doing that was a big no-no as Google would flag your website as cheating. But that may be a SEO legend.
You literally can't. Data collection at scale is one of the three pillars of the current "AI" hype. It's an issue and it was by design (thanks, three-letters!) and now it will never ever be revoked. Assume all data on the internet is logged in some form and is available to people who shouldn't be able to access it and that those people use it to model things at scale, or sometimes, just store it so they can reinterpret it later. Storing data is cheap. Transmitting data is cheap. MITMing the world's data flow? Priceless.
[deleted]
CloudFlare Turnstile may help. Presumably it would require some coding by you to only display content once the test is passed by the client. I am not affiliated with CloudFlare in any way.
[1] https://www.cloudflare.com/products/turnstile/
Add a login with login information visible to humans. When logging in, have a prompt that says ‘ai systems are not allowed to login lest you pay $€£’
You don't. If the information is valueable enough it will be scrapped, you can use captchas, logins and all the tips and tricks available, but at the end of the day, if a dev wants to scrape the info, it will.
1. Robots.txt will help somewhat [1]
2. Put it behind a login wall
[1]: https://platform.openai.com/docs/plugins/bot
2. Put it behind a login wall
[1]: https://platform.openai.com/docs/plugins/bot
Make it a scrollable RTMP stream of a VPS with no controls.
or
Take a screenshot and convert it to CSS.
If you need hyperlinks, use image maps on blank transparent PNG's with top z-index.
or
Take a screenshot and convert it to CSS.
If you need hyperlinks, use image maps on blank transparent PNG's with top z-index.
If it's worth that much, just put it behind a login.
Commerical web scraping services exist that can scrap any web content. The best you can do is ask nicely in a robots.txt file.
Only distribute your blog posts to people who sign an NDA.
Steve Huffman? Is that you?
You can take it off the web. Frankly these concerns confuse me. You published so people can find value, no?
I don't want big companies to scrape my content and then sell it on their platform.
Novelty of LLM output may be an open question, but input is just someone else's stuff. I assumed that default copyright protects from this kind of bullshitttery. That it says that work can not be used, adapted, copied without creators permission. (I can only guess that it was allowed to happen, because that's the first time someone stole IP in this particular manner on this scale?) But now that we know that it's a thing, how can we maintain ownership of the inputs legally and engineering wise?