Avoiding Webscraping Throttling Using Python and Tor as a Proxy(boredhacking.com)
boredhacking.com
Avoiding Webscraping Throttling Using Python and Tor as a Proxy
https://boredhacking.com/tor-webscraping-proxy/
88 comments
If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want.
I don't think that the internet, as an organism, is responsive to ToC / T&C. Those are relics of the legacy legal system, and there is no good reason to imagine that the bots born tomorrow will see a need to integrate them.
Spending developer time fighting scraper bots is like spending watchmaker time trying to alter history; you are attempting the impossible and doing so via an unintelligible channel of influence.
I don't think that the internet, as an organism, is responsive to ToC / T&C. Those are relics of the legacy legal system, and there is no good reason to imagine that the bots born tomorrow will see a need to integrate them.
Spending developer time fighting scraper bots is like spending watchmaker time trying to alter history; you are attempting the impossible and doing so via an unintelligible channel of influence.
Well, you can spend a little time putting some anti-bot measures into place, or you can spend money paying a larger hosting bill for all the wasted CPU time, or you can spend some time explaining to legitimate users why the site fell over under the unwanted load.
The person who started this reply thread repeated part of it twice for good reason. Bots, unchecked, can and do take up a majority of the resources on small to medium sites.
I'd have been much happier with the article if the author had taken a moment to implement a bit of throttling on his end to at least go easy on the remote server.
The person who started this reply thread repeated part of it twice for good reason. Bots, unchecked, can and do take up a majority of the resources on small to medium sites.
I'd have been much happier with the article if the author had taken a moment to implement a bit of throttling on his end to at least go easy on the remote server.
> If you don't want bots loading your website and scraping it, make an API so that they can easily access the information they want.
Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind:
- Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article)
- Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!
Precisely. While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds, I have no qualms about scraping in certain situations. A couple that spring to mind:
- Where the site's UI is so dumb that it doesn't let me do what any reasonable person might naturally want to do (this would be the case in the original article)
- Where it's a govt. site and the data is only trickled out through some 90's era search function. Taxpayers paid for the collection of that data, taxpayers pay for its storage etc - give the taxpayers an API to access it!
> While scraping competitor sites for pricing info etc is exactly as sleazy as it sounds
Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
Why is it wrong to scrape your competitor's site for pricing info? If it's publically accessible, it's free game. As long as such scraping doesn't cause any undue DOS'ing or prevent anyone else from accessing such info.
It typically becomes rather "challenging" to justify to management why one shouldn't scrape "aggressively" until the competitor's site degrades or falls over. That's why it's generally kind of sleazy.
Large eCommerce sites are constantly scraped by competitors, market research companies and even vendors (to ensure sellers honor MSRP). Scraping activity is usually throttled at reasonable levels.
Could you tell us more about your "typical" experiences?
> scraping competitor sites for pricing info etc is exactly as sleazy as it sounds
So not even a tiny bit sleazy then?
So not even a tiny bit sleazy then?
Maybe the information being scrapped isn’t something you’re allowed to share via an API. Maybe you paid as licens fee to use it yourself.
We offered an API for just two customers, with images and product information for video games. One of them preferred scrapping our page anyway, so we throttled them.
We offered an API for just two customers, with images and product information for video games. One of them preferred scrapping our page anyway, so we throttled them.
Get over it, don't want them to scrape it, protect it and don't make it world accessible. I'm troubled by the fact that you're on the side that wants to control "public information". If your information is valuable and other's want it, provide API/RSS or ways to get it without impacting your site. If you don't want other's to get it, put it behind a walled garden.
Oh, this has a very easy counter. Make the price on your site to be multiple info coming from separate requests. TOR will most likely do those requests from different IP's and when you detect this you can either not show the price or show a bad one.
There is nothing you can do to stop scraper bots. People will always find a way to scrape your content.
Another problem is that blocking tor nodes is very easy. And with abusing tor nodes as free proxies, it makes a huge disservice to the Tor network, because more websites will ban it and ordinary users won't be able to use it.
This problem is being partly addressed with Privacy Pass. The addon lets you bypass recaptcha including cloudflare recaptcha minus one initial solve per session. Of course this doesn't account for all blocks and of course some malicious bots bypass recaptcha anyway using AI/speech recognition or human labor farms, but it will make the ecosystem better for everyone.
(note: it is not a captcha solving service, its a collaborative project by cloudflare, google and others)
https://privacypass.github.io/
(note: it is not a captcha solving service, its a collaborative project by cloudflare, google and others)
https://privacypass.github.io/
I'm actually not so sure that there terms of services prohibits scraping for personal use (or scraping at all) but maybe my understanding is incorrect. I have read through them here https://www.zillow.com/corp/Terms.htm and looked through there robots.txt here https://streeteasy.com/robots.txt. This is most likely because they use Distil Networks and rely on them to block web scrapers. But you are right this is something of concern and I can add a section talking about the dangers of breaking a sites terms and conditions and how scraping must be done correctly even for personal use
From that page:
5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services;
And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms.
Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.
5. Prohibited Use. BY USING THE SERVICES, YOU AGREE NOT TO: ... conduct automated queries (including screen and database scraping, spiders, robots, crawlers, bypassing “captcha” or similar precautions, and any other automated activity with the purpose of obtaining information from the Services) on the Services;
And while probably not legally recognized, the fact that they're obviously trying to prevent such usage should give the author ethical qualms.
Maybe at the bottom line there is some legal right to do it, but I'd like to see the author at least consider the ethics of what he's doing.
In many jurisdictions, "BY USING THE SERVICES, YOU AGREE ..." language is legally meaningless.
The parent suggested that it may not be legally binding, so not sure what you're pointing out. What about his main point of the ethical question - people are doing something with their service that they clearly don't want them to, because it hurts/costs them. Just because someone can get away with it doesn't make it right.
If you put data online for anyone to use, people will use it as they see fit. That's just how it is.
And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively.
If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
And TFA isn't about scraping for data to monetize. It's about finding an apartment. The author mentions that scraping is the only way to use the site's data effectively.
If it were about scraping to build an apartment-finding site, then issues about "what's right" would be relevant.
I'd imagine the site has some reason for not including that functionality. Maybe they don't have developer resources to implement it, but on the other hand, maybe for some reason the computational cost puts too much strain on their servers.
It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.
It must be true that the people offering the service have the right to decide where to draw a line between their investment in processing power versus the richness of the service's features. And if potential users don't like that decision, they're free to move on and use a different service. It can't be ethically right for everyone to just say "tough nuggies" and extract the desired service at untold cost, when the service's T&C have forbidden it.
See this: https://news.ycombinator.com/item?id=20430944
It's what I'd say, except said more clearly.
It's what I'd say, except said more clearly.
That describes things you want (and I agree that an api is a fairly reasonable thing to want), but they are not things you have the right to have just because you want them.
It may be naive to assume people will honor the desires of a website owner, but that doesn't change the moral argument - you shouldn't try to convince yourself that you're in the right just because its difficult for the website owner to enforce their wishes for how people use their system, or because everyone does it - "that's just the way it is" doesn't make it right.
I don't think we're talking about discrimination or equal access or even right to back up issues which might apply some moral right to access a site in ways not desired by a service owner - the vast majority of cases for scraping are where the scraper goes on to use that information in ways that are for the commercial benefit of the scraper and usually to the detriment of the site owner.
The arguments about data access to publicly funded information has more merit but I don't see them applying in any way to private services.
Am I missing some other fundamental right that applies here?
It may be naive to assume people will honor the desires of a website owner, but that doesn't change the moral argument - you shouldn't try to convince yourself that you're in the right just because its difficult for the website owner to enforce their wishes for how people use their system, or because everyone does it - "that's just the way it is" doesn't make it right.
I don't think we're talking about discrimination or equal access or even right to back up issues which might apply some moral right to access a site in ways not desired by a service owner - the vast majority of cases for scraping are where the scraper goes on to use that information in ways that are for the commercial benefit of the scraper and usually to the detriment of the site owner.
The arguments about data access to publicly funded information has more merit but I don't see them applying in any way to private services.
Am I missing some other fundamental right that applies here?
> scraper goes on to use that information in ways that are for
the commercial benefit of the scraper and usually to the detriment of the site owner
Yes, and they extract that benefit because in the process they add a value to that information for end-user.
> Am I missing some other fundamental right that applies here?
Yes. When it's not prohibited by law, it's allowed.
Yes, and they extract that benefit because in the process they add a value to that information for end-user.
> Am I missing some other fundamental right that applies here?
Yes. When it's not prohibited by law, it's allowed.
Sorry to say, but it seems your moral compass isn't working correctly. Not sure its productive at all to continue this conversation with someone who's understanding of that fundamental premise is so completely different than pretty much anyone I know. Hopefully that's something you'll think about.
Well, I beg to differ. I'm against kindergarten level of moral.
Look, luddism is bad for everybody in a long term. We saw riots against Uber in some country where people turn cars upside down or burn them. It hurts their earnings and they feel it's unfair. And you may say it's immoral for Uber to do so. But what they did is simple. They identified multiple huge inefficiencies in that market: pricing, negotiation, checkout, reputation. Solved it and found a way to profit from it. And that will eventually happen at every corner where huge inefficiency exists due to mere lack of communication and price negotiation. And yes, prices in most cases will have to go down and become less disperse and some won't like it at all. But that's just competition reenabled by technology.
Is it immoral? I don't think so, because net effect is positive. Is that your right to scam tourist 10x more for airport-hotel trip or take 2x longer route just to earn a bit more? Technically yes, but who would indorse such behaviour? You don't like the price? Don't take that client. Platform is systemically lowering prices or violates existing regulations? Vote for better regulations, vote for enforcing those that already exist. Platform is fundamentally broken? Well, make a better one. Technically, it's not that hard, Uber is one of the most replicated business ideas at the moment.
And on that guy complaining that oh those competitors who are scraping their prices. So they're scraping each other and protecting their websites from being scraped by each other? So at the end they all have competitors information, but pumped lots of resources in scraping, protection and trying to serve content to both bots and clients? Wow, what a tragedy. What a horrible person would want it to stop. If that's it, they could just as simply pick up a phone and agree to share that data between them, because in the end outcome will be the same, minus resources wasted on arms race.
Or better, from the beginning make your data machine-friendly. Because eventually, they'll do that. Eventually somebody like Google or Amazon or some other big company will find an incentive to make them gradually and willingly share and structure that data. And somebody will find a way and resources to integrate that data into reusable knowledge graphs and somewhere along the way create a positive feedback loop. And somebody will profit from that huge. Consumers will surely benefit, that somebody will, data-donor companies that adapt will do.
And don't forget there's some progress in ML, automated decision making and all that. I personally as a customer would love to have best prices, objective products comparison with zero interaction with multiple whacky small vendors websites. I'd better have smth like Siri do that for me.
One way or another, it's happening. Small businesses have little to say here if anything. My unpopular opinion was to recognise that process, do something to stop wasting resources on war between scrapers and anti-scrapers and hopefully avoid appearance of another single monopolist from solving the problem if we don't. Because if we don't we'll just have another few years of HN headlines about how bad that X unicorn company is to somebody.
Now what's wrong with that?
Look, luddism is bad for everybody in a long term. We saw riots against Uber in some country where people turn cars upside down or burn them. It hurts their earnings and they feel it's unfair. And you may say it's immoral for Uber to do so. But what they did is simple. They identified multiple huge inefficiencies in that market: pricing, negotiation, checkout, reputation. Solved it and found a way to profit from it. And that will eventually happen at every corner where huge inefficiency exists due to mere lack of communication and price negotiation. And yes, prices in most cases will have to go down and become less disperse and some won't like it at all. But that's just competition reenabled by technology.
Is it immoral? I don't think so, because net effect is positive. Is that your right to scam tourist 10x more for airport-hotel trip or take 2x longer route just to earn a bit more? Technically yes, but who would indorse such behaviour? You don't like the price? Don't take that client. Platform is systemically lowering prices or violates existing regulations? Vote for better regulations, vote for enforcing those that already exist. Platform is fundamentally broken? Well, make a better one. Technically, it's not that hard, Uber is one of the most replicated business ideas at the moment.
And on that guy complaining that oh those competitors who are scraping their prices. So they're scraping each other and protecting their websites from being scraped by each other? So at the end they all have competitors information, but pumped lots of resources in scraping, protection and trying to serve content to both bots and clients? Wow, what a tragedy. What a horrible person would want it to stop. If that's it, they could just as simply pick up a phone and agree to share that data between them, because in the end outcome will be the same, minus resources wasted on arms race.
Or better, from the beginning make your data machine-friendly. Because eventually, they'll do that. Eventually somebody like Google or Amazon or some other big company will find an incentive to make them gradually and willingly share and structure that data. And somebody will find a way and resources to integrate that data into reusable knowledge graphs and somewhere along the way create a positive feedback loop. And somebody will profit from that huge. Consumers will surely benefit, that somebody will, data-donor companies that adapt will do.
And don't forget there's some progress in ML, automated decision making and all that. I personally as a customer would love to have best prices, objective products comparison with zero interaction with multiple whacky small vendors websites. I'd better have smth like Siri do that for me.
One way or another, it's happening. Small businesses have little to say here if anything. My unpopular opinion was to recognise that process, do something to stop wasting resources on war between scrapers and anti-scrapers and hopefully avoid appearance of another single monopolist from solving the problem if we don't. Because if we don't we'll just have another few years of HN headlines about how bad that X unicorn company is to somebody.
Now what's wrong with that?
When it's not prohibited by law, it's allowed.
Frankly, I'm glad I don't live in your community.
I just got back from grocery shopping. Before I went in, I stomped off some mud I had on my feet. As I walked down the aisles, I accidentally knocked a bag of something off a shelf, so I turned around, picked it up, and returned it to its proper spot. When I got to the ice cream coolers, I first looked through the glass to decide what I wanted, and only opened the freezer door when I'd made my selection, sealing the door again once I was done.
There aren't any laws forcing me to do those things. But I'm happy to do them, because (a) when we all cooperate in this manner, it makes life better for all of us; and (b) the additional costs of cleaning and shrinkage would make the groceries more expensive for us if we didn't do it.
Have you ever been to a part of the world that doesn't have the same kind of "high trust" environment that most of us in the West enjoy? For example, in America we can safely assume that at a bus stop, people will of their own accord form up an orderly line when the bus comes, each person waiting their turn. My experience in some other nations is that people will swarm around the bus door, each jockeying to be the next one in. There aren't any laws forcing people to act as they do here, but I think we're all better off with the societal norm that drives people to do it our way.
Frankly, I'm glad I don't live in your community.
I just got back from grocery shopping. Before I went in, I stomped off some mud I had on my feet. As I walked down the aisles, I accidentally knocked a bag of something off a shelf, so I turned around, picked it up, and returned it to its proper spot. When I got to the ice cream coolers, I first looked through the glass to decide what I wanted, and only opened the freezer door when I'd made my selection, sealing the door again once I was done.
There aren't any laws forcing me to do those things. But I'm happy to do them, because (a) when we all cooperate in this manner, it makes life better for all of us; and (b) the additional costs of cleaning and shrinkage would make the groceries more expensive for us if we didn't do it.
Have you ever been to a part of the world that doesn't have the same kind of "high trust" environment that most of us in the West enjoy? For example, in America we can safely assume that at a bus stop, people will of their own accord form up an orderly line when the bus comes, each person waiting their turn. My experience in some other nations is that people will swarm around the bus door, each jockeying to be the next one in. There aren't any laws forcing people to act as they do here, but I think we're all better off with the societal norm that drives people to do it our way.
Oh hey, I got where you were mistaken! :)
You assumed that people who don't want their data being scraped are the nice guys and people who want to use that data are the baddies. Nope, my point was that it's the other way around and if you're truly concerned with community wellbeing you actually have to be on my side of this argument.
I support every point in your response expect that I don't get your motivation to fight for folks who either don't understand how internet works or consciously use dirty tricks to block information access just to protect their shaky profits.
Look it up, my first comment in this thread was about making it illegal because I think everybody would benefit from making it so. If you publish your data and it's accessible by browser and not copyrighted, we shouldn't make it hard to collect that data for automated processing.
You assumed that people who don't want their data being scraped are the nice guys and people who want to use that data are the baddies. Nope, my point was that it's the other way around and if you're truly concerned with community wellbeing you actually have to be on my side of this argument.
I support every point in your response expect that I don't get your motivation to fight for folks who either don't understand how internet works or consciously use dirty tricks to block information access just to protect their shaky profits.
Look it up, my first comment in this thread was about making it illegal because I think everybody would benefit from making it so. If you publish your data and it's accessible by browser and not copyrighted, we shouldn't make it hard to collect that data for automated processing.
How valuable is that legally? If my bot is scraping that automatically, have I then agreed to that?
> ethical qualms
What ethical qualms are those? The fact that someone else tells you that by using something you agree with a bunch of agreements that they made up on the spot?
What ethical qualms are those? The fact that someone else tells you that by using something you agree with a bunch of agreements that they made up on the spot?
I think many people aren't going to stress too much about breaking a site's T&C because a) nobody ever reads them and b) they're practically unenforceable in many places.
However, I do agree that people should care more about the ethics of web scrapping.
However, I do agree that people should care more about the ethics of web scrapping.
>I think many people aren't going to stress too much about breaking a site's T&C because a) nobody ever reads them and b) they're practically unenforceable in many places.
Tell that to Aaron Swartz.
Tell that to Aaron Swartz.
"the author seems not to have given the slightest thought that he's breaking the site's T&C"
LOL! Never read 'em, never going to, couldn't care less about any of them.
If it was that important they'd make people create and account and sign in and limit access that way. Don't expect people to take all the blah blah blah seriously. It's just a web page.
LOL! Never read 'em, never going to, couldn't care less about any of them.
If it was that important they'd make people create and account and sign in and limit access that way. Don't expect people to take all the blah blah blah seriously. It's just a web page.
And this is how you get the entire Internet locked behind login pages.
Abusing free and open leads to people locking down free and open.
Abusing free and open leads to people locking down free and open.
I'm not sure that's true. Free and open drives more page views and is a more successful marketing strategy where it makes sense.
"lol" was my first thought as well, and I suspect many peoples, despite the down votes.
For non-commercial projects I always use at least a one second delay, even for large sites. For commercial projects I have actually advocated to clients that they contact the site's owners and ask to arrange a data use agreement. And a couple have really done it and been satisfied by the result which was cleaner data and continuous updates at a reasonable cost.
Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!
Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!
Yeah, I totally agree with you, but also make sure you're not being an asshole with websites and hammer them to the point of bringing them down. Be respectful of their hardware _and_ take the data you need. No need to be burn everything down.
I certainly wouldn't want to burn it down as long as I still need the data, other than that I couldn't care less.
You could easily put your pricing info on a single page. You’re not paying for bandwidth. You’re paying for obscuring information. You’re paying for going against the purpose and spirit of the internet.
Can the site sit behind a CDN like Cloudflare? Seems like that would solve any issues caused by bots - and furthermore, Cloudflare can captcha challenge suspected bot traffic
Unpopular opinion here. I think that it would be better for most if scraping prevention was illegal if there's no API that allows authorised application to perform every action accessible to authorised user. Just imagine, how much better UX would be if we could get rid of those small imaginary data silos.
It sounds like offering a paid API is the perfect pivot for your business.
> breaking the site's T&C
T&C's have absolutely no bearing on publicly accessible information [1][2][3]. They only apply to registered users accessing login-required portions of your website or application. Browsewrap does not represent a legally binding contract, no matter how much you pay your lawyers to write up your irrelevant T&C's.
[1] https://www.eff.org/deeplinks/2018/04/scraping-just-automate...
[2] https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...
[3] https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...
T&C's have absolutely no bearing on publicly accessible information [1][2][3]. They only apply to registered users accessing login-required portions of your website or application. Browsewrap does not represent a legally binding contract, no matter how much you pay your lawyers to write up your irrelevant T&C's.
[1] https://www.eff.org/deeplinks/2018/04/scraping-just-automate...
[2] https://www.eff.org/deeplinks/2018/04/dc-court-accessing-pub...
[3] https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-...
The world is a lot larger than just the United States. Besides that, even publicly available information can be copyrighted. So the scraping itself may be legal, storing that information; using it and re-publishing it very likely is not.
> The world is a lot larger than just the United States.
Where would this be significantly different? For T&C to be valid, you'll have to agree to them before using the service. Since you generally can't know, much less agree to, the T&C of a website before using that website, you have a chicken & egg problem where the law tends to come down on "these T&C aren't binding" imho.
You're right about the re-publishing though. Storing is likely a different issue, even browser-caching is storing; iirc there were court decisions in Germany that essentially argued that even having data in memory is creating a copy.
Where would this be significantly different? For T&C to be valid, you'll have to agree to them before using the service. Since you generally can't know, much less agree to, the T&C of a website before using that website, you have a chicken & egg problem where the law tends to come down on "these T&C aren't binding" imho.
You're right about the re-publishing though. Storing is likely a different issue, even browser-caching is storing; iirc there were court decisions in Germany that essentially argued that even having data in memory is creating a copy.
In Germany there was a recent ruling that scraping, even for competing purposes, is fine.
https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scr...
https://www.heise.de/newsticker/meldung/Urteil-zu-Screen-Scr...
In the EU there is the concept of ‘database rights’ which protects your investment creating and publishing large datasets. This legislation has nothing to do with site T&Cs.
> iirc there were court decisions in Germany that essentially argued that even having data in memory is creating a copy
My point exactly.
My point exactly.
For a non-US (England and Wales) example:
> After seeking brief details of his employment, gross income, and savings, and whether he was a property owner, [the online bookmaker] asked him to specify a password, a memorable question and answer, and a bank account number.[...]
> In seeking to identify such a pre-existing contract counsel for the claimant faced significant difficulties. In particular he was unable to point to any promise or commitment by the claimant which might form part of such a contract and provide the consideration necessary to make it legally binding.[...]
> The provision of an on-line interactive platform is in effect simply a more modern equivalent of the expressed readiness of a potential contracting party [...] to enter into contracts by receiving and responding orally to telephone calls.
> Accordingly, the claimant has failed to establish the existence of any legal contract to host and give binding effect to [the clause]
Also of note:
> Many, one might suspect most, would have passed up on that invitation [to read the documents] and proceeded directly to click on "Agree", even though it was suggested that they should do so only when they had read and understood the documents. Even if, exceptionally, the defendant in fact chose to look at the documents, he would have been faced [with a long and complex agreement]. It would have come close to a miracle if he had read [the clause], let alone appreciated its purport or implications, and it would have been quite irrational for the claimant to assume that he had. This was an entirely inadequate way to seek to make the customer liable [...], and is a further factor rendering the [clause] an unfair term.
Spreadex Ltd v Cochrane [2012] EWHC 1290 (Comm) http://www.bailii.org/ew/cases/EWHC/Comm/2012/1290.html
Note that this is not necessarily a full appraisal of the law, but merely meant to illustrate that forming a binding contract is more difficult than having terms available. Furthermore, even if there is no contract, there are IP considerations (copyright, database rights depending on the data) before you can use the data.
> After seeking brief details of his employment, gross income, and savings, and whether he was a property owner, [the online bookmaker] asked him to specify a password, a memorable question and answer, and a bank account number.[...]
> In seeking to identify such a pre-existing contract counsel for the claimant faced significant difficulties. In particular he was unable to point to any promise or commitment by the claimant which might form part of such a contract and provide the consideration necessary to make it legally binding.[...]
> The provision of an on-line interactive platform is in effect simply a more modern equivalent of the expressed readiness of a potential contracting party [...] to enter into contracts by receiving and responding orally to telephone calls.
> Accordingly, the claimant has failed to establish the existence of any legal contract to host and give binding effect to [the clause]
Also of note:
> Many, one might suspect most, would have passed up on that invitation [to read the documents] and proceeded directly to click on "Agree", even though it was suggested that they should do so only when they had read and understood the documents. Even if, exceptionally, the defendant in fact chose to look at the documents, he would have been faced [with a long and complex agreement]. It would have come close to a miracle if he had read [the clause], let alone appreciated its purport or implications, and it would have been quite irrational for the claimant to assume that he had. This was an entirely inadequate way to seek to make the customer liable [...], and is a further factor rendering the [clause] an unfair term.
Spreadex Ltd v Cochrane [2012] EWHC 1290 (Comm) http://www.bailii.org/ew/cases/EWHC/Comm/2012/1290.html
Note that this is not necessarily a full appraisal of the law, but merely meant to illustrate that forming a binding contract is more difficult than having terms available. Furthermore, even if there is no contract, there are IP considerations (copyright, database rights depending on the data) before you can use the data.
I totally agree with you. Yes there is some national laws which differ but in the end it's open data.
Today everything open enough can be considered "free to do whatever people want to do with it". Before doing anything in this world I think you'll have to face this and think about how to deal with it if you have to.
OT: I don't think it's great that small competitors have problems because of this but on the other side I'm not one of those people who repeat the idea of "competition as being good for innovation & customers" that we all heard a lot in some economic lectures.
I think in an ideal world there'd be no need for more than 1 basic model of smartphone, notebook or car. In this disney world we'd have global efforts to innovate, develop an manufacture what we need and build that stuff in the best possible quality with fairness, modularity and reparability in mind.
Yes you'd call this monopolies but in the end the freedom from this endless competition makes another way to work possible. Also innovation doesn't have to come from competition as we've seen from many examples (including what the first intelligent "homo XYZ" invented).
Then of course there'd be no need to scrape for battling your competition. We'd have APIs for everything we'd like to know. Wouldn't this be nice?
Today everything open enough can be considered "free to do whatever people want to do with it". Before doing anything in this world I think you'll have to face this and think about how to deal with it if you have to.
OT: I don't think it's great that small competitors have problems because of this but on the other side I'm not one of those people who repeat the idea of "competition as being good for innovation & customers" that we all heard a lot in some economic lectures.
I think in an ideal world there'd be no need for more than 1 basic model of smartphone, notebook or car. In this disney world we'd have global efforts to innovate, develop an manufacture what we need and build that stuff in the best possible quality with fairness, modularity and reparability in mind.
Yes you'd call this monopolies but in the end the freedom from this endless competition makes another way to work possible. Also innovation doesn't have to come from competition as we've seen from many examples (including what the first intelligent "homo XYZ" invented).
Then of course there'd be no need to scrape for battling your competition. We'd have APIs for everything we'd like to know. Wouldn't this be nice?
Sounds like you need to make things easier rather than harder. They have a clear economic incentive to scrape your site and will continue to do so. Rather than spending your employees time on it, open it up as an API. If someone starts hitting it too hard, send them an email to back off.
If your only competitive advantage is your pricing, then you're in a losing game. Otherwise, you're probably better off spending your efforts on making your product/service better.
If your only competitive advantage is your pricing, then you're in a losing game. Otherwise, you're probably better off spending your efforts on making your product/service better.
This approach is not very realistic. 1. API design and auth takes engineering time and potentially brings more traffic you need to pay for. 2. You don't have an email address to send "back off" to. Even if you had, no enforcement means no change. 3. Blocking / throttling takes much less time and money than creating a new endpoint.
Charge your competitors for API tokens?
It sounds like your problem is your confusion about trying to prevent devices from accessing your public data over the Internet. Have you tried adding an rss feed or another API instead?
Your site is public, what do you expect? It's not abuse to load public websites. T&C is meaningless for public sites. Equivalent to real life sign hanging in a shop "whoever reads this is bound to the following conditions". That's not how contracts work
If you block the small guys from scraping you're just helping already huge monopolies tighten their grip on the content market. Even "nice" scrapers like Google blatantly steal your content, blocking everyone else just makes their monopoly worse. They steal pricing data for "Google Shopping" regularly.
If it annoys you that much just build an API and charge tiny amount to use it.
If you block the small guys from scraping you're just helping already huge monopolies tighten their grip on the content market. Even "nice" scrapers like Google blatantly steal your content, blocking everyone else just makes their monopoly worse. They steal pricing data for "Google Shopping" regularly.
If it annoys you that much just build an API and charge tiny amount to use it.
It's overkill to use Tor for this. And I'm a little surprised that it works at all well, because Tor exits so commonly trigger CAPTCHAs.
Better, I think, would be using HTTPS proxies. But not free ones, which tend to get burned down pretty quickly. There are sites that lease private proxies, and guarantee that they work.
Better, I think, would be using HTTPS proxies. But not free ones, which tend to get burned down pretty quickly. There are sites that lease private proxies, and guarantee that they work.
There are such paid proxy services that provide you with a refreshable IP pool. Unfortunately, you can't control which IP they give you at each request (or when you can, your request rate drops significantly) so they do not work with websites which require you to keep a session alive.
I use crawlera, and it does have the capability to establish and maintain a session. I can't pick the precise initial IP but once the connection is made I can certainly tell it to keep using that IP.
Sessions are the only way to use crawlera with libraries like cloudflare-scrape, which pin your authentication to a specific IP.
Sessions are the only way to use crawlera with libraries like cloudflare-scrape, which pin your authentication to a specific IP.
That is interesting. $1000 tier is a considerable amount, but perhaps one can find a profitable line of products to justify the price. Of course, this is too expensive for initial development most of the time, so I'd go for an in-house network of machines initially.
HTTP/S proxies probably wouldn't want you burning their IPs on web scraping though - it might be against the license/lease terms of the site.
OK, top DDG result (an ad) for "https proxies web scraping":
> https://smartproxy.com/scraping/proxies
> Global Residential Proxy Network for Web Scraping. Never Get Blocked Again. High Reliability, Easy to Use. Choose Your Plan Now!
> https://smartproxy.com/scraping/proxies
> Global Residential Proxy Network for Web Scraping. Never Get Blocked Again. High Reliability, Easy to Use. Choose Your Plan Now!
What are the odds that this is essentially a botnet made of infected computers and IoT devices with unsuspecting owners?
Infecting computers and IoT devices takes too much effort these days. The latest method is to have the users brought to you - just publish a SDK and pay developers to stick it into their apps. Here's an example: https://luminati.io/sdk
Luminati has not killed off the other providers in the market.
Extremely low. A temporary IP is cheaper, easier, and less legally risky to obtain compared to a hacked endpoint. Remember you're taking real, trackable money for that service. Iot devices also don't have an amazing throughout.
Well, Tor is usually blocked. If you want something more reliable, use private proxy services, for example https://gimmeproxy.com
<span class="pull-right" id="ipv4">2a0b:f4c1::7</span>
xxx.xxx.xxx.xxx and the mask. in his example:
2a = 42; 0b = 11; f4 = 244; c1 = 193; mask = 7 (weird i know)
so the IP is actually 42.11.244.193. A short trip to any free IP locator site says that IP is from South Korea
If only websites have their data dumps for free instead of html reverse engineering.
Not sure how long this will last since there are a limited number of exit nodes. Many networks throw up CAPTCHA for such nodes.
Ah, yes, the captcha arms race. I've encountered bots that can solve simple text captchas (especially bots routed through Tor) and as ML gets cheaper and more accurate I do wonder what the end result will be...
Well, for Tor in particular, the most likely end result is Tor being banned from an increasing number of websites.
That already happened.
Yeah honestly surprised scraping through tor would work. I imagine most commercial solutions routinely block exit nodes.
Depending on how you read the embargo laws for your country, you may be required to block them. If you identify the source of the traffic as a non-country, dropping it may be what you want to do.
[deleted]
It's higly likely that a web site will greet any connections from Tor with captcha.
Cool article but couldn't you just spoof your IP using something like socket?
Replying to myself here. This will not work because the packet still has to go back to the spoofed address. What a poorly thought out comment.
One could test to change ipv6 address, it might work.
This method will not work for many websites that require you to stay logged in.
[deleted]
I'm particularly sensitive to this because I'm constantly dealing with scraper bots from competitors that are trying to monitor our pricing. Without our ongoing policing, and a fair amount of developer time going into it, the traffic coming from these bots - and hence the amount it costs us to operate the site - is significantly larger than that of actual customers. Let me say that again: scraper bots account for more traffic on our sites than do legitimate customers.