We've used https://foundershield.com/ for a few years now. They have managed to help us from a small E&O and supplier liability insurance, as well as cyber liability– to a much larger international insurance spanning multiple organizations.
We don't expect anyone to pay for Scrapy or Portia!
We provide the best Platform (as a Service) to run Scrapy or Portia spiders, and will soon be supporting most standard web scraping technologies. This is free for light users, but we charge for people who need extra or dedicated computing or network resources.
We also provide help to startups or enterprise orgs looking to get help in building a web data harvesting system (more than just parsing pages!), either by building it ourselves or by helping our partners train their engineers in using our technologies.
This has worked so far, and we're very healthy from a revenue perspective – more than doubling every year for a few years now, and good enough to grow to become the largest fully distributed company outside of the US.
We're pretty happy with being a brand that gives to the community, it tends to get repaid 10x in the long run.
If you need to build a solid web scraping stack which is going to be maintained by many people and is critical to your business, you have two options… to use Scrapy or to build something yourself.
Scrapy has been tried and tested over 6-7 years of community development, as well as being the base infrastructure for a number of >$1B businesses. Not only this, but there is a suite of tools which you have been built around it – Portia for one, but also other lots of useful open source libraries: http://scrapinghub.com/opensource/).
Right now most people still have the issue of having to use xpath or css selectors to run your crawl or get the data, but not for too long.
Scrapy (and also lots of python tools, likely a majority of them created by people using it and BeautifulSoup) have lowered the cost of building web data harvesting systems to the point where one guy can build crawlers for an entire industry in a couple of months.
Hi! This is definitely not legal advice, so consult with a lawyer and do your own research if you are thinking of applying this to your own practices.
I work for Scrapinghub as well and try to understand the law around this. I can help with some pointers to why I think some web scraping isn't illegal… there are of courses some limits to this.
When the data scraped is "is publicly available on the Internet, without requiring any login, password, or other individualized grant of access", the Eastern District Court of Virginia in Cvent vs Eventbrite (https://casetext.com/case/cvent-inc-v-eventbrite) ruled one could not be deemed to be exceeding unauthorized access.
There are two ways, that I know of, that courts have ruled you can exceed your authorization:
- When the site owner has contacted you and removed your authorization in a written manner, as happened on Craigslist vs 3taps.
- By accepting the terms of service and agreeing against scraping. You have to do this through a "clickwrap" ToS, rather than a "browsewrap". You can read about the differences here: https://termsfeed.com/blog/browsewrap-clickwrap/
As a matter of policy, we don't scrape any site with a ToS with clear anti-scraping language and which forces us to create an account or "constructively agree" as part of the use of the site.
Any user wishing to revoke authorization for anyone using our platform can make an abuse report on our site– we tend to handle these within 24 hours and haven't had a single claim go further than this stage, as we aim to be reasonable and look for a way to provide value to both sides.
Scrapinghub Ltd. is looking for PYTHON (Scrapy, Django) and ERLANG Engineers, as well as SALES and SUPPORT engineers.
We're a fully distributed company (largest founded outside of the US!) with 107 engineers and staff. So totally REMOTE.
Based around open source, we maintain Scrapy, Portia, Webstruct, Frontera, and a lots of other tools made for crawling and scraping massive web datasets– everyone at SH helps makes these projects grow, and we offer to pay you to work on open source if you're good enough.
You'll have the chance to work on projects that harvest and transfer datasets of thousands of millions of records, as well as build some of the systems that will deliver data to current Fortune 500 companies and the startups that are building great products on top of our stack.
We have a very engineering-driven culture (two engineer-founders) and a great place to work if you're self-directed and curious, and interested in working in open source environments.
NOTE: For some reason, I’ve been getting a LOT (like more than 1, which is a lot for me considering how long this site has been going) of messages from people asking me if I’m some kind of famous programmer called #why from Pittsburg or something.
Whilst I wish I was this famous and did get this much attention, unfortunately I’m not. I’m just a dude trying to turn his life around and take a bit of control over what’s going on.
This is actually a very good content marketing strategy. I'd enjoy being the canonical publisher of these pieces of unique long-form content about an industry that just gets bigger and bigger every year (and interest in it at 3x).
It will make an even bigger name from them outside of the "core" startup community... can piggyback off any big brands, directly from their own, as all they need to do is call someone and say "hey, we're from Basecamp" since so many people love them.
From my experience in internet businesses in developing countries, you tend to get quite strong weekly patterns as a lot of people only have access to computers at work, so usage tends to drop on the weekend.
A lot of this isn't actually the CBO's fault... because they're totally non-political, or need to appear to be, they score things according to the letter of the law. The thing is, the letter of the law is never followed, as whenever the painful (cost reduction) parts start to apply, Congress votes to nullify the pain, which makes them more expensive.
That's what so many people miss. They see "Bitcoin" and forget about the technology behind it. The ability to create "cryptocurrencies" at no cost is going to be very interesting, I can't wait to see what we can build on this basic logic.