Ask HN: How to Get Sports Data?
I am thinking of creating a free REST API service that will allow users to list tournaments, schedules, match results etc - for basketball, cricket, badminton. How do I get the data into my database? Do I need to purchase this from someone or I simply do web scraping of different websites to get the data I want?
12 comments
If you want to give this data away for free, thats noble of you, but might trigger lawsuits.
In general, larger players have connections to private apis that come right from the field. Sports betting, espn, etc all pay for expensive connections to get live data.
The historical data will be time consuming but most likely legally safe.
Its the live match results data that you should not scrape in my opinion. Technically it will take some time to set up, but its possible. Its the legal part of all this that I would not recommend.
If i were you, id make two services: free historical data, and a paid live data api. I dont know much about sports data api in particular, but my question is why isnt this already out there? Probably because its very expensive to get enough feeds to have a live api anyone wants to use.
In general, larger players have connections to private apis that come right from the field. Sports betting, espn, etc all pay for expensive connections to get live data.
The historical data will be time consuming but most likely legally safe.
Its the live match results data that you should not scrape in my opinion. Technically it will take some time to set up, but its possible. Its the legal part of all this that I would not recommend.
If i were you, id make two services: free historical data, and a paid live data api. I dont know much about sports data api in particular, but my question is why isnt this already out there? Probably because its very expensive to get enough feeds to have a live api anyone wants to use.
Would it be possible to follow your progress somehow?
Betfair API might help but check their terms of service to make sure you are not breaking it.
How does Betfair get this data into their system? Who provides them with this data or they too write their own web scraping application to collect and consolidate the data?
Probably it is manual because they need to take bets and decide what to pay out on. If they get it wrong, they lost a lot of money and/or credibility. To be totally sure, DYOR.
There is no way they can trust scraped data. Especially when you can have different matches with slightly different names. For example U21 version of A vs B for a givens sport (learned that the hard way when arbing!).
There is no way they can trust scraped data. Especially when you can have different matches with slightly different names. For example U21 version of A vs B for a givens sport (learned that the hard way when arbing!).
Don’t purchase anything if it isn’t for live play by play feeds. That can run you almost 500-1k a year per sport (so it will add up).
A personal solution I’ve been considering is a scraper that knows how to scrape across a few different sites (so you have fallbacks). This can be used to scrape real time data and distribute data fetching to avoid rate limits or reliance on one site (let’s say cbs sports).
For historical data, you’ll have to get attuned to wiki or specific sports sites that have archives (they are out there). Perfect job to outsource to anyone really if you don’t feel like doing the tedious work.
I’d say scraping is the way so long as your scraper knows how to get the same data from multiple sources. If you got the cash then just google a sports data api.
A personal solution I’ve been considering is a scraper that knows how to scrape across a few different sites (so you have fallbacks). This can be used to scrape real time data and distribute data fetching to avoid rate limits or reliance on one site (let’s say cbs sports).
For historical data, you’ll have to get attuned to wiki or specific sports sites that have archives (they are out there). Perfect job to outsource to anyone really if you don’t feel like doing the tedious work.
I’d say scraping is the way so long as your scraper knows how to get the same data from multiple sources. If you got the cash then just google a sports data api.
Thanks. But i am still curious to know how does ESPN gets all the schedule for all sports in their persistence store? Does NFL, NBA, IPL or other sports league provide them with APIs to fetch data or does ESPN too has web scraping scripts that goes after NBA/NFL/MLB websites to get the data they need or do they have people actually doing manual data entry for all of this information via some UI into their own database?
I know a guy who works in sports betting software, a pretty large player. They have direct access feeds that they pay a lot for bulk access, to cbs sports, nbc, etc. I dont know how much but its no small sum. ESPN does not use web scraping for live stats- they use high speed apis that get info right from the field (that either their crew gathers or cbs gathers and they pay for access). The overwhelming majority is not manually entered.
I also do some scraping in my spare time and i can tell you that large companies will have... precautions in place to prevent you from scraping their by-the-minute sports stats they pay millions for. Id also say right now scraping is a gray market, airlines in particular have proven that they are willing to fight for their (public) data, so tread carefully.
Scraping is a problematic way to do this too, because how can you verify what is up to date? You have to rely on one single authoritative source, otherwise youll be relying on some stat calc which “guesses” what the right stat is. Additionally, if you take live data that multiple large businesses feel they own, now they can split the lawyering fees on you and even if they lose, it will be cheaper for them together.
The historical content probably would be trivial to get, and its hosts most likely dont care if you have it. its the recent/live data which may be particularly difficult to take. In my professional opinion this is two separate projects- a paid up to date version with live stats and a free historical wiki.
I also do some scraping in my spare time and i can tell you that large companies will have... precautions in place to prevent you from scraping their by-the-minute sports stats they pay millions for. Id also say right now scraping is a gray market, airlines in particular have proven that they are willing to fight for their (public) data, so tread carefully.
Scraping is a problematic way to do this too, because how can you verify what is up to date? You have to rely on one single authoritative source, otherwise youll be relying on some stat calc which “guesses” what the right stat is. Additionally, if you take live data that multiple large businesses feel they own, now they can split the lawyering fees on you and even if they lose, it will be cheaper for them together.
The historical content probably would be trivial to get, and its hosts most likely dont care if you have it. its the recent/live data which may be particularly difficult to take. In my professional opinion this is two separate projects- a paid up to date version with live stats and a free historical wiki.
Thanks. I do not have any intention to do web scraping. Let's reduce the scope of my request: I want to do two things - 1) Get the schedules of the upcoming (future) matches in most of the popular sports across globe - football, soccer, cricket, tennis etc. How does ESPN or other sources get this? Do they get this information 3 months/6 months in advance from all these sports organizations? Or do they simply have people to enter this data manually in their own system? 2) I want to keep history of who won/lost the particular match and any other interesting tidbits about the match that already took place. So I am not planning on doing any web scraping and neither I plan to offer any live feed API or data. So my question is - how do I accomplish my above stated 2 goals efficiently without doing web scraping? Thanks.
http://www.espn.com/apis/devcenter/docs/calendar.html#using-...
Thats a start
I tried looking for an nba api but i couldnt find one- but i found nba api projects like https://pypi.org/project/nba-api/
I dont know espn gets their historical data but scraping it wouldnt be problematic. I would think the pro sports orgs would have an api for it
Looks like espn has pretty good historical data records and their api has some documentation. Using the espn api might be your best bet here
Thats a start
I tried looking for an nba api but i couldnt find one- but i found nba api projects like https://pypi.org/project/nba-api/
I dont know espn gets their historical data but scraping it wouldnt be problematic. I would think the pro sports orgs would have an api for it
Looks like espn has pretty good historical data records and their api has some documentation. Using the espn api might be your best bet here
I think ESPN does not support APIs anymore. http://www.espn.com/apis/devcenter/blog/read/publicretiremen...
Are there any legal problems with scraping sports data?