Minebase: a free data mining tool for social networks(minebaseapp.com)
minebaseapp.com
Minebase: a free data mining tool for social networks
https://minebaseapp.com
52 comments
[deleted]
This is my first web app and, while rather simple, should provide some use to people needing to export large amounts of data from Twitter and Facebook. In the coming weeks I'll be adding some new exports and support for more social networks. I'm really excited to be able to start showing it off and can't wait to get some feedback. Thanks!
Wouldn't undersell yourself by saying "rather simple." This is an ambitious and impressive effort, even if it were your 50th web app.
Good job shipping.
Good job shipping.
Thanks! I suppose the "simple" part was referring the fact that I'm not performing any manipulation on the data.
Im a sociologist. You made my life a little bit easier. Thank you.
Glad to hear it!
With all due respect, it is not really a "free tool", it is seemingly a site where you have to sign up to get something.
This is also not particularly reassuring (I understand that being a beta it is probably only a temporary page copied from somewhere on the web):
https://minebaseapp.com/privacy
>Credit card information
We do not store your credit card information directly. It is passed through to a third party credit card processing merchant using 128-bit SSL encryption. We only keep your credit card's brand, last 4 digits and expiration date to assist and remind you of any billing related issues.
...
>Last updated April 1st 2042
Same goes for "Terms of Service":
https://minebaseapp.com/terms
This is also not particularly reassuring (I understand that being a beta it is probably only a temporary page copied from somewhere on the web):
https://minebaseapp.com/privacy
>Credit card information
We do not store your credit card information directly. It is passed through to a third party credit card processing merchant using 128-bit SSL encryption. We only keep your credit card's brand, last 4 digits and expiration date to assist and remind you of any billing related issues.
...
>Last updated April 1st 2042
Same goes for "Terms of Service":
https://minebaseapp.com/terms
The "SSL" cryptographic operator is generally secure when used with 128-bit keys.
To be honest, I just took a boilerplate privacy statement, but you're right, I should have checked that first and removed that bit.
Yep, and please do change/correct the date, as is it does seem like an April Fool's day (in the future).
Fixed!
Come on, requiring an account doesn't make something "not a free tool".
Mind you this is certainly NOT the case, the Author is seemingly a very nice chap and has all the good intentions in this world.
But "free" still means "free", and "tool" still means "tool".
It is not strictly speaking a "tool" it is some form of SaaS, you have to connect to that site, authenticate yourself, etc.
It is not "free" the sheer moment it asks (indirectly, through your e-mail address) your identification.
That there is "nothing bad" in it is another thing.
But "free" still means "free", and "tool" still means "tool".
It is not strictly speaking a "tool" it is some form of SaaS, you have to connect to that site, authenticate yourself, etc.
It is not "free" the sheer moment it asks (indirectly, through your e-mail address) your identification.
That there is "nothing bad" in it is another thing.
It doesn't have to make it "not a free tool", but in my experience it often does. Why even bother with an account system? Nothing on the front page of this site suggests an account is technically necessary.
Yet the owner of the site is presumably paying for a DB to store all this information. Why?
Usually there is one of two answers:
A) They want to sell you shit, and they need some kind of identifier they can use to unlock content
B) They want to collect info on you to sell to other people
Either way, that's not free. If it's A, I am being asked to invest time in a tool without being told it's potential limitations (there's nothing on the front page mentioning limitations on unpaid accounts), and that's a cost.
If it's B, I'm being data-mined, and companies only pay for data-mining if they think they can turn it into more profit, data-mining costs included. So if I'm being data-mined, it's because someone thinks they can use that data to get me to part with more money at some point down the line than I would choose to part with sans manipulation. That's a cost.
Of course, there's always option C) The site creator hasn't thought about why they need an account system, and made one because that's what all the other cool websites do.
Usually when C comes into play, you can expect account data leakages to follow due to piss poor security practices, and that's a cost.
Yet the owner of the site is presumably paying for a DB to store all this information. Why?
Usually there is one of two answers:
A) They want to sell you shit, and they need some kind of identifier they can use to unlock content
B) They want to collect info on you to sell to other people
Either way, that's not free. If it's A, I am being asked to invest time in a tool without being told it's potential limitations (there's nothing on the front page mentioning limitations on unpaid accounts), and that's a cost.
If it's B, I'm being data-mined, and companies only pay for data-mining if they think they can turn it into more profit, data-mining costs included. So if I'm being data-mined, it's because someone thinks they can use that data to get me to part with more money at some point down the line than I would choose to part with sans manipulation. That's a cost.
Of course, there's always option C) The site creator hasn't thought about why they need an account system, and made one because that's what all the other cool websites do.
Usually when C comes into play, you can expect account data leakages to follow due to piss poor security practices, and that's a cost.
It's easier to throttle people if you have to use an account to access the information.
> It's easier to throttle people if you have to use an account to access the information.
That's not true in the slightest. I can create as many accounts as I can have email addresses, and I can have as many email addresses as I want.
Robust throttling _requires_ an alternative throttling mechanism that works independent of account information, otherwise I can round-robin between accounts to bypass all throttling.
If you're gonna have that anyway, we're back to "no technical need for accounts".
That's not true in the slightest. I can create as many accounts as I can have email addresses, and I can have as many email addresses as I want.
Robust throttling _requires_ an alternative throttling mechanism that works independent of account information, otherwise I can round-robin between accounts to bypass all throttling.
If you're gonna have that anyway, we're back to "no technical need for accounts".
> Robust throttling _requires_ an alternative throttling mechanism that works independent of account information
If they use Facebook login instead of email, how do you plan on getting thousands of Facebook accounts?
If they use Facebook login instead of email, how do you plan on getting thousands of Facebook accounts?
A) They didn't require facebook login
B) It's been a while since I last checked, but I don't believe there's anything stopping me from just making shittonnes of facebook accounts
C) Facebook accounts are about 90c each, if you're desperate: https://buyaccs.com/en/buy-bulk-facebook-accounts.php
Accounts are not a throttling mechanism. In and of themselves, they cannot be.
B) It's been a while since I last checked, but I don't believe there's anything stopping me from just making shittonnes of facebook accounts
C) Facebook accounts are about 90c each, if you're desperate: https://buyaccs.com/en/buy-bulk-facebook-accounts.php
Accounts are not a throttling mechanism. In and of themselves, they cannot be.
Account system reduces the amount of people bypassing the 1 concurrent export per person limit (although I could do more on this front). I used this site so I can get a job coding since I'm an English Lit major who doesn't want to spend more money on school, which, thankfully, has recently paid off as I'm starting a new position next month. Also, $10 a month isn't a lot to pay for a fun project that will (maybe) help a bunch of people, but I can understand the suspicion.
>Account system reduces the amount of people bypassing the 1 concurrent export per person limit (although I could do more on this front).
It doesn't do that at all, see here:
https://news.ycombinator.com/item?id=14640988
It doesn't do that at all, see here:
https://news.ycombinator.com/item?id=14640988
Craigslist put the kabash on something similar.
Why won't other sites?
Why won't other sites?
I was thinking that since I wouldn't be charging for any of the services that it shouldn't be a problem and there are plenty of paid sites that offer the same thing. But who knows.
Congrats on shipping this! Personally, I wish that social networks themselves offered bulk export of public assets for academic purposes instead of having people re-invent the wheel every time.
Just a heads up, IANAL, but make sure that you're considering Twitter (and other social network) API terms of service:
> If you provide Content to third parties, including downloadable datasets of Content or an API that returns Content, you will only distribute or allow download of Tweet IDs, Direct Message IDs, and/or User IDs.
> You may, however, provide export via non-automated means (e.g., download of spreadsheets or PDF files, or use of a “save as” button) of up to 50,000 public Tweet Objects and/or User Objects per user of your Service, per day.
> Any Content provided to third parties remains subject to this Policy, and those third parties must agree to the Twitter Terms of Service, Privacy Policy, Developer Agreement, and Developer Policy before receiving such downloads.
> You may not distribute more than 1,500,000 Tweet IDs to any entity (inclusive of multiple individual users associated with a single entity) within any given 30 day period, without the express written permission of Twitter.
[0] https://dev.twitter.com/overview/terms/agreement-and-policy
Just a heads up, IANAL, but make sure that you're considering Twitter (and other social network) API terms of service:
> If you provide Content to third parties, including downloadable datasets of Content or an API that returns Content, you will only distribute or allow download of Tweet IDs, Direct Message IDs, and/or User IDs.
> You may, however, provide export via non-automated means (e.g., download of spreadsheets or PDF files, or use of a “save as” button) of up to 50,000 public Tweet Objects and/or User Objects per user of your Service, per day.
> Any Content provided to third parties remains subject to this Policy, and those third parties must agree to the Twitter Terms of Service, Privacy Policy, Developer Agreement, and Developer Policy before receiving such downloads.
> You may not distribute more than 1,500,000 Tweet IDs to any entity (inclusive of multiple individual users associated with a single entity) within any given 30 day period, without the express written permission of Twitter.
[0] https://dev.twitter.com/overview/terms/agreement-and-policy
I guess I missed this bit when I reviewed the terms. I suppose I'll have to reach out to them to see if I can get their permission- and thanks!
I'm guessing you might get a cease and desist. Instagram just shutdown Instagress, MassPlanner and others. Granted this isn't a bot but scrapers usually aren't met with much love either. Especially if you start selling the data.
Do the companies have a claim to the actual data if they make it publicly accessible (intentionally)? Curious as to whether there have been any court cases over this.
I'm sure you can't republish it anywhere, but if for example I wanted to scrape some basic demographic data off FB and use it as part of an ML model, would there be something wrong with that?
I'm sure you can't republish it anywhere, but if for example I wanted to scrape some basic demographic data off FB and use it as part of an ML model, would there be something wrong with that?
Just because it's publicly accessible doesn't mean anyone who can access it gets usage rights on it, but some metadata might not qualify for protection. There is also the issue of ToS forbidding it, with varying legal consequences – and this site requires you to log in with Twitter/Facebook, so they have accepted a developer agreement with the service, which might forbid what they are doing.
There are different forms of usage rights, though. Like I said, I'm positive it would not be a good idea to simply republish all of FB/insta/twitter's data elsewhere, but just as some software licenses allow you to use the code however you want so long as you do not redistribute it, I'm wondering if there are legal/ToS policies that allow limited use of the data (or specifically, what the official policies of these companies are w.r.t this).
INAL but usually the data, or at least some portion of it, is owned by the users and the service's TOS and Privacy Policy will state how the service is allowed to use it. So they have a responsibility to keep their users' data protected to the degree specified in those policies. But really, the data is usually the service's biggest asset when it comes to creating an advertising business which is maybe the biggest reason they want to protect it.
But data aside, the company owns the servers you are sending requests to in order to get the data. So that opens another angle for them to litigate. This is where things like robots.txt comes in which I don't think has ever fully played out in court.
Yet another angle to go after folks on is the use of trademarks in any marketing copy. It's hard to sell Instagram bots and scrapers if you can't use the word Instagram.
I don't even really think a lawyer would have a clear cut answer without going to court but the cease and desist is usually enough to scare people into shutting down. MassPlanner was a pretty widely used piece of software so you'd have to imagine it was making at least a decent amount of money. Not crazy venture backed business money but definitely lifestyle business money if not more. And FB owned Instagram seemingly got them to close up shop without going to court. Even more interesting is that MassPlanner wasn't SaaS. It was desktop software. So MassPlanner wasn't even running servers that were accessing any of the social networks.
On the other hand, sites like SocialBlade and tools like Scrapebox have been around for years. So who knows?
Ultimately, if you're just an individual scraping for research purposes or maybe you run a social media consulting firm and you're doing it for a client, if you don't publicize it like crazy then you're fine. But just from watching things like this over the years I've noticed most services like this eventually get shutdown if they get really popular. Scrapers tend to fare better than bots but if you get on their radar and they don't like what you're doing you're up against some deep pockets.
But data aside, the company owns the servers you are sending requests to in order to get the data. So that opens another angle for them to litigate. This is where things like robots.txt comes in which I don't think has ever fully played out in court.
Yet another angle to go after folks on is the use of trademarks in any marketing copy. It's hard to sell Instagram bots and scrapers if you can't use the word Instagram.
I don't even really think a lawyer would have a clear cut answer without going to court but the cease and desist is usually enough to scare people into shutting down. MassPlanner was a pretty widely used piece of software so you'd have to imagine it was making at least a decent amount of money. Not crazy venture backed business money but definitely lifestyle business money if not more. And FB owned Instagram seemingly got them to close up shop without going to court. Even more interesting is that MassPlanner wasn't SaaS. It was desktop software. So MassPlanner wasn't even running servers that were accessing any of the social networks.
On the other hand, sites like SocialBlade and tools like Scrapebox have been around for years. So who knows?
Ultimately, if you're just an individual scraping for research purposes or maybe you run a social media consulting firm and you're doing it for a client, if you don't publicize it like crazy then you're fine. But just from watching things like this over the years I've noticed most services like this eventually get shutdown if they get really popular. Scrapers tend to fare better than bots but if you get on their radar and they don't like what you're doing you're up against some deep pockets.
This is part of the reason I'm not planning on ever providing this as a paid service. Regardless of the ToS, I don't really think it would be right to sell this data in the first place.
For the second time today, I am going to suggest seeking the council of a qualified legal professional. Up above, you admitted you just copied some site's terms of service. Now, I am seeing that you haven't actually checked the various services for their list of rules.
You're quite likely to end up in court, should you actually get noticed. Seriously, hire a lawyer. I am not kidding. These sites kinda value their data and have a variety of limits for using the data - even if it is public facing.
You're quite likely to end up in court, should you actually get noticed. Seriously, hire a lawyer. I am not kidding. These sites kinda value their data and have a variety of limits for using the data - even if it is public facing.
He'd maybe get a cease and desist.. IANAL but he is not 'quite likely to end up in court' unless he fights against them and refuses to shut down / make changes.
[deleted]
I'm having trouble connecting
yeah I'm getting a lot more traffic than intended. I may have to resize the droplet.
Me too. Australia
Hello @hobbescotch, is it possible to describe the techniques used to scrape the data, what technology stack are you using, where are you hosting, what would be the premium features?
Hi riston, the data is just being pulled from the Twitter and Graph APIs. The site itself is built using Flask and written in Python and is hosted using Digital Ocean. I don't plan on having any paid/premium features.
The follower export widget on the homepage has 'What number will be rolled?' as the title text for the user drop-down (`#user_count').
Is that the actual text that is meant to be there or was it mixed up?
Is that the actual text that is meant to be there or was it mixed up?
Hi, what data does this output?
FYI your confirmation email uses https://www.minebaseapp.com (not working) whereas the main site uses https://minebaseapp.com (working). I guess you didn't configure your DNS records to handle the www subdomain because that results in a DNS error.
For anyone who's signing up now or asked to get an email notification when a report is complete, my ESP cut me off saying I'm having an unusual amount of requests. I'm contacting them to get it fixed, but be aware you might not get emails sent right now.
How isn't that breaking Facebook's TOS?
+1, don't want to be exited if it's out in 24 hrs
Has anyone noticed one of the site maintenance template?
https://web.archive.org/web/20170627020124/https://minebasea...
You can find it on
https://gist.github.com/pitch-gist/2999707
https://web.archive.org/web/20170627020124/https://minebasea...
You can find it on
https://gist.github.com/pitch-gist/2999707