Startup Mistakes: Choice of Datastore(stavros.io)
stavros.io
Startup Mistakes: Choice of Datastore
https://www.stavros.io/posts/startup-mistakes-datastore/
118 comments
Just pick something and build an MVP with it. Then get on with the hard part of finding paying customers. You can fix the bad tech decisions later. Without customers it won't matter which database you used before your startup failed.
I would suggest that something be PostgreSQL. With JSONB and and all the other goodies in version 10 they are too numerous to list there is no reason not to use it for 90% of apps. It WILL scale until you need something more specialized.
Postgres is far from a boring relational database. Don’t make the mistake of building a mvp on rickety stilts and then swap them out later for proper columns while trying to run.
Postgres is far from a boring relational database. Don’t make the mistake of building a mvp on rickety stilts and then swap them out later for proper columns while trying to run.
Postgre will be good for 99.99% of all workloads. And when you need something else, you should have enough revenue to migrate the parts that make sense to a different datastore.
BIG ERROR.
Like, MONUMENTAL.
That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later.
Is incredible. Nobody remember that the most cost effective way to fix a problem is in the early stages?
And, yes, the best overall primary datastore, like 90% of the cases, is a RDBMS. Very few actually need to use something else.
Like, MONUMENTAL.
That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later.
Is incredible. Nobody remember that the most cost effective way to fix a problem is in the early stages?
And, yes, the best overall primary datastore, like 90% of the cases, is a RDBMS. Very few actually need to use something else.
Could you please not use uppercase for emphasis? It's basically yelling. Asking not to do that is one of the original HN guidelines: https://news.ycombinator.com/newsguidelines.html. Imagine this comment in all caps and I'm sure you'll understand why.
What's the use of fixing something in a cost effective way if your startup is going to fail anyway?
You need to validate and test first. Get customers ASAP. If that means writing shitty code and using a shitty database or not even a database at all, than that's great. You can always convert to a better system afterwards.
I do agree that the longer you wait, the harder it gets to rewrite code and convert to different systems, but the most important thing for any startup is to get their idea validated as soon as possible.
You need to validate and test first. Get customers ASAP. If that means writing shitty code and using a shitty database or not even a database at all, than that's great. You can always convert to a better system afterwards.
I do agree that the longer you wait, the harder it gets to rewrite code and convert to different systems, but the most important thing for any startup is to get their idea validated as soon as possible.
This is just the balance fallacy. Relational and document databases aren't equally likely to be the type you need. Most of the times you have relational data, so you want relational databases. Pick the datastore that's more likely to fit your data from the start.
Yes and no.
Yes, choose carefully at the beginning, and use relational. More specifically, use Postgres.
But choose the right things to worry about at the beginning. So worry about something that can accommodate changing requirements, i.e., a relational database. Don't worry about scalability. You will be very, very lucky to ever have that problem. Worry about it then.
Yes, choose carefully at the beginning, and use relational. More specifically, use Postgres.
But choose the right things to worry about at the beginning. So worry about something that can accommodate changing requirements, i.e., a relational database. Don't worry about scalability. You will be very, very lucky to ever have that problem. Worry about it then.
Also, if you find that some data is better suited for some other datastore, just move that data off. For example, I start with Postgres but almost always end up storing session data in Redis, for a bit more speed.
>something that can accommodate changing requirements, i.e., a relational database
Wouldn't NoSQL be better suited for this scenario? Genuinely curious.
Wouldn't NoSQL be better suited for this scenario? Genuinely curious.
Postgres is one of the best NoSQL datastores you'll find, until/unless you get to the point where scaling a single Postgres database becomes a problem.
But the point is 1) you're unlikely to ever get there. 2) if you're lucky enough to get there, if you start with your data in a structured database and gradually e.g. move things to JSON blobs in your Postgres DB, then move the things that really, seriously needs it off into a more easily shardable NoSQL datastore, it's a far easier direction to go in than the opposite.
Recovering structure when you've tossed it out the window is a massive pain.
And even a lot of the time when you end up needings things like e.g. ElasticSearch for search or something like that, it may still be better to keep the canonical data store in Postgres and stream changes to a secondary store to scale reads. Or add caching.
I'm not suggesting there are no cases where NoSQL datastores can't be the right choice from the beginning. But if you don't have a very compelling reason to, it's probably not.
But the point is 1) you're unlikely to ever get there. 2) if you're lucky enough to get there, if you start with your data in a structured database and gradually e.g. move things to JSON blobs in your Postgres DB, then move the things that really, seriously needs it off into a more easily shardable NoSQL datastore, it's a far easier direction to go in than the opposite.
Recovering structure when you've tossed it out the window is a massive pain.
And even a lot of the time when you end up needings things like e.g. ElasticSearch for search or something like that, it may still be better to keep the canonical data store in Postgres and stream changes to a secondary store to scale reads. Or add caching.
I'm not suggesting there are no cases where NoSQL datastores can't be the right choice from the beginning. But if you don't have a very compelling reason to, it's probably not.
Exist the myth RDBMS are not flexible (because it have schemas). To the contrary, the relational model is very flexible, and allow to model everything you could want. Even model a NoSql :)
And the Relational model is fairly simple. You could learn anything you need in minutes and just remember in add a few index here and there. For the small amount of things you need to do, is incredible how many features a RDBMS give you for free (like transactions, some basic axioms, and a query engine).
----
I don't remember the original quote, but is alike:
"Novices worry for code, Experts focus on data structures".
Properly chose your data-structures, schemas and data-layout will have a huge net impact in your code. That is why a well modeled database will perform well and allow to easily code against it.
This is the hard part, and most novices like to "defer" it.
In my times, we start designing the app first on the DB layer, including the queries, reports, etc. Now most start with the front-end or back-end... Imaging the are focusing in the abstract logic (because some can't imagine the datastore is part of it!) and ignore or reject the concept of learn what are the datastore capabilities.
That is like ignoring the documentation about arrays, building a layer on top of it, rejecting the idea of use arrays as full, and then wondering why his code performs bad and re-creating, badly, what it already have.
---
For fast prototypes, sqlite (stored in RAM) could be good. In the early stages I erase the DB in each run of the code (after the initial DB design, tweaking it). I continue to do that as far as possible, and only worry about migrations and all that when start shipping to customers.
Also, not be afraid of build several copies of the tables - like experiments- (customerA, customerB, etc), use views and peek on your DB documentation.
And the Relational model is fairly simple. You could learn anything you need in minutes and just remember in add a few index here and there. For the small amount of things you need to do, is incredible how many features a RDBMS give you for free (like transactions, some basic axioms, and a query engine).
----
I don't remember the original quote, but is alike:
"Novices worry for code, Experts focus on data structures".
Properly chose your data-structures, schemas and data-layout will have a huge net impact in your code. That is why a well modeled database will perform well and allow to easily code against it.
This is the hard part, and most novices like to "defer" it.
In my times, we start designing the app first on the DB layer, including the queries, reports, etc. Now most start with the front-end or back-end... Imaging the are focusing in the abstract logic (because some can't imagine the datastore is part of it!) and ignore or reject the concept of learn what are the datastore capabilities.
That is like ignoring the documentation about arrays, building a layer on top of it, rejecting the idea of use arrays as full, and then wondering why his code performs bad and re-creating, badly, what it already have.
---
For fast prototypes, sqlite (stored in RAM) could be good. In the early stages I erase the DB in each run of the code (after the initial DB design, tweaking it). I continue to do that as far as possible, and only worry about migrations and all that when start shipping to customers.
Also, not be afraid of build several copies of the tables - like experiments- (customerA, customerB, etc), use views and peek on your DB documentation.
> I don't remember the original quote, but is alike: "Novices worry for code, Experts focus on data structures".
The origin is Linus Torvalds on the Git mailing list. Here is a copy from LWN: https://lwn.net/Articles/193245/
The quote is ironic considering that Git uses a bespoke and not particularly well-designed key/value database, which has resulted in notorious usability problems in Git.
The origin is Linus Torvalds on the Git mailing list. Here is a copy from LWN: https://lwn.net/Articles/193245/
The quote is ironic considering that Git uses a bespoke and not particularly well-designed key/value database, which has resulted in notorious usability problems in Git.
In general a SQL db will be a better choice for CRUD apps. They get slow if you try to recreate a SQL engine in your DB abstraction layer rather than letting the SQL engine do all the work.
NoSQL db should be used if you are doing something that they are designed for (eg: storing data that can’t have a schema).
NoSQL db should be used if you are doing something that they are designed for (eg: storing data that can’t have a schema).
RMDBs often has great schema migration tools which usually are no problem to use on small datasets.
NoSQL often rely on handling difference in the data model in the application layer, which can become messy and cumbersome if you switch a lot of requirements.
NoSQL often rely on handling difference in the data model in the application layer, which can become messy and cumbersome if you switch a lot of requirements.
Well, they rely on doing pretty much everything in the application layer, which can become messy and cumbersome. Relational databases are the way they are for a reason. NoSQL is often selected, not because they address requirements better, but because developers are not even aware of the problems that relational databases solved years ago.
The point OP is making is that in a startup, you don't do engineering for the sake of engineering. In the end, what matters is generating wealth.
Obviously, you want a sane legacy for the future but unless you already have hit that sweet product-market fit spot it is likely that you will spend a non-trivial amount of time optimizing (prematurely) for something that's in fact orthogonal to what you actually need.
The trick, as usual, is to balance engineering and business depending on which stage your at.
Obviously, you want a sane legacy for the future but unless you already have hit that sweet product-market fit spot it is likely that you will spend a non-trivial amount of time optimizing (prematurely) for something that's in fact orthogonal to what you actually need.
The trick, as usual, is to balance engineering and business depending on which stage your at.
This, I find, is one of the hardest things about hiring developers for a startup. Because on one hand you want people who knows how to do things right.
On the other hand, you want people who are prepared to accept that decisions that are insane in a stable business can be very much rational in a business where the cost of cash to fund large engineering changes is dropping at a crazy rate if you're successful, and doesn't matter if you're not (because you'll never get to it).
Of course it's good to avoid completely pointless mistakes. But it often matters far more to deliver.
On the other hand, you want people who are prepared to accept that decisions that are insane in a stable business can be very much rational in a business where the cost of cash to fund large engineering changes is dropping at a crazy rate if you're successful, and doesn't matter if you're not (because you'll never get to it).
Of course it's good to avoid completely pointless mistakes. But it often matters far more to deliver.
It won't.
That something (if chose very wrong) will totally derail you progress and will cost a lot of fix it later.
A bad tech decision in a problem you can fix later, even if it's an expensive problem[1]. Having no customers isn't a problem you can fix later because your startup will fail and you'll have to close it. Consequently, at the start, you should work on getting paying customers over everything else, including prevaricating about what database to use. Anything is better than not making a decision.
All that said, "just use Postgres" as many people here are suggesting is actually pretty good advice.
[1] Unless it's so bad fixing it will sink your startup, in which case you have to live with it. That does happen but it has to be really bad.
A bad tech decision in a problem you can fix later, even if it's an expensive problem[1]. Having no customers isn't a problem you can fix later because your startup will fail and you'll have to close it. Consequently, at the start, you should work on getting paying customers over everything else, including prevaricating about what database to use. Anything is better than not making a decision.
All that said, "just use Postgres" as many people here are suggesting is actually pretty good advice.
[1] Unless it's so bad fixing it will sink your startup, in which case you have to live with it. That does happen but it has to be really bad.
If your goal is to build a sustainable business you must address two questions: Will I provide value people are willing to pay for? Will I be able to find those people profitably? Both answers must be yes.
You might be able to answer one of the questions without an MVP but very unlikely to answer both without. So you need to ship something.
It would be clearly stupid to not consider your technology choices at all in the early stages. But the best decisions are usually the ones that allow you to determine the answers to both questions as soon as possible. Because if you aren’t able to eventually answer yes to both questions you will not build a sustainable business.
If you are providing value, people are paying you and you can find customers, the technology sins from the past can be addressed. It might be really hard. But at least you’ll be working on something that matters.
Spending a massive amount of time on getting it “just right” will most likely teach you that you’ve developed the perfect solution to a problem no one cares about.
Speaking from experience of being involved with two acquisitions of companies I started (on the selling side) I can tell you that the “technology stack” discussion is a tiny footnote to the discussions on revenue, sales, customer lifetime value, etc.
You might be able to answer one of the questions without an MVP but very unlikely to answer both without. So you need to ship something.
It would be clearly stupid to not consider your technology choices at all in the early stages. But the best decisions are usually the ones that allow you to determine the answers to both questions as soon as possible. Because if you aren’t able to eventually answer yes to both questions you will not build a sustainable business.
If you are providing value, people are paying you and you can find customers, the technology sins from the past can be addressed. It might be really hard. But at least you’ll be working on something that matters.
Spending a massive amount of time on getting it “just right” will most likely teach you that you’ve developed the perfect solution to a problem no one cares about.
Speaking from experience of being involved with two acquisitions of companies I started (on the selling side) I can tell you that the “technology stack” discussion is a tiny footnote to the discussions on revenue, sales, customer lifetime value, etc.
>Spending a massive amount of time on getting it “just right” will most likely teach you that you’ve developed the perfect solution to a problem no one cares about.
And this is the reason a RDBMS is totally better. You NOT need to "spend a massive amount of time". With this:
1- You get a totally proven and mature tech 2- With massive tolling support and documentation 3- With the capabilities to model even most NoSql structures. 4- With a query engine that will outperform most developers. And more flexible that most NoSql give 5- With enough scalability that you will ever need for your MVP and beyond 6- And with the ability to plug specialized storage like column stores, time stores and more without complicating your code duplicating efforts.
---
P.D: I have been part of several projects that drink the NoSql koolaid, and be part of the weeks-long coding efforts that could be summarized as:
"This could have been a one line sql code/or index ..."
My last one, rebuilding core aspects of a RDBMS and still building features... dedicating large part of dev time is really "Spending a massive amount of time on getting it “just right”"
And this is the reason a RDBMS is totally better. You NOT need to "spend a massive amount of time". With this:
1- You get a totally proven and mature tech 2- With massive tolling support and documentation 3- With the capabilities to model even most NoSql structures. 4- With a query engine that will outperform most developers. And more flexible that most NoSql give 5- With enough scalability that you will ever need for your MVP and beyond 6- And with the ability to plug specialized storage like column stores, time stores and more without complicating your code duplicating efforts.
---
P.D: I have been part of several projects that drink the NoSql koolaid, and be part of the weeks-long coding efforts that could be summarized as:
"This could have been a one line sql code/or index ..."
My last one, rebuilding core aspects of a RDBMS and still building features... dedicating large part of dev time is really "Spending a massive amount of time on getting it “just right”"
I was speaking less about the specific debate between NoSQL and RDBMS and more about being cautious to not over-think the technical aspects in general.
Speaking specifically about NoSQL: I’ve been able to launch entire MVPs using DynamoDB, some Lambda functions and HTML / JavaScript in about the same amount of time it would have taken me to plan out, build and refactor a proper schema. And I usually learned through the MVPs that my idea was stupid and no one cared about what I built, much less about my choice of database.
Speaking specifically about NoSQL: I’ve been able to launch entire MVPs using DynamoDB, some Lambda functions and HTML / JavaScript in about the same amount of time it would have taken me to plan out, build and refactor a proper schema. And I usually learned through the MVPs that my idea was stupid and no one cared about what I built, much less about my choice of database.
Also, later on if you decided that a RDBMS doesn't suit your needs it's significantly easier to migrate RDBMS -> NoSQL versus the other way around.
Consider that for a startup money now is worth a lot more than money even 6-12 months from now, unless you're failing anyway.
Let's say the valuation increases 10-fold between the MVP and the next round. Any $1 in investment I can defer from the first round to the second round then costs shareholders a tenth as much dilution. So even if I end up spending far more money, I may come out ahead, as long as the choices won't hold us back until then.
This doesn't take into account that I might not even be able to raise enough for a more expensive solution, so I might not have a choice.
So a lot of the time the right choice is what costs the least amount right now even if you know it will cause costly re-engineering down the road if you're successful.
For a mature business the tradeoffs are different. If you know your growth rates will be modest, then getting it right from the start matters a lot more.
Let's say the valuation increases 10-fold between the MVP and the next round. Any $1 in investment I can defer from the first round to the second round then costs shareholders a tenth as much dilution. So even if I end up spending far more money, I may come out ahead, as long as the choices won't hold us back until then.
This doesn't take into account that I might not even be able to raise enough for a more expensive solution, so I might not have a choice.
So a lot of the time the right choice is what costs the least amount right now even if you know it will cause costly re-engineering down the road if you're successful.
For a mature business the tradeoffs are different. If you know your growth rates will be modest, then getting it right from the start matters a lot more.
Exactly. This article typifies premature optimisation. If MongoDB lets you prototype faster, then use it until you have enough paying customers to warrant a refactor.
I think you're greatly underestimating both the effort of changing a datastore from under a live application and the free time developers in a startup have.
Oh, wait. So you should actually think about what you're trying to build and what would work best for it? I just read this hilarious blog post that said the opposite. This is all too hard, I'll just use Postgres(R).
By "Said the oppposite" do you mean this part?:
> Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me.
> Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me.
[deleted]
Knuth's aphorism on premature optimization is perhaps the most misunderstood point in software development. It applies to code-level algorithmic optimization, not architectural decisions, where a bad choice can lead down a very long dead-end. You can recover if you recognize this in time, but you are better of if you do not have to. Obligatory rewriting is not what you want to be doing just as competitors take notice of your initial success.
> If MongoDB lets you prototype faster
Does it? Even a prototype involves some iteration, which can be derailed by the kind of missing-schema problems that the OP mentions. Throw in a few operational problems, and your prototype isn't really any faster than something with better features and reliability. What's better for that first few lines of code on your laptop might not even get you to a prototype.
Does it? Even a prototype involves some iteration, which can be derailed by the kind of missing-schema problems that the OP mentions. Throw in a few operational problems, and your prototype isn't really any faster than something with better features and reliability. What's better for that first few lines of code on your laptop might not even get you to a prototype.
...as long as that something is Postgres.
Agree! Most of us techies think tech is more important than sales. We also loathe bad design decisions. But your customers (in most cases) don't care!
But I always wonder exactly "what" is easier with mongo or another NoSQL db. Just get a relation db like PostgreSQL, MySQL or SQLite for that matter. If your site starts to get performance problems go to the pub, celebrate a bit of success and then optimise!
But I always wonder exactly "what" is easier with mongo or another NoSQL db. Just get a relation db like PostgreSQL, MySQL or SQLite for that matter. If your site starts to get performance problems go to the pub, celebrate a bit of success and then optimise!
Yup, people stuck on optimising (or overthinking which datastore to use) before actually having a business are missing the point.
Or they miss the point when overthinking the whole code for the program/website. Get it done. Start finding customers.
The worst code I worked with was at travel agency startup and they are REALLY successful now. It was REALLY REALLY bad and hard to maintain. The owner of the business just said: well, bugs happen. But when a customer (not too many of course) encounters one, they probably call the helpdesk. Sounds stupid, worked great, business wise.
4 years later, I am sure they still have problems with the crappy code, but the business is fine!
The worst code I worked with was at travel agency startup and they are REALLY successful now. It was REALLY REALLY bad and hard to maintain. The owner of the business just said: well, bugs happen. But when a customer (not too many of course) encounters one, they probably call the helpdesk. Sounds stupid, worked great, business wise.
4 years later, I am sure they still have problems with the crappy code, but the business is fine!
Agree, the mantra of success is: GET SHIT DONE!
Customers do not care what's your code coverage or how maintainable is your code. Or if it's Mongo or Postgres. They want features, they want it now, and if you can't deliver that because you are too busy refactoring that's bad.
People starting a business need to be able to consider more than one thing at a time.
I don't disagree, but you'd probably rather pick "something" tried, true, and appropriate for most use cases, and that's RDBMS in this case.
you pick "something" based on the requirements of the application or whatever solution you're building. if your requirements mention that you need to have multiple people accessing the database at the same time, RDBMS environments like MySQL, Postgres, Oracle would be appropriate. If it is a single application instance with no multi-user access, SQLite might be appropriate. However, if you're choosing SQLite, FileMaker, FoxPro, MS Access, etc and your requirements specify that you needed multi-user access, then you've made a large mistake.
The reason MongoDB is simultaneously lauded and derided is because there is a bimodal distribution of people using it: those who build databases for a living, and those who query them.
There is something to be said for being able to rapidly prototype ideas, especially if your primary skillset is jockeying JSON in the context of web/app development. However, deciding whether or not NoSQL is the right fit for you or your project/business depends on how much of your time you will be spending getting down and dirty with the database.
There is something to be said for being able to rapidly prototype ideas, especially if your primary skillset is jockeying JSON in the context of web/app development. However, deciding whether or not NoSQL is the right fit for you or your project/business depends on how much of your time you will be spending getting down and dirty with the database.
yes, yes, yes.
So he used a database 7 years ago that was about one year old at the time and had a bad experience with it and now continues to judge it to this day without tracking the evolution since, including schemas in the current release candidate.
No, the article isn't about Mongo. Quite the opposite.
I feel like MongoDB/NoSQL is a horse that's been beaten so much in the last few years that no one is actually making that choice nowadays. Hasn't everyone already learned to stick with Postgres?
Unfortunately, no. Some friends of mine picked another nonrel database (not Mongo) for their new company, which prompted this article. I still have to give this advice quite often.
My condolences.
I think one thing that's not mentioned enough in the SQL vs NoSQL debate is the benefit of powerful storage types. For example, when storing IP addresses in Postgres, you can use the inet datatype and easily query results if they fall within a given cidr range. Example:
I think one thing that's not mentioned enough in the SQL vs NoSQL debate is the benefit of powerful storage types. For example, when storing IP addresses in Postgres, you can use the inet datatype and easily query results if they fall within a given cidr range. Example:
SELECT * FROM audits WHERE ip_address << '10.0.0.0/20'
That's one of the reasons why I love Postgres. Relatedly, people swear by PostGIS.
same thing goes with single page applications. i've haven't seen the benefit yet, a fast full stack app sprinkled with react/vue at the last minute does just as good a job.
Those are more frequently useful, mainly for CRUD apps where you're changing a lot of state often and don't want to have to reload every time. I agree that for news-type sites, for example, they are overkill.
I think that there are various reasons to pick DBs.
Blindly picking a SQL DB (mysql/postgres etc.) is quite expensive from the get-go (a production ready mysql/postgres would cost ~30$/m).
Mongo costs ~$10/m (MongoDB Atlas), Google's Datastore is Pay as you Go (so your initial cost is close to $0 till you get paying customers), AWS's DynamoDB is similarly priced as well.
Sure, sadly all those noSQL solutions get really expensive as your usage goes up to normal non-webscale proportions, but at that point you have the $ to invest in a SQL solution.
The above was mentioned with bootstrapped startups/services in mind. Not your usual million funded valley companies.
Blindly picking a SQL DB (mysql/postgres etc.) is quite expensive from the get-go (a production ready mysql/postgres would cost ~30$/m).
Mongo costs ~$10/m (MongoDB Atlas), Google's Datastore is Pay as you Go (so your initial cost is close to $0 till you get paying customers), AWS's DynamoDB is similarly priced as well.
Sure, sadly all those noSQL solutions get really expensive as your usage goes up to normal non-webscale proportions, but at that point you have the $ to invest in a SQL solution.
The above was mentioned with bootstrapped startups/services in mind. Not your usual million funded valley companies.
$30/month doesn't seem "quite expensive" to me, not if it saves you a single hour of developer time each month. If that is breaking the bank then you're already in trouble. In the case of a company in heavy development mode you can run the database locally or on a cheap $5 VM. Databases are pretty easy to migrate around when they're not in production and can be taken offline.
Hmm, how are you deploying things that a database costs $30/m? I just put it on the same server as the application worker and split it off if I need more performance later.
I'm personally a fan of:
https://www.heroku.com/postgres
Since it's free below 10K rows and only $9 for 10M. And, the dataclips feature always comes in handy.
Since it's free below 10K rows and only $9 for 10M. And, the dataclips feature always comes in handy.
maybe I'm doing this wrong, but if I use RDS / Cloud SQL / Compose etc - a normal base cluster + another failover cluster + hourly/daily backups (based on your release schedule) all add up to around 30 - 50 based on various cloud providers.
Hmm, can't you just install a database on one of the servers and later migrate to one of the managed datastores if you need it? That migration should be pretty simple with minimal downtime.
You can start with one server with postgres installed on it and when you need to scale to multiple application servers, you can run postgres on another server.
For backing up postgres, all you have to do is setup a cron job that backs up the postgres' data directory to S3/Google Drive/Dropbox every hour/day.
If you want proper replication and failover then you can probably use 2 digital ocean droplets each for $5/month and another $5 VM for the application server itself.
For backing up postgres, all you have to do is setup a cron job that backs up the postgres' data directory to S3/Google Drive/Dropbox every hour/day.
If you want proper replication and failover then you can probably use 2 digital ocean droplets each for $5/month and another $5 VM for the application server itself.
MongoDB/NoSQL is deployed and used at large scale by companies like Facebook, Ebay and many others. That's quite far from "no one is actually making that choice nowadays".
Using MongoDB/NoSQL for the parts of your data where it makes sense is one thing. Thinking "I'm starting a new accounting application, I'll just use Mongo as the datastore because it has better support for my stack" is another.
> companies like Facebook, Ebay
They didn't chose NoSQL, they were forced to. I'm fairly convinced they started with relational stores. If a company or product grows to a point where relational data doesn't work, that's a problem you want to have.
The mistake is either thinking you need to design for facebook scale from the beginning OR thinknig that you can cut time in a startup by not having to bother with those pesky schemas that just slow you down.
They didn't chose NoSQL, they were forced to. I'm fairly convinced they started with relational stores. If a company or product grows to a point where relational data doesn't work, that's a problem you want to have.
The mistake is either thinking you need to design for facebook scale from the beginning OR thinknig that you can cut time in a startup by not having to bother with those pesky schemas that just slow you down.
Facebook started with normalized tables, though they altered the schema really frequently. They added more MySQL servers as they grew to more schools. Then their users started graduating and moving around and things got complicated.
The fact that it's used today might just reflect choices made (and possibly regretted) ages ago. It doesn't prove that anyone's making that choice today, or refute a claim to the contrary. Also "MongoDB/NoSQL" is a slippery phrase. There are plenty of people who use other NoSQL databases who would never have touched MongoDB with a twenty-foot pole, so your statement's truth varies according to which part of the subject you look at.
Well, I meant for startups, which I thought was implicit given that the original blog post was about "Startup Mistakes".
Yes, I agree that an key-value store that lets you do embarrassingly parallel reads and writes is useful for scaling, but has't it been said enough that You Are Not Google[1]?
[1] https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
Yes, I agree that an key-value store that lets you do embarrassingly parallel reads and writes is useful for scaling, but has't it been said enough that You Are Not Google[1]?
[1] https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
NoSQL is the most useful when you're dealing with event data which aren't the main part of the application. We use MongoDB for analytics on our API endpoints because storing that kind of data in postgres will add unnecessary bloat to it
Hi Kash. Genuinely curious, any other reason to choose MongoDB for analytics? It seems like a poor choice if you want expressive queries for data science.
Why not just directly `COPY` your log files to something like Redshift?
Why not just directly `COPY` your log files to something like Redshift?
Hey! For convenience, mostly. We used to depend on Mixapanel for analytics and MongoDB was a good drop-in replacement with only 20-30 extra lines of code. I agree that Redshift would be a better choice for complicated queries afterwards but the only queries we run on the event data is looking up 2 events with the same "training_id" and calculate the time difference between their timestamps for billing purposes.
The "important" events for data science are still stored on postgres (i.e. logins for checking if a login was malicious)
The "important" events for data science are still stored on postgres (i.e. logins for checking if a login was malicious)
...and large companies still use stack ranking to judge whom of their employees to keep and which to fire. Just 'cause someone very big is doing it, doesn't mean it's right.
Unfortunately, the horse is not quite dead enough
If anyone remembers the most prominent and convincing articles on this topic, I would love to share that with my team
If anyone remembers the most prominent and convincing articles on this topic, I would love to share that with my team
FWIW, my personal anecdote: I chose Google App Engine for a large enterprise application almost a decade ago and it's been absolutely, undeniably one of the best choices we made.
The application is locked into Google, but that hasn't proven a problem yet and can be designed around if need be.
The application is locked into Google, but that hasn't proven a problem yet and can be designed around if need be.
My personal anecdote is the exact opposite of yours. I picked GAE for one of my personal projects years ago and it has been terrible. You can't do certain kinds of lookups unless you build the indexes first, there are no delete cascades, no intra-table constraints, etc.
I suspect the difference between us is that you spent the time working around these problems, whereas I expected it to just work.
I suspect the difference between us is that you spent the time working around these problems, whereas I expected it to just work.
This article is a bit combative but I generally agree with the idea.
I use both MySQL and MongoDB in my daily work on a classifieds site that does a few hundred million pageviews a month. Both are pretty solid performers. The article is correct that with Mongo you just move the schema into the code (new versions not withstanding). I think it's nicer to have the schema on the database side but it's really just user preference. We typically end up creating a schema class and defining it up-front anyway. There is also a small subset of cases where not having a schema at all is actually a benefit.
Starting with a popular SQL engine is a really good tried and tested method though.
I use both MySQL and MongoDB in my daily work on a classifieds site that does a few hundred million pageviews a month. Both are pretty solid performers. The article is correct that with Mongo you just move the schema into the code (new versions not withstanding). I think it's nicer to have the schema on the database side but it's really just user preference. We typically end up creating a schema class and defining it up-front anyway. There is also a small subset of cases where not having a schema at all is actually a benefit.
Starting with a popular SQL engine is a really good tried and tested method though.
It's more than user preference in the long run; When your schema is encoded in the database (with triggers, constraints, foreign keys, the whole shebang), you can be sure that whichever way you access the database, it is still consistent.
When your schema is "in the code", even if you completely abstract it into a library, it means that the quick-one-off Ruby/Perl/C#/Python script will not have the integrity checks, and may corrupt your DB.
When your schema is "in the code", even if you completely abstract it into a library, it means that the quick-one-off Ruby/Perl/C#/Python script will not have the integrity checks, and may corrupt your DB.
To the people saying "just do Postgres", what would you say to a startup that wants to create offline first apps? CouchDB, for example is way better at that than Postgres.
1) Use CouchDB and Postgres?
2) Somehow implement revisions and Postgres?
3) Use Postgres anyway and scrap the offline first?
1) Use CouchDB and Postgres?
2) Somehow implement revisions and Postgres?
3) Use Postgres anyway and scrap the offline first?
If you want to create offline-first apps, I would use Postgres as the main datastore and use Couch to sync data between client and server. You'd have to decide which of your data would live where (or if you wanted to use Couch as a way to transfer data from the server to the client).
Couch is a very good datastore for that use case, though, so I would definitely use it in some capacity for your purpose.
Couch is a very good datastore for that use case, though, so I would definitely use it in some capacity for your purpose.
Can you expand on this? Do you mean you'd use Postgres as your main store and periodically "flush" data from CouchDB to Postgres (probably as JSON)?
I don't see how that would work reliably. On the client would you source CouchDB or Postgres? Presumably you'd access CouchDB directly, but then why even use Postgres (for that subset of data, anyway).
I don't see how that would work reliably. On the client would you source CouchDB or Postgres? Presumably you'd access CouchDB directly, but then why even use Postgres (for that subset of data, anyway).
It really depends on what your data looks like. If it's just game settings and state, put them in Couch and that's it, and keep user data like payments and activity there.
If it's data you're going to want to run analytics on and sync to the client, you're probably going to have to store it in both places, I think.
If it's data you're going to want to run analytics on and sync to the client, you're probably going to have to store it in both places, I think.
I think the real mistake is one of:
1. I'm gonna use NoSQL because I've heard its really cool.
2. I'm not going to use SQL because its hard for me to write queries and change stuff on the fly.
If either of those are true (gotta be honest with yourself when deciding) then STOP. You have to think about the best tool for the job overall - sometimes that will be NoSQL, sometimes it will be an RDBMS. If you can't decide which of the two, then Postgres with its JSON support is IMO the best starting point.
If either of those are true (gotta be honest with yourself when deciding) then STOP. You have to think about the best tool for the job overall - sometimes that will be NoSQL, sometimes it will be an RDBMS. If you can't decide which of the two, then Postgres with its JSON support is IMO the best starting point.
OP forgot the biggest mistake for choice of datastore: Blockchain.
@StavrosK: (aka the author) - what is the TLDR of the article? Ask since there appear to be a number of users including myself that appear to not get the intent of your article.
—
Meta-comment: Feel like if the poster is self-identifying as the author when posting the link, it’s verified via say email/domain, an HN username has been ID’d in the past as the author, etc. — it should be automatically obvious in post, comments, etc.
—
Meta-comment: Feel like if the poster is self-identifying as the author when posting the link, it’s verified via say email/domain, an HN username has been ID’d in the past as the author, etc. — it should be automatically obvious in post, comments, etc.
It's what I explicitly call out as a TLDR in the article itself: "Think before you pick a database. If you insist on not thinking, pick PostgreSQL. Trust me."
Basically, "Postgres is a better default".
Basically, "Postgres is a better default".
There is no TLDR, summary, etc. clearly marked as such; by one estimate it takes 140 seconds or 468 words to get to the start of the sentence you quoted above; which is strange given your point is to stick to best practices unless there’s a strong reason not to do so. Strongly recommend moving that info to the top of your post.
1) Read the image captions - they made my day.
2) Pick Postgres unless you have a compelling reason not to.
2b) Use the time gained by not over-engineering upfront, to focus on users and business logic.
3) Optimize/Evolve away from Postgres later as needed.
4) Profit.
If your datastore has no schema your application must be the schema, you have to version your models and have those migrate when old data is accessed.
Wow. This fell off the front page in the time it took for me to drive to work. Guess the NoSQL crowd ain't got time for that.
While I'm generally sympathetic to your post, there are some things that are red flags.
>If you have ten services accessing the same database and sharing data between themselves
Whoever access the database schema owns it. If you have ten systems accessing your database then ten teams own it. And if ten teams own it then nobody owns it. Nobody can change it. Seen this at a successful start-up that got big, but then couldn't rev order management, because every team had their finger in the pie, we couldn't do a schema update without breaking everyone, and of course one team was "under tremendous pressure to hit a major milestone and we just can't do that now" for over a year.
Now I would turn this around into a win for RDBMS by suggesting the use of functions or stored procedures: with an RDBMS we can construct an API, and then we can version those APIs. And then the team that owns the database can do what they like. That said, we can do the same with NoSQL databases by not allowing other teams to access them. The team that owns the NoSQL database is required to maintain an API for it.
I've only ever had nightmares with other teams coding against my schema. GraphQL worries me in that respect and I'd love to hear how people here have fared with long lasting GraphQL, in the real world.
>Django, for example, makes migrations trivial, as you just change your application-level classes and the database gets migrated automatically.
I've had automated migration systems grind to a halt and leave the DB fucked too.
>Priscilla used a graph database when her data was relational. Her husband, furious, filed for divorce.
There are no schema that are "relational" but not "a graph". However there are plenty of schema where a graph database is a natural fit but that require either one-table-per-node-type or building a graph model on top of your RDBMS (e.g. an Entity-Attribute schema). Oy.
There are also many schema where there is only one entity type, but every join is against itself, and we're looking to join all the way out to the clique. In this case would you suggest an RDBMS, and then put the iteration in the application? You suggested earlier that making up for the inadequacies of the datastore in the application is a bad idea.
I've got a graph application that uses NoSQL and it was the right call. An RDBMS would have allowed us to write something that worked for simple cases, but that would have brought the system to its knees based on some customer usage. The solution for the RDBMS would be the same as what we had to do for NoSQL. But up to that moment, the NoSQL allowed us to iterate far faster than an RDBMS.
>For example, if you later need to compile a list of all the brands of all the products on your store, an RDBMS can easily do that by reading the “brands” table
Only if you built a brands table. You can't argue that we can't predict the future, so use an RDBS because its easy to change, but then make arguments that require that the builder accurately predicted the future and built the schema with that foresight. Sure, we could go and pull a brands table out of the existing tables but thats work, and it might be work on a live database that brings it down.
A graph database would be just as likely to have 10 brand nodes since the overhead of creating the first such node is far lower than creating an entire table and updating the schema.
>Relational databases excel at easily providing answers to questions that weren’t predicted at the time when the data model was designed.
Or a NoSQL database with spark. "But spark is something new to learn"
And this is the biggest flaw in your argument. You're pro RDBMS because you know SQL and how to run RDBMS, create the schema, and write the queries. It is incredibly easy to get started with MongoDB. That right there is why it is popular. Not because its good. But when you say "Just get something started and use an RDBMS", you're actually saying "I know you know javascript, but I need you to learn Modula-2 for this part of the system". Fundamentally different syntax and strict types (or "schema").
>For all its unparallelizability, ACID is pretty damn nice.
Until someone holds transactions open across network calls and kills throughput. I've seen that issue lose a company a multi-million dollar contract because of contention on a single row. Or until someone chooses the wrong isolation level ("But I used a transaction!") and two transactions happily decrement non-atomically. This shit be hard, and part of the "hard" is not knowing you're doing it wrong.
>You’ll have plenty of time figuring out what to use when you know your exact usage patterns, if the business manages to not die until then.
So start with something quick and easy then. You've already managed to describe two scenarios where an RDBMS blew up in practice (multiple teams hitting a DB causing schema lock; its a network, not a flat hierarchy).
If you have an experienced SQL team, by all means go with RDBMS. But lets not pretend they are a panacea. Honestly, I'd use a graph database pretty much all the time - if I could only trust them. But its the quality, reliability and longevity that I have a problem with, not with the nature of how the database organizes data.
While I'm generally sympathetic to your post, there are some things that are red flags.
>If you have ten services accessing the same database and sharing data between themselves
Whoever access the database schema owns it. If you have ten systems accessing your database then ten teams own it. And if ten teams own it then nobody owns it. Nobody can change it. Seen this at a successful start-up that got big, but then couldn't rev order management, because every team had their finger in the pie, we couldn't do a schema update without breaking everyone, and of course one team was "under tremendous pressure to hit a major milestone and we just can't do that now" for over a year.
Now I would turn this around into a win for RDBMS by suggesting the use of functions or stored procedures: with an RDBMS we can construct an API, and then we can version those APIs. And then the team that owns the database can do what they like. That said, we can do the same with NoSQL databases by not allowing other teams to access them. The team that owns the NoSQL database is required to maintain an API for it.
I've only ever had nightmares with other teams coding against my schema. GraphQL worries me in that respect and I'd love to hear how people here have fared with long lasting GraphQL, in the real world.
>Django, for example, makes migrations trivial, as you just change your application-level classes and the database gets migrated automatically.
I've had automated migration systems grind to a halt and leave the DB fucked too.
>Priscilla used a graph database when her data was relational. Her husband, furious, filed for divorce.
There are no schema that are "relational" but not "a graph". However there are plenty of schema where a graph database is a natural fit but that require either one-table-per-node-type or building a graph model on top of your RDBMS (e.g. an Entity-Attribute schema). Oy.
There are also many schema where there is only one entity type, but every join is against itself, and we're looking to join all the way out to the clique. In this case would you suggest an RDBMS, and then put the iteration in the application? You suggested earlier that making up for the inadequacies of the datastore in the application is a bad idea.
I've got a graph application that uses NoSQL and it was the right call. An RDBMS would have allowed us to write something that worked for simple cases, but that would have brought the system to its knees based on some customer usage. The solution for the RDBMS would be the same as what we had to do for NoSQL. But up to that moment, the NoSQL allowed us to iterate far faster than an RDBMS.
>For example, if you later need to compile a list of all the brands of all the products on your store, an RDBMS can easily do that by reading the “brands” table
Only if you built a brands table. You can't argue that we can't predict the future, so use an RDBS because its easy to change, but then make arguments that require that the builder accurately predicted the future and built the schema with that foresight. Sure, we could go and pull a brands table out of the existing tables but thats work, and it might be work on a live database that brings it down.
A graph database would be just as likely to have 10 brand nodes since the overhead of creating the first such node is far lower than creating an entire table and updating the schema.
>Relational databases excel at easily providing answers to questions that weren’t predicted at the time when the data model was designed.
Or a NoSQL database with spark. "But spark is something new to learn"
And this is the biggest flaw in your argument. You're pro RDBMS because you know SQL and how to run RDBMS, create the schema, and write the queries. It is incredibly easy to get started with MongoDB. That right there is why it is popular. Not because its good. But when you say "Just get something started and use an RDBMS", you're actually saying "I know you know javascript, but I need you to learn Modula-2 for this part of the system". Fundamentally different syntax and strict types (or "schema").
>For all its unparallelizability, ACID is pretty damn nice.
Until someone holds transactions open across network calls and kills throughput. I've seen that issue lose a company a multi-million dollar contract because of contention on a single row. Or until someone chooses the wrong isolation level ("But I used a transaction!") and two transactions happily decrement non-atomically. This shit be hard, and part of the "hard" is not knowing you're doing it wrong.
>You’ll have plenty of time figuring out what to use when you know your exact usage patterns, if the business manages to not die until then.
So start with something quick and easy then. You've already managed to describe two scenarios where an RDBMS blew up in practice (multiple teams hitting a DB causing schema lock; its a network, not a flat hierarchy).
If you have an experienced SQL team, by all means go with RDBMS. But lets not pretend they are a panacea. Honestly, I'd use a graph database pretty much all the time - if I could only trust them. But its the quality, reliability and longevity that I have a problem with, not with the nature of how the database organizes data.
TL;DR: "just use Postgres"
TL;DR: "just use an MySQL or Postgres or SQLite"
Don't use SQLite for production, for all the love I have for it, the client libraries are usually locking accesses and don't work properly with concurrent reads/writes.
It's designed for single client, and works very well in production for single client. Mobile apps, desktop apps, and anywhere you can serialize access it works great.
Let's say "don't use it for production for a multiuser server app" then I agree. But that isn't really a supported scenario at all.
Let's say "don't use it for production for a multiuser server app" then I agree. But that isn't really a supported scenario at all.
But in all those scenarios the downsides of just writing JSON to disk or something are smaller too.
Concurrent writes block each other, but following one-time "pragma journal=wal", read and writes can work concurrently, and you can expect x2-x5 performance (or ~2% degradation, depending on use pattern, but most people see x2 performance).
The cost is that access through network e.g. NFS or SMB in wal mode is impossible (but you shouldn't have done that anyway), and that you can't just ship the sqlite file - ship a dump/backup instead, or you'll have to do recovery on the wal file you sent.
Of course, it's a good idea to use pgsql from the get-go; but SQLite deserves more credit than it gets, and is much more capable than it is usually assumed to be.
The cost is that access through network e.g. NFS or SMB in wal mode is impossible (but you shouldn't have done that anyway), and that you can't just ship the sqlite file - ship a dump/backup instead, or you'll have to do recovery on the wal file you sent.
Of course, it's a good idea to use pgsql from the get-go; but SQLite deserves more credit than it gets, and is much more capable than it is usually assumed to be.
Agreed, SQLite is a fantastic piece of software. Thanks for the WAL tip, I didn't know it wasn't enabled by default.
If your site is light on writes and always will be, I think SQLite is a good choice. Especially if it reduces the complexity of the system up front. I use it for a lot of personal projects that never gain more than a few thousand impressions a month.
SQLite is also not too difficult to switch to a more advanced SQL in the future.
SQLite is also not too difficult to switch to a more advanced SQL in the future.
I agree, I run a production (very write-light) site on SQLite and it has been great, but the site is pretty much almost static.
have you ever had an issue with that?
There are only 7 billion people, most of whom don't need your database updated faster than their ping time...
seems like a non-issue.
There are only 7 billion people, most of whom don't need your database updated faster than their ping time...
seems like a non-issue.
It's an issue when two people try to write something at the same time and one of them gets a crash because acquiring the lock failed.
So that's an actual issue for you? This happened?
Can't you just idle in a loop waiting for the lock, perhaps waiting a random number of milliseconds (200-800)?
(So that rather than crash, the client just "hangs" waiting for the lock, in a busy loop.)
It just seems like this should not be an issue.
Can't you just idle in a loop waiting for the lock, perhaps waiting a random number of milliseconds (200-800)?
(So that rather than crash, the client just "hangs" waiting for the lock, in a busy loop.)
It just seems like this should not be an issue.
Yes, it happened. I could do lots of things, and "apt-get install postgres" is the simplest
That's fair. As an aside, while it solved your crash issue and obviously scales, when you switched did you get an immediate measurable (and noticeable) performance hit due to the overhead of a proper concurrent database?
No, no noticeable hit. SQLite is a proper concurrent database as well, so I don't think I would have.
I don't mean to be dense but why did you just say "SQLite is a proper concurrent database as well" after sharing a story to the contrary and advising "Don't use SQLite for production".
I'm just trying to understand your advice and learn from your experience, I am just confused by this followup.
I'm just trying to understand your advice and learn from your experience, I am just confused by this followup.
It's not an issue with SQLite itself, the python bindings don't work for concurrent access (and I think some other languages' bindings as well). If you're going to embed SQLite yourself, I think it works fine (I haven't tried it).
In theory, SQLite works quite well with concurrent accesses, I have just never tried it because the libraries in the languages I've used didn't work well for that.
In theory, SQLite works quite well with concurrent accesses, I have just never tried it because the libraries in the languages I've used didn't work well for that.
ohhhhh, I get it. Yes, you mentioned this in your original comment ("the client libraries are usually locking accesses") but I guess I read too quickly.
Though unless you did a truly comprehensive shootout it might be fairer to write: "Be careful using sqlite in production, the client libraries I tried in Python did not handle concurrent reads/writes, whereas clients for postgresql handle it just fine."
Anyway, thanks for the clarification!
Though unless you did a truly comprehensive shootout it might be fairer to write: "Be careful using sqlite in production, the client libraries I tried in Python did not handle concurrent reads/writes, whereas clients for postgresql handle it just fine."
Anyway, thanks for the clarification!
From the article
> I hope I have helped you choose PostgreSQL for your next business
> I hope I have helped you choose PostgreSQL for your next business
TL;DR: Trust me. I should know.
After doing "big data" for the past 5+ years, I always tell people: use postgres, unless you have a proven, compelling reason not to. It disappoints almost everyone that asks, but it's the truth.
> startup mistakes
> .io domain
oh my.
> .io domain
oh my.
Just because the post belongs to an .io domain doesn't mean you are allowed to automatically negate what the author may be trying to say.
What's the problem with a .io domain? Honest question.
There was a recent failure that affected everyone using the TLD and a popular blog post explaining it and suggesting that you use something more reliable. Some of the comments on the article pointed out that the same company runs .org and a handful of other reliable TLD's.
Thanks very much