RethinkDB: why we failed (2017)(defmacro.org)
defmacro.org
RethinkDB: why we failed (2017)
https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.html
236 comments
Pretty much the entire sector is bad.
In general, it's much better to be in a market where you customers cannot do you your job, not just one where they don't want to do your job. In the latter markets, they start asking "Why should I pay you to do something I can do myself?", or just start doing it themselves, and that holds down the price you can charge.
This applies to a lot of familiar service businesses as well: housekeepers, gardeners, delivery/taxi/rideshare drivers, childcare, etc. Think of how much a doctor gets paid vs. a nurse vs. a home healthcare worker. The difference is all in skills that the provider can do but the customer cannot.
In general, it's much better to be in a market where you customers cannot do you your job, not just one where they don't want to do your job. In the latter markets, they start asking "Why should I pay you to do something I can do myself?", or just start doing it themselves, and that holds down the price you can charge.
This applies to a lot of familiar service businesses as well: housekeepers, gardeners, delivery/taxi/rideshare drivers, childcare, etc. Think of how much a doctor gets paid vs. a nurse vs. a home healthcare worker. The difference is all in skills that the provider can do but the customer cannot.
But most companies could do things their suppliers do, because companies always have the option of hiring people with those skills and developing those competencies in house. E.g. Google moved from running their software in commercial datacenters to building their own, or how Amazon has progressively taken over more and more of the supply chain. Android is sold to firms that can build their own mobile operating systems and have done so in the past, yet it's still profitable for Google to do it for them.
Fundamentally, business is about specialisation. Your customers always have the option of becoming jacks-of-all-trades but generally recognise it's best not to. Obviously if the value you're selling is so thin that your customers regularly and genuinely feel they could roll their own without incurring much cost, that's no good, but that's going to typically be the case in the early days before you had a chance to build up much value.
Fundamentally, business is about specialisation. Your customers always have the option of becoming jacks-of-all-trades but generally recognise it's best not to. Obviously if the value you're selling is so thin that your customers regularly and genuinely feel they could roll their own without incurring much cost, that's no good, but that's going to typically be the case in the early days before you had a chance to build up much value.
That's an option in the same way that anyone can go back to college, specialize in pre-med, go to med school, complete residency, and then become a doctor. It's possible in theory. In practice, you're looking at a decade+ of your life, with significant risk that each step fails.
So it is with companies bringing core competencies in-house. Each step takes time, and is fraught with risk. It's not a matter of simply hiring people with the right skillsets. You need to know what the right skillsets are, and how to judge them. You need to hire managers and executives to oversee them. You need to convince all of the above people that they should work for this new company that's just entering the market and may reconsider in a couple years, vs. stick with their bread and butter that's been doing it for decades. Company leadership needs to prioritize integrating the new competency, which often means challenging a lot of their assumptions about how the business works. Investors need to be on board: integrating your suppliers often does bad things to your margins and costs. Tooling and capital investments need to be made, and they can take years to plan out and execute.
With developer tools specifically, the alternative to buying the product is often "We'll have a team of engineers spend a quarter or two building something that is specific to our needs", and no changes need to be made at the management/executive/personnel/financial levels. If, say, you're building a SaaS for a non-tech business, the alternative is "We need to learn how to manage software development processes, and hire a bunch of engineers, and get executive buy-in, and re-build the communication pathways in our org so this new division knows what to build." A lot of businesses actually try this, but the results have been predictably bad, and that creates a ready market for non-tech SaaS vendors.
So it is with companies bringing core competencies in-house. Each step takes time, and is fraught with risk. It's not a matter of simply hiring people with the right skillsets. You need to know what the right skillsets are, and how to judge them. You need to hire managers and executives to oversee them. You need to convince all of the above people that they should work for this new company that's just entering the market and may reconsider in a couple years, vs. stick with their bread and butter that's been doing it for decades. Company leadership needs to prioritize integrating the new competency, which often means challenging a lot of their assumptions about how the business works. Investors need to be on board: integrating your suppliers often does bad things to your margins and costs. Tooling and capital investments need to be made, and they can take years to plan out and execute.
With developer tools specifically, the alternative to buying the product is often "We'll have a team of engineers spend a quarter or two building something that is specific to our needs", and no changes need to be made at the management/executive/personnel/financial levels. If, say, you're building a SaaS for a non-tech business, the alternative is "We need to learn how to manage software development processes, and hire a bunch of engineers, and get executive buy-in, and re-build the communication pathways in our org so this new division knows what to build." A lot of businesses actually try this, but the results have been predictably bad, and that creates a ready market for non-tech SaaS vendors.
I think you may be over-exaggerating the difficulty of doing this. Companies both insource and outsource things pretty frequently. For example it's common for tech firms to insource catering. Building restaurants is not a core competency, yet they are perfectly able to do it and haven't suffered any great distraction from doing so.
In practice one company acquiring another, or even just poaching some employees, is a common thing that happens so frequently we take it for granted. When Apple acquired PA Semi nobody batted an eyelid even though they were in-sourcing something as complex and difficult as CPU design. And there are many examples outside of the computing industry too.
With developer tools specifically, the alternative to buying the product is often "We'll have a team of engineers spend a quarter or two building something that is specific to our needs"
If it takes a small team of engineers only six months to roll their own version of your product, that's what I meant by the value proposition being quite thin (or your price being too high). Also, most companies outside of very rich tech firms will not actually staff up an entire project just to clone a tool they could acquire off the shelf, unless that really is the only reasonable path forward e.g. their needs genuinely are unique.
Now, in computing we do have the issue that the prevalence of VC money, and indifference of VCs to whether it's being well spent, has created a large population of "very rich tech firms" with lots of apparently bored engineers working at them. So they spend a lot of time churning out open source frameworks that they maintain for a few years and then abandon, even when they could have used something else. That's rather unique to software but I don't think it's something inherent to developers as a type of person. Rather, it's inherent to a market where money is free and firms compare themselves to each other by the sheer size of their engineering teams. Outside of the Valley that problem mostly goes away.
In practice one company acquiring another, or even just poaching some employees, is a common thing that happens so frequently we take it for granted. When Apple acquired PA Semi nobody batted an eyelid even though they were in-sourcing something as complex and difficult as CPU design. And there are many examples outside of the computing industry too.
With developer tools specifically, the alternative to buying the product is often "We'll have a team of engineers spend a quarter or two building something that is specific to our needs"
If it takes a small team of engineers only six months to roll their own version of your product, that's what I meant by the value proposition being quite thin (or your price being too high). Also, most companies outside of very rich tech firms will not actually staff up an entire project just to clone a tool they could acquire off the shelf, unless that really is the only reasonable path forward e.g. their needs genuinely are unique.
Now, in computing we do have the issue that the prevalence of VC money, and indifference of VCs to whether it's being well spent, has created a large population of "very rich tech firms" with lots of apparently bored engineers working at them. So they spend a lot of time churning out open source frameworks that they maintain for a few years and then abandon, even when they could have used something else. That's rather unique to software but I don't think it's something inherent to developers as a type of person. Rather, it's inherent to a market where money is free and firms compare themselves to each other by the sheer size of their engineering teams. Outside of the Valley that problem mostly goes away.
> That's an option in the same way that anyone can go back to college, specialize in pre-med
[ ... ]
Another analogy: Tesla's customers get the idea to just design and build their own cars instead.
Thanks for an insightful reply -- slightly helps me when thinking about my own projects
Another analogy: Tesla's customers get the idea to just design and build their own cars instead.
Thanks for an insightful reply -- slightly helps me when thinking about my own projects
[deleted]
Valuable insightful comment - thanks!!
Creating a database startup is hard. Source: Friends who created database startups.
Databases are particularly hard because it’s the bedrock of your persistent state. Trust grows like compound interest and it’s very difficult for a new player to present enough value to compete at a non-zero price point.
Think about the developer tools and systems you use, then think "what would I expect to pay for these?" Chances are the answer is that you'd expect them to be free or you'd find something else. That's what makes it a bad market.
I think jetbrains, github, atlassian et al would like a word with you.
Those are the exceptions that prove the rule. It's not impossible to succeed in developer tools; it's just a lot harder than other markets. And you usually also need an accidental luck factor like being the perfect product at the perfect time.
More like all dev tools that I can buy as an individual are easy to pay for.
Things like a database would have to go through procurement if I wanted to use them and they weren’t free, and I avoid procurement like the plague.
Why the hell wants to spend their time justifying why something is the best solution to people that don’t have a clue.
I’d only go through that if the difference in quality was so palpable that I basically had no choice.
Things like a database would have to go through procurement if I wanted to use them and they weren’t free, and I avoid procurement like the plague.
Why the hell wants to spend their time justifying why something is the best solution to people that don’t have a clue.
I’d only go through that if the difference in quality was so palpable that I basically had no choice.
I have a whole rant about procurement -- don't get me started. The basic idea is that procurement should exist to enable mission performers to do their jobs with less friction, while ensuring legal compliance and minimizing cost to the company. But in fact procurement organizations often seem to think they exist solely to impede mission performers. This is not really their fault; it's just how their incentives are structured.
Just some other anecdotal thing, but if you have someone from procurement worth their salt they will make sure to get the licenses at good terms without tying the company to lock-in, increasing prices YoY etc. Of course, if there is a free alternative, all the better, but if you need to buy something it is definitely worth being on their good side.
Those are outliers, and even then most people I know with a GitHub account are using it for free. Same with IntelliJ - most of us at Google were using the Community Edition.
Andy Gocke put it on twitter better than I've ever seen it anywhere else:
'Developer tools seemed like a good industry to be in, "sell shovels in the gold rush" and all, but it turns out developers prefer to dig for gold with their teeth.'
https://twitter.com/andygocke/status/1017509689695715328?lan...
'Developer tools seemed like a good industry to be in, "sell shovels in the gold rush" and all, but it turns out developers prefer to dig for gold with their teeth.'
https://twitter.com/andygocke/status/1017509689695715328?lan...
I chuckled a bit, but you and I know that devs are not savages :)
I'd probably word it differently:
Developer tools seemed like a good industry to be in, "sell shovels in the gold rush" and all, but it turns out developers prefer to make their own shovels to dig for gold (because it is often cheaper than buying).
I'd probably word it differently:
Developer tools seemed like a good industry to be in, "sell shovels in the gold rush" and all, but it turns out developers prefer to make their own shovels to dig for gold (because it is often cheaper than buying).
Based on the tools that I've often seen developers build in-house, "dig for gold with their teeth" is fair, much of the time.
Unless one works for boring corporations, it might not work for glam blog posts, fame and fortune, but I get to keep my teeth.
I mean, as a hobbyist, and for side projects, I would use my teeth.
But for the company I work for, I don't care. I would recommend my employer the fastest and easiest tool. And my employer usually don't want to spend engineering time on reinventing wheels.
If I were to launch a business in this market, the targeted customers can't be individual developers, more likely other businesses.
Like gihub, individual developers who only use their free account are part of github's marketing team.
But for the company I work for, I don't care. I would recommend my employer the fastest and easiest tool. And my employer usually don't want to spend engineering time on reinventing wheels.
If I were to launch a business in this market, the targeted customers can't be individual developers, more likely other businesses.
Like gihub, individual developers who only use their free account are part of github's marketing team.
> I think jetbrains, github, atlassian et al would like a word with you.
Yeah, but those are outliers drowning in a sea of free development tools. The price that they can charge is effectively limited by how much it would cost some to replicate the subset of that tool that they actually use.
A business that is paying $X/year for $DEVSOFTWARE for their entire devteam pays that only while $X is less than the cost of paying for developing the subset of $DEVSOFTWARE that they use. That limits what Atlassian can charge.
Compare to selling a tool to accountants - it doesn't matter how much $ACCSOFTWARE costs, the accounting department will only shop around, they will never just write their own (outliers excepted, of course).
Yeah, but those are outliers drowning in a sea of free development tools. The price that they can charge is effectively limited by how much it would cost some to replicate the subset of that tool that they actually use.
A business that is paying $X/year for $DEVSOFTWARE for their entire devteam pays that only while $X is less than the cost of paying for developing the subset of $DEVSOFTWARE that they use. That limits what Atlassian can charge.
Compare to selling a tool to accountants - it doesn't matter how much $ACCSOFTWARE costs, the accounting department will only shop around, they will never just write their own (outliers excepted, of course).
Do you know any company that seriously wrote their own development IDE or project management solution comparable to Jira for internal use? That sounds like absolute madness to me.
I agree though that you should never sell to developers, most of them really don't care how they spend their time as long as they are occupied and work on something they deem interesting, so they will happily spend months or years reinventing existing solutions if given the opportunity (I've seen this again and again). You need to sell to the people that run the company and pay the developers or to the engineering management, because they know how much their developers cost, and they understand ROI better than they do. That said often developers are quite creative when making up reasons why not to use an external solution or tool and instead write it themselves, so whether you succeed in selling to a company depends on who's ultimately in charge of technology.
I agree though that you should never sell to developers, most of them really don't care how they spend their time as long as they are occupied and work on something they deem interesting, so they will happily spend months or years reinventing existing solutions if given the opportunity (I've seen this again and again). You need to sell to the people that run the company and pay the developers or to the engineering management, because they know how much their developers cost, and they understand ROI better than they do. That said often developers are quite creative when making up reasons why not to use an external solution or tool and instead write it themselves, so whether you succeed in selling to a company depends on who's ultimately in charge of technology.
> Do you know any company that seriously wrote their own development IDE or project management solution comparable to Jira for internal use? That sounds like absolute madness to me.
I worked for one and you're right, it was madness. They still did it though.
I worked for one and you're right, it was madness. They still did it though.
The alternative to IntelliJ is not rolling your own IDE, but contributing to an open source IDE like Eclipse to get the same or similar features.
The problem is that this market is also one that is familiar to every other engineer. It's is a terrible one to get into because developers hate paying for their tools.
With a modest pricing, I pay for stuff like JetBrains (even the full package), Office 365, Photoshop and TablePlus as subscriptions and some other one time payments that boosted my productivity.
Stuff I don't pay is stuff I feel I can do better if I had some free time and rather gets in the way than properly solve problems and stuff that tries to charge more than what they're worth which can be said for many online developer services.
Stuff I don't pay is stuff I feel I can do better if I had some free time and rather gets in the way than properly solve problems and stuff that tries to charge more than what they're worth which can be said for many online developer services.
Did RethinkDB ever actually attempt to get people to pay for it? The massive, gaping hole at the heart of this analysis is "maybe we failed because we gave our product away for free as open source"?
Yes, my company paid for RethinkDB over a year before they shut down. They sold support to us for $100, and of course their engineers did far more than $100 of bug fixing for us. The reality was that RethinkDB was a very difficult product to create and when you pushed it hard in production it had some serious bugs. The other core issue is that pretty much the same unique problem RethinkDB solved for us (data storage and changefeeds) could be much better solved using a different architecture and better database (PostgreSQL with LISTEN/NOTIFY). https://blog.cocalc.com/2017/02/09/rethinkdb-vs-postgres.htm...
A support contract for $100 !!! Wow, I thought I'd heard it all but that one is new. Even cheap consultants I know won't sell you only $100 worth of time.
[deleted]
A DBMS is not a developer tool. It's a crucial building block of your application, i.e. your product or service. If a dev tool fails, only the developer suffers. If a DBMS fails, your product/service suffers. Swapping a tried and tested DBMS for another requires a very compelling business case.
I feel like the market for any fundamentally hard problem is bad, simply because it either solves a problem people didn't know they had, does it in a way that requires educating people for them to be able to use it or requires scale to be effective.
Take the internet for example - before the internet existed, the 'market' for the internet sucked.
For me, any company that failed due to solving too ambitious of a problem is the best kind group of people to invest into again and again until they succeed and succeed big. The people running it will have learned the easy lesson of solving a smaller problem faster, while retaining the chops to actually solve hard problems.
All too often, people fail because they solve too small of a problem - there was a recent discussion about that on here regarding a macos calendar widget.
Take the internet for example - before the internet existed, the 'market' for the internet sucked.
For me, any company that failed due to solving too ambitious of a problem is the best kind group of people to invest into again and again until they succeed and succeed big. The people running it will have learned the easy lesson of solving a smaller problem faster, while retaining the chops to actually solve hard problems.
All too often, people fail because they solve too small of a problem - there was a recent discussion about that on here regarding a macos calendar widget.
You can monetize databases through DBaaS (as long as you avoid tying yourself in knots over "the AWS problem"). Developer tools that can't be implemented as SaaS are tricky.
To break out of "developers making tools for developers" you need to get some domain knowledge.
To break out of "developers making tools for developers" you need to get some domain knowledge.
You know at some point in the last 20ish years, people really got used to the idea of free music. Once people get used to the idea of ‘free’, it’s like, good luck arguing against that.
I don’t think something like VSCode should be free, or a single api call on any platform, etc. Absolutely nothing should be free because people get too used to the idea of free. It’s a human failure.
I don’t think something like VSCode should be free, or a single api call on any platform, etc. Absolutely nothing should be free because people get too used to the idea of free. It’s a human failure.
This post requires a Hacker News Gold account to view.
And it would be worth it. Never give an inch unless you want yards gone.
Has it occurred to you that paying for things is a recent, temporary mode of interaction between humans while free and good will is the default?
You have it confused - prices is an arguably necessary, temporary condition, until we get back to simply from everyone according to their ability, to each according to their needs.
You have it confused - prices is an arguably necessary, temporary condition, until we get back to simply from everyone according to their ability, to each according to their needs.
I’d say that’s about as utopian of an idea as the unrestricted free market.
In theory, sure, sounds great, but the pragmatic timescale it would take to get there would be beyond lifetimes. We’ll all be dead by the time these systems reach their true form.
In your particular example for instance, human nature would wreck havoc in the form of abuse. I will point to (incoming snark) the very rare cases of slavery, underpaid labor, serfdom, like you know, these rare things that happened through the centuries. So yeah, I suppose that idea is ‘still working itself out’ two thousand years later.
Fuck all that is my point, as many souls have been lost waiting for utopia. Ask the guys that built the pyramids.
In theory, sure, sounds great, but the pragmatic timescale it would take to get there would be beyond lifetimes. We’ll all be dead by the time these systems reach their true form.
In your particular example for instance, human nature would wreck havoc in the form of abuse. I will point to (incoming snark) the very rare cases of slavery, underpaid labor, serfdom, like you know, these rare things that happened through the centuries. So yeah, I suppose that idea is ‘still working itself out’ two thousand years later.
Fuck all that is my point, as many souls have been lost waiting for utopia. Ask the guys that built the pyramids.
What's utopian about it?
What do children pay their parents, or parents pay their kids? That's the relationship people start out with and one that is most natural.
People seem to think that there is money, and before money, there was barter. That's fake news made up by economists (the barter before money bit).
People have been around for hundreds of thousands of years. Money has been around for a few thousand in select places and never used for most transactions by most people until extremely recently.
I'd argue it's still not used for the most important transactions and the existence of food stamps, universal health care in every sane developed country, free education and subsidies for parents with children is simply some of the numerous examples of most people preferring free for things they deeply value and care about.
Money is great for veblen goods and exchange among adversaries, everything else wants to be free. Free as in decided based on factors other than money, not free as in oxygen.
I'm not articulating this too well, please have a look here for a take I'm trying to express, done by a proper journalist.
https://www.theatlantic.com/business/archive/2016/02/barter-...
What do children pay their parents, or parents pay their kids? That's the relationship people start out with and one that is most natural.
People seem to think that there is money, and before money, there was barter. That's fake news made up by economists (the barter before money bit).
People have been around for hundreds of thousands of years. Money has been around for a few thousand in select places and never used for most transactions by most people until extremely recently.
I'd argue it's still not used for the most important transactions and the existence of food stamps, universal health care in every sane developed country, free education and subsidies for parents with children is simply some of the numerous examples of most people preferring free for things they deeply value and care about.
Money is great for veblen goods and exchange among adversaries, everything else wants to be free. Free as in decided based on factors other than money, not free as in oxygen.
I'm not articulating this too well, please have a look here for a take I'm trying to express, done by a proper journalist.
https://www.theatlantic.com/business/archive/2016/02/barter-...
> What's utopian about it?
It's been known as utopian since at least the time of Thomas More[1].
> What do children pay their parents, or parents pay their kids? That's the relationship people start out with and one that is most natural.
Yes, it's natural for that relationship. That does not mean that:
a) it applies to other kinds of relationship
b) it scales to all of society
In fact, it's that kind of thinking that leads to the authoritarianism of Confucianism, the tyranny of communism, and the stifling paternalism of western social democracy (among other things such as outright fascism).
Gifting worked amongst tribes because they're all related. As to Graeber, he's saying that slavery is the creation of money? No, he's too mealy mouthed for that “perhaps not creating but at least enabling institutions such as slavery”.
Wow, money enables slavery. What a pathetic attempt at insight and guilt by association.
Money is simply a better system than the competitors, as is liberalism - the wish that people be free to choose how they use their spending power (or gifting power, ha) - to its competitors. A gift economy was not able to lift almost the entire world out of poverty (and it will hopefully complete that effort soon[2]). Communism and socialism (not that there's a real difference) have only managed to emiserate and imprison every society it's been tried in. No thanks.
[1] https://en.wikipedia.org/wiki/Utopia_(book)
[2] https://www.youtube.com/watch?v=5JiYcV_mg6A
It's been known as utopian since at least the time of Thomas More[1].
> What do children pay their parents, or parents pay their kids? That's the relationship people start out with and one that is most natural.
Yes, it's natural for that relationship. That does not mean that:
a) it applies to other kinds of relationship
b) it scales to all of society
In fact, it's that kind of thinking that leads to the authoritarianism of Confucianism, the tyranny of communism, and the stifling paternalism of western social democracy (among other things such as outright fascism).
Gifting worked amongst tribes because they're all related. As to Graeber, he's saying that slavery is the creation of money? No, he's too mealy mouthed for that “perhaps not creating but at least enabling institutions such as slavery”.
Wow, money enables slavery. What a pathetic attempt at insight and guilt by association.
Money is simply a better system than the competitors, as is liberalism - the wish that people be free to choose how they use their spending power (or gifting power, ha) - to its competitors. A gift economy was not able to lift almost the entire world out of poverty (and it will hopefully complete that effort soon[2]). Communism and socialism (not that there's a real difference) have only managed to emiserate and imprison every society it's been tried in. No thanks.
[1] https://en.wikipedia.org/wiki/Utopia_(book)
[2] https://www.youtube.com/watch?v=5JiYcV_mg6A
I don’t think there’s enough liquidity in your system. Let’s say I think everyone should own a house. I then go ahead and help build a house for you. I would expect many others to have the same belief and build a house for me.
If at any time people decide they no longer believe in home ownership, then the house I built for you had no value since no one will build my home when it’s my turn.
This wouldn’t be a problem if you just paid me cash since it stores value.
If at any time people decide they no longer believe in home ownership, then the house I built for you had no value since no one will build my home when it’s my turn.
This wouldn’t be a problem if you just paid me cash since it stores value.
Right, you can't please everyone, all of the time, you can only please some people, some of the time.
We can come up with pro and con examples for any set of ideas.
I remain of the mindset that monetary transactions within a community are a net negative, largely because I think human dignity should not be something you can purchase, sell or bargain for.
If we provide human dignity for every member and let money be something we use for nice-to-have items/services, that strikes me as a significant improvement.
We can come up with pro and con examples for any set of ideas.
I remain of the mindset that monetary transactions within a community are a net negative, largely because I think human dignity should not be something you can purchase, sell or bargain for.
If we provide human dignity for every member and let money be something we use for nice-to-have items/services, that strikes me as a significant improvement.
Get back? Please tell me at what point in history could a person just pick every item and have every service for free.
Even if there was a society like that, I don't think I'd want to go - not only did they go extinct, they don't have mRNA vaccines, spacecraft, and Telemundo soaps.
Even if there was a society like that, I don't think I'd want to go - not only did they go extinct, they don't have mRNA vaccines, spacecraft, and Telemundo soaps.
What sort of nonsense are you talking about, monetary systems have existed for 1000s of years.
Humans have been humans for a lot longer than that!
Yeah but we weren't very good at it.
When you set the price at free, you set the value of the work represented, at zero.
The DB sector is great. As long as you have something differentiating, like a type of query that some businesses need, the growing market size and rise of DBaaS makes it super attractive because of conversion and stickiness.
However... who you sell to isn't the people giving you GitHub stars and retweets, and they will have diff needs. if you lose sight of that....
However... who you sell to isn't the people giving you GitHub stars and retweets, and they will have diff needs. if you lose sight of that....
Apparently, you can make a five digit income from selling an IDE colorscheme.
https://news.ycombinator.com/item?id=26262989
https://news.ycombinator.com/item?id=26262989
This sounded ridiculous at first, but after looking at it, I understood the value preposition.
It’s a colorscheme, yes, but you pay for the consistent support of over 160 apps, which is a big deal IMO. Heck, every program under the sun seems to offer a solarized dark theme, but they’re all wrong in different ways and use different accent colors for different things, so I rolled my own in a lot of cases to get an actually consistent appearance. It’s a huge hassle, and just paying 80$ would probably have saved money compared to the hours and days I spent manually theming stuff.
It’s a colorscheme, yes, but you pay for the consistent support of over 160 apps, which is a big deal IMO. Heck, every program under the sun seems to offer a solarized dark theme, but they’re all wrong in different ways and use different accent colors for different things, so I rolled my own in a lot of cases to get an actually consistent appearance. It’s a huge hassle, and just paying 80$ would probably have saved money compared to the hours and days I spent manually theming stuff.
If you want to make any money in developer tools, the enterprise corporations are the best customers.
Don't expect to make big bucks selling to single devs.
Don't expect to make big bucks selling to single devs.
It's great market but things are very competitive. If you are starting a business, you won't succeed unless you bring some unique value to the table.
Open source means that all value and good ideas turn into commodities. Whatever mongo did ten years ago that set them apart is now a commodity. You can get it from dozens of different OSS products that have since built similar features or that have been created since then. That does not devalue mongo's product and they seem to be making some nice profits monetizing it. It's just that their value proposition has now shifted to hosting and support.
RethinkDB as a database actually did not fail. It's just the company behind it that failed. And strictly speaking, it merely failed to deliver on unrealistic expectations its investors had. Companies fail all the time; usually it has to do with product market fit, figuring out a sane business model, etc. However, the database still exists. People still use it. People still work on it. I've never used it myself. But it looks like an interesting product.
Whining that people won't pay for a database is pointless. There are plenty of successful companies making money from their OSS software. Most companies don't get to be unicorns though. That's disappointing if you are a VC but otherwise not even a goal for many healthy companies that are just looking to make healthy profits doing honest work.
In this case, VCs helped fund an unsuccessful company that happened to have created a successful product. That happens a lot in that world. VCs pay for people to take risks and sometimes that just doesn't work out and they walk away from it. But because it was open source, they are the only ones that lost. Mysql has seen many companies come and go but the product seems to stick around. Same for Postgresql. Both have a very healthy ecosystem of large and small players depending on and making money with these products. Some of these are unicorns even.
Open source means that all value and good ideas turn into commodities. Whatever mongo did ten years ago that set them apart is now a commodity. You can get it from dozens of different OSS products that have since built similar features or that have been created since then. That does not devalue mongo's product and they seem to be making some nice profits monetizing it. It's just that their value proposition has now shifted to hosting and support.
RethinkDB as a database actually did not fail. It's just the company behind it that failed. And strictly speaking, it merely failed to deliver on unrealistic expectations its investors had. Companies fail all the time; usually it has to do with product market fit, figuring out a sane business model, etc. However, the database still exists. People still use it. People still work on it. I've never used it myself. But it looks like an interesting product.
Whining that people won't pay for a database is pointless. There are plenty of successful companies making money from their OSS software. Most companies don't get to be unicorns though. That's disappointing if you are a VC but otherwise not even a goal for many healthy companies that are just looking to make healthy profits doing honest work.
In this case, VCs helped fund an unsuccessful company that happened to have created a successful product. That happens a lot in that world. VCs pay for people to take risks and sometimes that just doesn't work out and they walk away from it. But because it was open source, they are the only ones that lost. Mysql has seen many companies come and go but the product seems to stick around. Same for Postgresql. Both have a very healthy ecosystem of large and small players depending on and making money with these products. Some of these are unicorns even.
> just the db sector is bad?
how is it bad when snowflake just had the biggest tech ipo of all time.
how is it bad when snowflake just had the biggest tech ipo of all time.
(2017)
There was a large discussion (948 points, 267 comments) at the time: https://news.ycombinator.com/item?id=13421608
There was a large discussion (948 points, 267 comments) at the time: https://news.ycombinator.com/item?id=13421608
Really good write up. If you've ever listened to an episode of "How I Built This" from NPR, this is the opposite of that. But still equally informative and useful.
2017
They failed because they focused on marketing features instead of essential ones like data compression.
Recent developments - RethinkDB is back and is being backed by the Linux Foundation
yes I highly recommend it. There are quirks, the Go api is a little sketch (community effort) but otherwise it's been fantastic.
I thought they failed cause unbelievably powerful databases are available totally free.
What more reason is needed?
What more reason is needed?
Mongodb has a 15b market cap, and mongo is widely derided.
But it’s the only real player in their market. RethinkDB with MongoDB ‘performance’ would have blown them out of the water.
But it’s the only real player in their market
Not at all. All of MongoDB is just a feature in Postgres, and Postgres does it better, and Postgres does hundreds more things too.
Not at all. All of MongoDB is just a feature in Postgres, and Postgres does it better, and Postgres does hundreds more things too.
I like both databases. Saying that MongoDB is a subset of PostgreSQL is not right, IMO. They are both powerful databases. For legacy off-Oracle or off-SQL Server migration, PostgreSQL sure is an easier place to start (though you still likely have to re-write/re-tune your application). For new apps, you could start with either.
The idea that PostgreSQL is a superset of MongoDB is incorrect. As some examples, PostgreSQL doesn't shard, have multi-region, multi-cloud, enforce JSON schema, data governance, full JSON support, etc.
That said, PostgreSQL does do a lot of things MongoDB doesn't do.
The two products have a big feature overlap - both are great databases. I think you could build a modern app on either one. If you want to build in modern languages faster and have a more available system out of the box, I'd suggest MongoDB. If you have legacy skillsets, legacy apps, or just love relational, I'd suggest PostgreSQL.
The idea that PostgreSQL is a superset of MongoDB is incorrect. As some examples, PostgreSQL doesn't shard, have multi-region, multi-cloud, enforce JSON schema, data governance, full JSON support, etc.
That said, PostgreSQL does do a lot of things MongoDB doesn't do.
The two products have a big feature overlap - both are great databases. I think you could build a modern app on either one. If you want to build in modern languages faster and have a more available system out of the box, I'd suggest MongoDB. If you have legacy skillsets, legacy apps, or just love relational, I'd suggest PostgreSQL.
MongoDB is not in the same market as postgres.
I don’t disagree with you, but the people that choose Mongo do not see postgres as a valid alternative.
I don’t disagree with you, but the people that choose Mongo do not see postgres as a valid alternative.
>and Postgres does hundreds more things too.
being hip, which is one of the most important factors for choosing a database database technology, is not one of them though.
being hip, which is one of the most important factors for choosing a database database technology, is not one of them though.
They finally got their engineering act together (via the Sleepycat acquisition), long after they'd made it big.
Didn’t Oracle buy Sleepycat?
Yes, my bad, I was referring to WiredTiger (still the same folks).
Mongo bought WiredTiger.
Both founded by Keith Bostic, from Berkeley DB heritage.
Both founded by Keith Bostic, from Berkeley DB heritage.
"Read The Economist religiously. It will make you better faster."
Huh?
Huh?
What he means is that if you're completely ignorant about business, then reading a financial magazine will help you level up over time.
(Related: I've worked with new managers in SV who couldn't define what the words "leadership", "responsibility", "planning" or "accountability" meant, since all they knew was PHP programming.)
It's a good post mortem read.
I gave the same advice about "use case" as in the post mortem to a YC database company a few years ago. The "founders" also thought they were building a better (technical) mousetrap, but I told them nobody was going to pay anything if it did the same as an existing database - it had to do something unique and be cloud-based, like Snowflake, or RDS. They were crestfallen, but still didn't listen to me. I even got hate mail from them before they went under - talk about being clueless to the bitter end.
What RethinkDB did wrong was what was in their post mortem, but also not having experienced DBAs to review the use cases. As a DBA, my default is to say no to new databases, and being the audience, that would have been valuable for them to know.
An example of how DBAs and devs think differently is the Jepsen database reports - devs fetishize those, but DBAs are like, "We already know corner cases aren't going to work most of the time. What's your point exactly?"
Source: experienced DBA.
(Related: I've worked with new managers in SV who couldn't define what the words "leadership", "responsibility", "planning" or "accountability" meant, since all they knew was PHP programming.)
It's a good post mortem read.
I gave the same advice about "use case" as in the post mortem to a YC database company a few years ago. The "founders" also thought they were building a better (technical) mousetrap, but I told them nobody was going to pay anything if it did the same as an existing database - it had to do something unique and be cloud-based, like Snowflake, or RDS. They were crestfallen, but still didn't listen to me. I even got hate mail from them before they went under - talk about being clueless to the bitter end.
What RethinkDB did wrong was what was in their post mortem, but also not having experienced DBAs to review the use cases. As a DBA, my default is to say no to new databases, and being the audience, that would have been valuable for them to know.
An example of how DBAs and devs think differently is the Jepsen database reports - devs fetishize those, but DBAs are like, "We already know corner cases aren't going to work most of the time. What's your point exactly?"
Source: experienced DBA.
Thanks for the reply. I liked the article and was confused just about this part. I've browsed the Economist's site and all I could see was political propaganda and drama.
Hard to believe that reading a financial magazine written by journalists will increase your knowledge rather than an authority in the domain or a classic book on the subject.
Now you know why they failed.
Read with "Liberalism at Large: The World According to the Economist" for context.
With four years of hindsight, the estimate of DBaaS and managed database hosting companies of being <10M revenue stood out at me. MongoDB Atlas hit $85M revenue in Q4 2020 (https://www.fool.com/earnings/call-transcripts/2021/03/10/mo...), which means that assuming RethinkDB's estimates were correct, the market they wrote off as too small was actually about to explode into something much larger.
That's not to say that they would have succeeded if they'd gone after that though; instead we might have a blog post about how they correctly predicted where the market would go and simply ran out of money before the market got there.
That's not to say that they would have succeeded if they'd gone after that though; instead we might have a blog post about how they correctly predicted where the market would go and simply ran out of money before the market got there.
MongoDB's stock price is quite remarkable, given how dismissive a lot of people (including myself) on HacherNews have been of their offering.
It goes to show that HackerNews consensus != reality.
I do wonder where/how Mongo makes their money - if anybody has some insight, please share.
It goes to show that HackerNews consensus != reality.
I do wonder where/how Mongo makes their money - if anybody has some insight, please share.
https://www.prnewswire.com/news-releases/mongodb-inc-announc...
- Revenue: Total revenue was $150.8 million in the third quarter fiscal 2021, an increase of 38% year-over-year. Subscription revenue was $144.1 million, an increase of 39% year-over-year, and services revenue was $6.7 million, an increase of 19% year-over-year.
- Gross Profit: Gross profit was $104.7 million in the third quarter fiscal 2021, representing a 69% gross margin, compared to 71% in the year-ago period. Non-GAAP gross profit was $108.6 million, representing a 72% non-GAAP gross margin.
- Loss from Operations: Loss from operations was $58.1 million in the third quarter fiscal 2021, compared to $38.7 million in the year-ago period. Non-GAAP loss from operations was $16.0 million, compared to $14.3 million in the year-ago period.
- Net Loss: Net loss was $72.7 million, or $1.22 per share, based on 59.4 million weighted-average shares outstanding in the third quarter fiscal 2021. This compares to $42.4 million, or $0.75 per share, based on 56.4 million weighted-average shares outstanding, in the year-ago period. Non-GAAP net loss was $18.2 million or $0.31 per share. This compares to $14.6 million, or $0.26 per share, in the year-ago period.
- Cash Flow: As of October 31, 2020, MongoDB had $966.8 million in cash, cash equivalents, short-term investments and restricted cash. During the three months ended October 31, 2020, MongoDB used $8.1 million of cash from operations, $5.6 million in capital expenditures and $1.2 million in principal repayments of finance leases, leading to negative free cash flow of $14.9 million, compared to negative free cash flow of $13.1 million in the year-ago period.
- Revenue: Total revenue was $150.8 million in the third quarter fiscal 2021, an increase of 38% year-over-year. Subscription revenue was $144.1 million, an increase of 39% year-over-year, and services revenue was $6.7 million, an increase of 19% year-over-year.
- Gross Profit: Gross profit was $104.7 million in the third quarter fiscal 2021, representing a 69% gross margin, compared to 71% in the year-ago period. Non-GAAP gross profit was $108.6 million, representing a 72% non-GAAP gross margin.
- Loss from Operations: Loss from operations was $58.1 million in the third quarter fiscal 2021, compared to $38.7 million in the year-ago period. Non-GAAP loss from operations was $16.0 million, compared to $14.3 million in the year-ago period.
- Net Loss: Net loss was $72.7 million, or $1.22 per share, based on 59.4 million weighted-average shares outstanding in the third quarter fiscal 2021. This compares to $42.4 million, or $0.75 per share, based on 56.4 million weighted-average shares outstanding, in the year-ago period. Non-GAAP net loss was $18.2 million or $0.31 per share. This compares to $14.6 million, or $0.26 per share, in the year-ago period.
- Cash Flow: As of October 31, 2020, MongoDB had $966.8 million in cash, cash equivalents, short-term investments and restricted cash. During the three months ended October 31, 2020, MongoDB used $8.1 million of cash from operations, $5.6 million in capital expenditures and $1.2 million in principal repayments of finance leases, leading to negative free cash flow of $14.9 million, compared to negative free cash flow of $13.1 million in the year-ago period.
MongoDB Atlas's business model seems very, very odd to me. The prices for hosted Mongo are so high that I really find myself asking who is paying for this and what the target market is. In particular I wonder how they can possibly be losing money when charging such insane markups unless they have virtually no customers.
For a dedicated database that has 32 GB of RAM, 8 virtualised CPUs, and only 160GB of storage you will pay $2 an hour! That's $17,520 a year for a database with the same level of compute and far less storage than my laptop.
I realise a big part of their target market is firms who are convinced they either can't hire skilled sysadmins or are convinced their time is worth more than the US President but at $17,520/yr for a tiny database it only takes two or three of those before you could, in fact, hire a full time sysadmin in a cheap location. And there's no way that maintaining these things would require an FTE, it's not even close to being a full time job. So how do the economics of this work out? Can anyone explain to me?
My best guess is that Atlas customers are companies that are totally price insensitive and have some sort of "everything must be cloud managed without exception" rule imposed from the top down. As Mongo is licensed such that only they can run a hosted service, such customers would find themselves forced by internal policy into paying whatever price Atlas charges. This would also explain their very low services revenue. But I don't know that, it's just a guess.
For a dedicated database that has 32 GB of RAM, 8 virtualised CPUs, and only 160GB of storage you will pay $2 an hour! That's $17,520 a year for a database with the same level of compute and far less storage than my laptop.
I realise a big part of their target market is firms who are convinced they either can't hire skilled sysadmins or are convinced their time is worth more than the US President but at $17,520/yr for a tiny database it only takes two or three of those before you could, in fact, hire a full time sysadmin in a cheap location. And there's no way that maintaining these things would require an FTE, it's not even close to being a full time job. So how do the economics of this work out? Can anyone explain to me?
My best guess is that Atlas customers are companies that are totally price insensitive and have some sort of "everything must be cloud managed without exception" rule imposed from the top down. As Mongo is licensed such that only they can run a hosted service, such customers would find themselves forced by internal policy into paying whatever price Atlas charges. This would also explain their very low services revenue. But I don't know that, it's just a guess.
> So how do the economics of this work out? Can anyone explain to me?
Idk, but maybe the engineers don't fully trust it, so they outsource the liability of running it with some contractual guarantees. And at the moment, this probably looks more fashionable, so no one dares question it in fear of seeming backwards.
Idk, but maybe the engineers don't fully trust it, so they outsource the liability of running it with some contractual guarantees. And at the moment, this probably looks more fashionable, so no one dares question it in fear of seeming backwards.
Don’t you get a replica set for that $? That's three of your laptops. And if any of them fail, a free replacement. And that FT sysadmin of yours is willing to work 24x7? Cool!
[deleted]
I'm always kind of amazed at the low services number on MongoDB earnings reports. It's 4-5%. This seems unusually low for enterprise database companies.
As percentage of total revenue, MongoDB service revenue is about half compared to what Oracle generates. My guess is customer churn is much higher for MongoDB.
Hmmm. Isn't customer churn public?
Anyways, my guess is that the product is
1) much easier to use (less bells and whistles)
2) mongodb provides more configuration etc without charging for it.
HackerNews is wildly detached from reality
No, I think it's more truthful than any other place with bunch of noises.
Alternatively: capitalism optimizes for charlatans, not quality.
Maybe so, but please don't post unsubstantive comments here.
HN consesus!=Market consesus
Reality!=Market consensus (startups that lost their data using the first versions of mongo)
More often: HN consensus=Reality
At least ES had the decency to tell you it might lose your data.
Reality!=Market consensus (startups that lost their data using the first versions of mongo)
More often: HN consensus=Reality
At least ES had the decency to tell you it might lose your data.
I’m on record here saying I really dislike Mongo DB and never want to work with it again. I stand by that. But Atlas is a really nice product (to work around the inherent failings of Mongo, but hey, you’re locked in already). Want to look at something in a backup? Select your snapshot, download and run a special binary, and point your client at it. It takes like 30 seconds to be pulling data from any backup. I did it this week. Now, would I have been able to make the mess I needed to recover from if I’d been using Postgres? No, of course not. But like I said: you’re locked-in already.
Edit: Mongo is a special kind of lock-in: your code ends up so much more tightly-coupled, and hard to extricate, because of the Mongo pattern of making you do the work to ensure consistency and correctness. I can’t imagine it was planned, but it’s highly effective: for many companies it’s just too much work, too expensive, and the benefits are too opaque (“but the application works now, and you’re telling me you want to switch databases and it’ll take how long?!”) for others to buy into.
I’d call it clever, but I think it’s a profitable accident that’s a braking force on the progress of many teams.
Edit: Mongo is a special kind of lock-in: your code ends up so much more tightly-coupled, and hard to extricate, because of the Mongo pattern of making you do the work to ensure consistency and correctness. I can’t imagine it was planned, but it’s highly effective: for many companies it’s just too much work, too expensive, and the benefits are too opaque (“but the application works now, and you’re telling me you want to switch databases and it’ll take how long?!”) for others to buy into.
I’d call it clever, but I think it’s a profitable accident that’s a braking force on the progress of many teams.
I think people are dismissive of the product not the business.
Its not a contradiction to dislike mongodb technically well still thinking its a good business.
Its not a contradiction to dislike mongodb technically well still thinking its a good business.
I have used Mongo for 9 years. First at a game company, then for several prototypes that made it to production in my current role. We have not lost data once. Many of my friends at medium-sized companies use Mongo for much larger projects and have not lost data a single time, either.
And it is it not just me and my friends -- HSBC recently announced they are moving 65 of its relational databases to mongodb. I am not sure that is wise, but I think it furthers my view that data-loss is not a concern in many contexts.
The data-loss thing was most certainly real, but has not been since wiredtiger if one manages it the right way or uses it as a service.
I would not say that I would say that HN != reality, maybe !== reality :). Neither side of the debate is wholly right, and neither is wholly wrong.
The reason I use it is that for many projects, all you need is a relatively fast JSON document store, and mongo is so developer-friendly such that speed-to-prototype is so much shorter than anything else I have used in 25 years of building database-backed applications. Rethink was the only product that I would have used in its place and I was extremely sad to see it go. I agree with the commenter elsewhere in the belief that it would have thrived had it survived. I was using it for my personal/learning projects and was ready to switch everywhere.
We have stuck with Mongo in production for a few products, one relatively large, and I think if you have separated your concerns well-enough, it is okay developing in Mongo while leaving yourself a path to postgresql or something else. You need to prove, first, that the thing you are building is useful, and Mongo will let you get to that really fast.
I believe what is happening here is that the happy folks are quiet, and the aggrieved folks' experiences are real but it likely was a bad fit for their particular situation. For the subset of situations where we use it, it has allowed my teams quickly in our experimental work.
All of that said, I work in an innovation space where we throw a lot of things at the wall to see what sticks, and our time-to-prototype is often a few months at most. For us, it has been a godsend, but our situation is rather unique.
-h
And it is it not just me and my friends -- HSBC recently announced they are moving 65 of its relational databases to mongodb. I am not sure that is wise, but I think it furthers my view that data-loss is not a concern in many contexts.
The data-loss thing was most certainly real, but has not been since wiredtiger if one manages it the right way or uses it as a service.
I would not say that I would say that HN != reality, maybe !== reality :). Neither side of the debate is wholly right, and neither is wholly wrong.
The reason I use it is that for many projects, all you need is a relatively fast JSON document store, and mongo is so developer-friendly such that speed-to-prototype is so much shorter than anything else I have used in 25 years of building database-backed applications. Rethink was the only product that I would have used in its place and I was extremely sad to see it go. I agree with the commenter elsewhere in the belief that it would have thrived had it survived. I was using it for my personal/learning projects and was ready to switch everywhere.
We have stuck with Mongo in production for a few products, one relatively large, and I think if you have separated your concerns well-enough, it is okay developing in Mongo while leaving yourself a path to postgresql or something else. You need to prove, first, that the thing you are building is useful, and Mongo will let you get to that really fast.
I believe what is happening here is that the happy folks are quiet, and the aggrieved folks' experiences are real but it likely was a bad fit for their particular situation. For the subset of situations where we use it, it has allowed my teams quickly in our experimental work.
All of that said, I work in an innovation space where we throw a lot of things at the wall to see what sticks, and our time-to-prototype is often a few months at most. For us, it has been a godsend, but our situation is rather unique.
-h
Genuine question: What does mongo give you as a JSON store that postgres's JSONB doesn't?
Querying is in pure JS, easy to do stuff programmatically. But if starting new, with a team that did not know Mongo well already, not a real advantage. I still prefer querying something nested in mongo to querying something nested in JSONB, but I can work with either.
And back when I started with it, JSONb did not exist. I would say JSONB was also a godsend though as our roadmaps now have a clear path to something we may prefer later. People up the chain are often terrified when they hear the word "mongo" so the existence of JSONB makes it easier to swap out your layer that handles database interactions.
But the speed advantage really lies in the early-stage prototype development. If I have to do it in postgres, the amount of code necessary up front, when inevitable schema changes happen, etc. makes mongo preferable until things settle down.
But again, it all depends on how well you know the thing up front. If you can anticipate a lot of iteration on the data model as you learn how people use your product, mongo is handy early. If I am given a spec for something where all of the things are known and foreseen, I am going with postgresql.
I assume that the hardest thing to do is making something people want to use. If I can achieve that with whatever I use, I have some good (and fun) problems on my hands. And it is all about structuring the work so that I am not tied to any database product. I am no longer working on startups with little-to-no funding, so right now I assume AWS or Azure will manage a database better than I can. And I assume new things are always coming.
And back when I started with it, JSONb did not exist. I would say JSONB was also a godsend though as our roadmaps now have a clear path to something we may prefer later. People up the chain are often terrified when they hear the word "mongo" so the existence of JSONB makes it easier to swap out your layer that handles database interactions.
But the speed advantage really lies in the early-stage prototype development. If I have to do it in postgres, the amount of code necessary up front, when inevitable schema changes happen, etc. makes mongo preferable until things settle down.
But again, it all depends on how well you know the thing up front. If you can anticipate a lot of iteration on the data model as you learn how people use your product, mongo is handy early. If I am given a spec for something where all of the things are known and foreseen, I am going with postgresql.
I assume that the hardest thing to do is making something people want to use. If I can achieve that with whatever I use, I have some good (and fun) problems on my hands. And it is all about structuring the work so that I am not tied to any database product. I am no longer working on startups with little-to-no funding, so right now I assume AWS or Azure will manage a database better than I can. And I assume new things are always coming.
I can't follow this through, I'm interested in nosql databases because there are indeed patterns of access where a db as a key value store is helpful.
> Querying is in pure JS, easy to do stuff programmatically.
You mean you do your logic at the application layer? Like if you need X objects and Y objects where Y have an identifier to the X objects they're related to you build them in JS after retrieving them?
What is the difference from getting a json object and doing the "querying" in pure js?
And isn't it easier "programatically" to just join the data at the source?
> But the speed advantage really lies in the early-stage prototype development. If I have to do it in postgres, the amount of code necessary up front, when inevitable schema changes happen, etc. makes mongo preferable until things settle down.
But again, what is the difference of having a record with a column with json, that you just do rodeo dump?
(I think pg even allows you to do all sorts of indexing on inner fields of json if I'm not mistaken?)
> Querying is in pure JS, easy to do stuff programmatically.
You mean you do your logic at the application layer? Like if you need X objects and Y objects where Y have an identifier to the X objects they're related to you build them in JS after retrieving them?
What is the difference from getting a json object and doing the "querying" in pure js?
And isn't it easier "programatically" to just join the data at the source?
> But the speed advantage really lies in the early-stage prototype development. If I have to do it in postgres, the amount of code necessary up front, when inevitable schema changes happen, etc. makes mongo preferable until things settle down.
But again, what is the difference of having a record with a column with json, that you just do rodeo dump?
(I think pg even allows you to do all sorts of indexing on inner fields of json if I'm not mistaken?)
Oh no, apologies for the lack of clarity. I would not get the whole thing and build it in the application layer before sending it to the client.
If I want a particular array item nested a few levels in a JSON, the query as JSON object instead of query as an SQL is just a bit easier to put together (which is what i mean by pure JS). Sometimes, putting complex queries together takes fewer lines of code. Neither is really better or worse.
But I want the database doing the work and want to transform as little as possible on the application layer. Sometimes it is inevitable, but sub-optimal, for sure.
In terms of anything where you need to do a lot of joins when querying -- NoSQL is not good and I would not advise Mongo at all. It depends on what you are building. A lot of applications (and even games) now just pass JSONs around. If I show a user profile, I get that json. If, on the profile, I need to show where she lies in a leaderboard, that react component hits a separate API call to get the data needed for it, etc.
But no difference really. And yes, if the JSON is just a column in postgres, even the early stuff is nearly as easy as it is in postgresql.
If I were looking at a DB for a simple key/value store, I would not look at Mongo. Others here likely know better, but I pick up Redis (out of habit, but also it is awesome even if overkill in many scenarios). Mongo would do it but I would not do so except temporarily. I tend to throw Redis in my stacks early in development as I tend to need it later.
BTW by no means do I love Mongo. For example, Mongo is awful when it comes to things simiilar to joining in SQL land. You end up in these hellscapes where there is no choice but to store a ton of redundant data. Is it the end of the world? No, storage is cheap. But is it good? No.
And I am old so I do not find the value in arguing about tools. It is like me hating a hammer. I am interested in what is better and what folks here think about things coming down the pipeline, or whether Arango is worth looking at, etc. But I do appreciate your sort of inquiry as I have learned from it -- for example I did not know you can index inner fields of JSONs. I really need to revisit is as recent versions of PG seem to get better and better.
Context matters a ton. If you are going right to production with something, that is not really where I operate. But in general, as many have said, if more than one thing will do it, the best tool is the one you know best. You are likely to implement it efficiently, etc.
If I want a particular array item nested a few levels in a JSON, the query as JSON object instead of query as an SQL is just a bit easier to put together (which is what i mean by pure JS). Sometimes, putting complex queries together takes fewer lines of code. Neither is really better or worse.
But I want the database doing the work and want to transform as little as possible on the application layer. Sometimes it is inevitable, but sub-optimal, for sure.
In terms of anything where you need to do a lot of joins when querying -- NoSQL is not good and I would not advise Mongo at all. It depends on what you are building. A lot of applications (and even games) now just pass JSONs around. If I show a user profile, I get that json. If, on the profile, I need to show where she lies in a leaderboard, that react component hits a separate API call to get the data needed for it, etc.
But no difference really. And yes, if the JSON is just a column in postgres, even the early stuff is nearly as easy as it is in postgresql.
If I were looking at a DB for a simple key/value store, I would not look at Mongo. Others here likely know better, but I pick up Redis (out of habit, but also it is awesome even if overkill in many scenarios). Mongo would do it but I would not do so except temporarily. I tend to throw Redis in my stacks early in development as I tend to need it later.
BTW by no means do I love Mongo. For example, Mongo is awful when it comes to things simiilar to joining in SQL land. You end up in these hellscapes where there is no choice but to store a ton of redundant data. Is it the end of the world? No, storage is cheap. But is it good? No.
And I am old so I do not find the value in arguing about tools. It is like me hating a hammer. I am interested in what is better and what folks here think about things coming down the pipeline, or whether Arango is worth looking at, etc. But I do appreciate your sort of inquiry as I have learned from it -- for example I did not know you can index inner fields of JSONs. I really need to revisit is as recent versions of PG seem to get better and better.
Context matters a ton. If you are going right to production with something, that is not really where I operate. But in general, as many have said, if more than one thing will do it, the best tool is the one you know best. You are likely to implement it efficiently, etc.
Thanks.
I can see the utility of having separate components do their "own" full lifecycle even if I usually do tend to approach it in other ways, it's a valid take and can make sense (although as you said I think it can be done with SQL).
I agree with you on redis-like utilities being better for pure KV stores. If you resort to that sort of thing regularly and want to give another look into postgres you might want to give Elixir a try. It has first class support for postgres with Ecto, and you can use redis-like stores from the language itself, skipping serialization (a note is that, one of the valuable things in redis usually comes from being in a separate instance, many times managed automatically, and so almost being a permanent cache - to have exactly the same properties you would need to dig a bit deeper in Elixir, but still pretty doable with the language building blocks).
I agree with you on redis-like utilities being better for pure KV stores. If you resort to that sort of thing regularly and want to give another look into postgres you might want to give Elixir a try. It has first class support for postgres with Ecto, and you can use redis-like stores from the language itself, skipping serialization (a note is that, one of the valuable things in redis usually comes from being in a separate instance, many times managed automatically, and so almost being a permanent cache - to have exactly the same properties you would need to dig a bit deeper in Elixir, but still pretty doable with the language building blocks).
> All of that said, I work in an innovation space where we throw a lot of things at the wall to see what sticks, and our time-to-prototype is often a few months at most. For us, it has been a godsend, but our situation is rather unique.
The very RAD reasons you note regarding Mongo also encourage Mongo Cancer (tm).
What is Mongo Cancer you ask? It is the phenomena where Mongo use is manifest from API to home-grown "relational logic" in business tier. This simply doesn't happen with any other NoSQL DB afaik, but Mongo Cancer uniquely marches right up to the user's face in the API.
> I think if you have separated your concerns well-enough, it is okay developing in Mongo while leaving yourself a path to postgresql or something else.
I'll hold you to that "I think" and assume you have never actually done this.
Because Mongo Cancer patients typically love the RAD aspect of grabing JSON hot off the http stack and shoving it in the "database", touching every API in the way. The last M.C. project I saw required effectively a rebuild to actually swap the misused Mongo.
tldr; Mongo Cancer is very easy to catch but requires very expensive surgery to cure.
The very RAD reasons you note regarding Mongo also encourage Mongo Cancer (tm).
What is Mongo Cancer you ask? It is the phenomena where Mongo use is manifest from API to home-grown "relational logic" in business tier. This simply doesn't happen with any other NoSQL DB afaik, but Mongo Cancer uniquely marches right up to the user's face in the API.
> I think if you have separated your concerns well-enough, it is okay developing in Mongo while leaving yourself a path to postgresql or something else.
I'll hold you to that "I think" and assume you have never actually done this.
Because Mongo Cancer patients typically love the RAD aspect of grabing JSON hot off the http stack and shoving it in the "database", touching every API in the way. The last M.C. project I saw required effectively a rebuild to actually swap the misused Mongo.
tldr; Mongo Cancer is very easy to catch but requires very expensive surgery to cure.
No, we have migrated prototypes to redshift. You just need to keep your database logic in a separate layer, which is something one should do anyway. I say "I think" because I agree with you that most will not. If you implement mongo such that you will be stuck on mongo, then do not use it. You always want to have a layer that handles your database operations, with any database, but obviously you are safer in SQL-land as switching costs are lower.
I do not think the "cancer" metaphor is valuable here. What you describe are bad ideas that I agree are bad ideas but really one has to know the tool they are using and what it is bad at.
I think the thing you describe can happen but that is just poor planning more than anything. Why does the database have anything to do with the design of your API layer? That sounds bonkers. And I would not blame the underlying database tech. I would blame the implementer.
But really, please consider that many of us have, or are as we speak, losing loved ones to a deadly sickness which you have compared to a poor database implementation.
I do not think the "cancer" metaphor is valuable here. What you describe are bad ideas that I agree are bad ideas but really one has to know the tool they are using and what it is bad at.
I think the thing you describe can happen but that is just poor planning more than anything. Why does the database have anything to do with the design of your API layer? That sounds bonkers. And I would not blame the underlying database tech. I would blame the implementer.
But really, please consider that many of us have, or are as we speak, losing loved ones to a deadly sickness which you have compared to a poor database implementation.
But the technology encourages this type of behavior. And I am not speaking of hypothetical "can happen". I very recently had to deal with this precise issue.
> But really, please consider that many of us have, or are as we speak, losing loved ones to a deadly sickness which you have compared to a poor database implementation.
Sorry for your loss. Use of the word for non-medical concerns is common. Mongo Metastasis works too.
Again disagree regarding "poor database implementation". Trivializing the "database implementation" is precisely Mongo's value proposition and the very reason they got their market share.
> But really, please consider that many of us have, or are as we speak, losing loved ones to a deadly sickness which you have compared to a poor database implementation.
Sorry for your loss. Use of the word for non-medical concerns is common. Mongo Metastasis works too.
Again disagree regarding "poor database implementation". Trivializing the "database implementation" is precisely Mongo's value proposition and the very reason they got their market share.
I agree that it is very easy to make bad decisions with this product, but I do not think it is unique to Mongo. And it is the role of a good architect or a good team lead to make sure that this does not happen or that developers do not go down these roads.
I would even say the strengths that I have pointed out are precisely what makes it easier to take shortcuts. It gets out of your way, makes collections if they do not exist, allows for different document types in the same collections. It has no guard rails whatsoever.
I think we are in agreement -- one needs to be very careful using this. You can use it and put yourself in a spot where you have to throw out your entire codebase.
Well, me saying "can happen" does not mean it does not happen or that people have not seen it. It is saying the opposite. My point is it does not have to or that it does not always happen.
Developers take shortcuts, often because they do not know better, and you are right that some products make it easier to go down these roads.
I still do not think you want to use medical diseases as metaphors. You would not say "mongo aids" or "mongo alzheimers" or "mongo diabetes." Even if it is common, it is not particularly useful. Metastasis is better.
I would even say the strengths that I have pointed out are precisely what makes it easier to take shortcuts. It gets out of your way, makes collections if they do not exist, allows for different document types in the same collections. It has no guard rails whatsoever.
I think we are in agreement -- one needs to be very careful using this. You can use it and put yourself in a spot where you have to throw out your entire codebase.
Well, me saying "can happen" does not mean it does not happen or that people have not seen it. It is saying the opposite. My point is it does not have to or that it does not always happen.
Developers take shortcuts, often because they do not know better, and you are right that some products make it easier to go down these roads.
I still do not think you want to use medical diseases as metaphors. You would not say "mongo aids" or "mongo alzheimers" or "mongo diabetes." Even if it is common, it is not particularly useful. Metastasis is better.
A happy conclusion. We're in general agreement.
I firmly believe that RethinkDB could have been a huge player along with the likes of MongoDB and Elasticsearch if they could have gotten additional funding or focused more on business development. Note: both MongoDB and Elasticsearch are now hugely successful public companies.
The best technology and product doesn't always win.
- Changefeeds were so useful and still to this day not really matched in quality.
- Official client libraries were very high quality. I mainly used Node.js.
- Performance (reads and writes) was impressive.
- Setting up a highly available cluster with sharding was incredibly easy with a few clicks and types in the aforementioned web U/I.
- Their web U/I was the best to ship with a database at the time.
The best technology and product doesn't always win.
I liked the RethinkDB API better too. Seemed more intuitive and you could do something like joins in it too.
I loved the API. It made it very clear and easy to think about how my data is being accessed, how my indexes are used etc. I miss working with rethinkdb :(
Honestly, as the person who is ending up having to do the RethinkDB migration... it's the lack of an AWS hosted solution that killed it for us. Even the import is a great experience and moving to MongoDB wasn't too bad (after a bit of cleanup)
Interesting, RethinkDB was really easy to setup. Three EC2 instances (cluster), install their apt repo, and install. There were a few config options[1], but really minimal and clean.
[1] https://rethinkdb.com/docs/config-file/
[1] https://rethinkdb.com/docs/config-file/
Yeah, that's what we ran too, but there's a heavy internal push to use managed services.
[deleted]
The setup process isn't the reason people want to use hosted services, the maintenance is.
Performance for my workload (writes/reads for a relatively small number of large documents) was absolutely atrocious.
To the point that retrieving data based on a secondary index (of like 10000 documents) took like 5s.
Then inserting the same documents took a significant (don’t remember the exact magnitude) amount of time as well.
By comparison mysql does the same stuff in 50ms (or less).
Everything around RethinkDB was fantastic, but I just couldn’t justify that since the basic operation was so terrible.
To the point that retrieving data based on a secondary index (of like 10000 documents) took like 5s.
Then inserting the same documents took a significant (don’t remember the exact magnitude) amount of time as well.
By comparison mysql does the same stuff in 50ms (or less).
Everything around RethinkDB was fantastic, but I just couldn’t justify that since the basic operation was so terrible.
But, but...the data structures were all lock-free! Who cares if throughput sucked?
Yeah, let me just spin up a few hundred more servers. That’ll take care of the problem right quick!
RethinkDB could have been a huge player along with the likes of MongoDB
MongoDB is a weird one because what they had didn't even meet the most basic requirements of "a database", that you could store something in it and reliably get it back out. But they somehow converted their joke of a product into enough money to buy WiredTiger, rebrand its product as theirs, and profit. It's the ultimate expression of "fake it 'til you make it" and they did it by giving away stickers at conferences!
The best technology and product doesn't always win.
Once you realise what we do is not engineering, it's fashion, it will all make sense.
MongoDB is a weird one because what they had didn't even meet the most basic requirements of "a database", that you could store something in it and reliably get it back out. But they somehow converted their joke of a product into enough money to buy WiredTiger, rebrand its product as theirs, and profit. It's the ultimate expression of "fake it 'til you make it" and they did it by giving away stickers at conferences!
The best technology and product doesn't always win.
Once you realise what we do is not engineering, it's fashion, it will all make sense.
Wiretiger replaced the mmap storage engine. It was more of a library than a server.
Has MongoDB "made it" now or are they still "faking it"? Under what conditions is data lost?
Read The Economist religiously. It will make you better faster.
The challenge in fast changing markets is that the best information isn't written down anywhere (and especially not in magazines!), but that it's locked inside people's heads.
At least, this is what has worked for me (TimescaleDB founder)!
Very much agree with this point but for a different reason -- reading The Economist might be nice to get your feet wet with business/markets/investment-oriented thinking, but the vast majority of the stories you read in there are just not going to be relevant. Cocoa tariffs in some far flung place that are influencing the price of hot chocolate just aren't going to be relevant to your database SaaS, though it's certainly an interesting article, and that's the kind of interesting but not really relevant article that the economist often has.
Also, generally the Financial Times is the better standard for strictly business related journalism AFAIK, maybe they're neck-in-neck.
Spending 30 minutes to learn from someone in your field about how to get your next lead or which part of the internet you should be advertising to is a much better use of your time.
Also, generally the Financial Times is the better standard for strictly business related journalism AFAIK, maybe they're neck-in-neck.
Spending 30 minutes to learn from someone in your field about how to get your next lead or which part of the internet you should be advertising to is a much better use of your time.
This is my favorite comment and I totally agree with you and more you talk to different people the more you discover and that discovery leads to some of the best ideas.
Just curious on what’s the best way to do market research for the product
Just curious on what’s the best way to do market research for the product
Just curious on what’s the best way to do market research for the product
But also good to "sanity check" your instincts by checking in with people whom you respect and whom you think will understand the market better than you (which is what my parent comment is trying to get at).
Hey
I totally agree with you and thanks for the advise I really appreciate as I’m currently early in my career and have been thinking about alot of ideas just need to put those ideas into execution and try to see what works and what doesn’t
I really like the Economist, but I cannot possibly imagine one’s decision to read it or not has any bearing on entrepreneurial success.
I learned this bit of insight from you directly several years ago and reflect on it often. The first thing you did when we sat down was start grilling me on my views of tech trends and emerging areas of innovation. I was caught a bit off guard and gave terrible answers, but adopting that attitude of curiosity has helped me tremendously to find my market and grow my business.
Nice! I agree that curiosity is an incredibly valuable trait, especially in an entrepreneur.
(In fact, one of our company values at TimescaleDB is, "Be a student in both work and life".)
BTW - who is this? My apologies, I don't recognize the username :-)
(In fact, one of our company values at TimescaleDB is, "Be a student in both work and life".)
BTW - who is this? My apologies, I don't recognize the username :-)
I learned this at a young age: "The reporter follows the actor." Which are you?
Agreed, that part struck me as odd. I think it's making the job more intellectual than it is.
I would add to your list: spend as much time as possible with customers and figure out the actual pain.
I would add to your list: spend as much time as possible with customers and figure out the actual pain.
Database startups suffer from an extreme case of an important truism in startups generally: the product you think you are selling is not the product the customer thinks they are buying. No one buys a database per se, they are just a means to some other end.
People that love database technology — that would be me — tend to start database companies. It is very difficult to sell a database. It is much, much easier to sell a compelling solution to a somewhat boring but very valuable business problem that just happens to require an amazing database capability behind it. That’s your moat, the customer doesn’t actually care that there is an amazing database engine behind it but it makes it difficult for competitors to replicate.
Filed under “lessons I learned the hard way”.
People that love database technology — that would be me — tend to start database companies. It is very difficult to sell a database. It is much, much easier to sell a compelling solution to a somewhat boring but very valuable business problem that just happens to require an amazing database capability behind it. That’s your moat, the customer doesn’t actually care that there is an amazing database engine behind it but it makes it difficult for competitors to replicate.
Filed under “lessons I learned the hard way”.
This is basically our play. We took SQLite and wrapped a very compelling business app around it. None of our customers give a single shit that we use SQLite vs SQL Server, aside from passing comments on the IT deep dive calls.
"You don't need us to install any database software?"
"nah we have our own integrated data persistence mechanism".
"Cool. So about those firewall rules..."
That is about the extent to which our customers care. They are more interested in what actually shows up on their display and how stable the solution is than any particular technology choices we decided to make on their behalf.
No one wants to be responsible for picking a database engine. Pick it for your customers. That's part of the engineering.
"You don't need us to install any database software?"
"nah we have our own integrated data persistence mechanism".
"Cool. So about those firewall rules..."
That is about the extent to which our customers care. They are more interested in what actually shows up on their display and how stable the solution is than any particular technology choices we decided to make on their behalf.
No one wants to be responsible for picking a database engine. Pick it for your customers. That's part of the engineering.
Out of all the database products out there, I probably trust SQLite the most to do what I ask of it.
Joel Spolsky wrote about this quite a few times, Architecture astronauts take over
"What is it going to take for you to get the message that customers don’t want the things that architecture astronauts just love to build. The people? They love twitter. And flickr and delicious and picasa and tripit and ebay and a million other fun things, which they do want, and this so called synchronization problem is just not an actual problem, it’s a fun programming exercise that you’re doing because it’s just hard enough to be interesting but not so hard that you can’t figure it out."
https://www.joelonsoftware.com/2008/05/01/architecture-astro...
"What is it going to take for you to get the message that customers don’t want the things that architecture astronauts just love to build. The people? They love twitter. And flickr and delicious and picasa and tripit and ebay and a million other fun things, which they do want, and this so called synchronization problem is just not an actual problem, it’s a fun programming exercise that you’re doing because it’s just hard enough to be interesting but not so hard that you can’t figure it out."
https://www.joelonsoftware.com/2008/05/01/architecture-astro...
On the one hand I usually love to drop in a bit of spolsky wisdom myself, and that quote stands well on its own, but the article itself is unfortunately marred by the fact the Microsoft products mentioned were just unsuccessful early attempts at iCloud and Google Accounts, both of which have since seen considerable "killer app" level success. I guess easy with hindsight to say "ah yes, but smartphones".
And the return to timesharing systems.
[deleted]
Is that a everything-new-is-old description for cloud offerings?
I once saw a fascinating quote that went something like the following:
"As computing technology develops it becomes more efficient to take centralised computing resources and distribute them closer to the user. As network technology develops it becomes more efficient to centralise them again. Further advances redistribute and yet further advances recentralise. This pattern has been noticed several times in the history of computation."
And the most fascinating thing was that it was from decades ago, perhaps even 1960! I've never been able to find the quote again. Does anyone recognise it? Perhaps I imagined it.
"As computing technology develops it becomes more efficient to take centralised computing resources and distribute them closer to the user. As network technology develops it becomes more efficient to centralise them again. Further advances redistribute and yet further advances recentralise. This pattern has been noticed several times in the history of computation."
And the most fascinating thing was that it was from decades ago, perhaps even 1960! I've never been able to find the quote again. Does anyone recognise it? Perhaps I imagined it.
I dont know the exact quoter but the phenomena has been noticed and commented on maaaany times; I remember reading dilbert jokes about it in the 90s.
Relevant piece of history on timesharing in relation to cloud computing by the legend that is Brian Kernighan:
https://youtu.be/O9upVbGSBFo?t=340 (05:40 - 10:50)
https://youtu.be/O9upVbGSBFo?t=340 (05:40 - 10:50)
Indeed, with the browser being a replacement for XWindows, RDP and even SSH/Telnet.
You mean the thing that is literally just managed hosting rebranded with new shiny?
> paying untenable salaries to kids with more ultimate frisbee experience than Python, whose main job will be to play foosball in the googleplex and walk around trying to get someone…anyone…to come see the demo code they’ve just written with their “20% time,” doing some kind of, let me guess, cloud-based synchronization… between Microsoft and Google the starting salary for a smart CS grad is inching dangerously close to six figures and these smart kids, the cream of our universities, are working on hopeless and useless architecture astronomy because these companies are like cancers, driven to grow at all cost,
> ...the product you think you are selling is not the product the customer thinks they are buying.
Your comment reminds me of April Dunford's post on how she helped position a database product: https://www.thefxck.com/interviews/product-positioning-april...
Your comment reminds me of April Dunford's post on how she helped position a database product: https://www.thefxck.com/interviews/product-positioning-april...
Beautiful, this is exactly what's been on my mind for a while. After building a ton of "cool" infrastructure technology, I got recommendations from friends to sell it. But in the back of my mind somewhere I had going on what you've now made clear - that people won't actually want to _buy_ that technology, they'd much rather buy the business-specific stuff that is made possible _because_ of that technology.
How’s Snowflake doing?
One thing these database companies have going for themselves, is that once a customer company’s data and code is locked in to a particular database, then it becomes very very hard to get out. Not impossible, but just very difficult.
One thing these database companies have going for themselves, is that once a customer company’s data and code is locked in to a particular database, then it becomes very very hard to get out. Not impossible, but just very difficult.
I've heard this line many times but is this actually true? Are there stories or statistics of these things?
Even the migrations that appear small end up being huge. Over a decade ago, I was working for a large biotech company, which at the time was having trouble with their Oracle installation. The technical team, being very conservative, and having never worked anywhere else before, decided that the easiest way out was to purchase a large Exadata machine, as that was the less risky proposition: It's still supposed to be the same Oracle they knew and loved, except faster, and in far bigger iron than they were running before. So they ignored the scary price tag, believing that they were going to save a mint in recreating their data store.
Well, things weren't this easy. It was Oracle alright, so all the queries, triggers and views still worked... except the performance characteristics were completely different, because Exadata does all kinds of interesting things to extract more performance. So all the hints, manual query plans? just downright wrong, and often slower than before. In the end, easily a third of the queries needed rewrites, plus all the internal tuning changes that the very large DBA team did to make queries perform well. The whole effort cost many thousands of man months, on top of the oracle hardware, and the never cheap oracle license.
So even a migration from Oracle to a different flavor of Oracle can end up freezing a department with hundreds of tech workers for a year, and this was considered to be the cheapest choice! Imagine how much fun this would be if it was a large lift that, say, went from a RDBMS to NoSQL.
Well, things weren't this easy. It was Oracle alright, so all the queries, triggers and views still worked... except the performance characteristics were completely different, because Exadata does all kinds of interesting things to extract more performance. So all the hints, manual query plans? just downright wrong, and often slower than before. In the end, easily a third of the queries needed rewrites, plus all the internal tuning changes that the very large DBA team did to make queries perform well. The whole effort cost many thousands of man months, on top of the oracle hardware, and the never cheap oracle license.
So even a migration from Oracle to a different flavor of Oracle can end up freezing a department with hundreds of tech workers for a year, and this was considered to be the cheapest choice! Imagine how much fun this would be if it was a large lift that, say, went from a RDBMS to NoSQL.
I concur. I was project manager for a migration from Microsoft SQL Server to Azure SQL. There were all sorts of problems, performance being one of them.
My team built our product on top of a very capable NoSQL database called MarkLogic. It had issues, but also had a lot of things it did well.
But then the original licensing agreement we had negotiated was coming to an end, and the new deal was going to be massively more expensive.
So we used this as an opportunity to rearchitect our monolith into services built on other open source data stores, that also fixed a lot of the architectural issues and technical debt we had acquired.
So to answer your question, yes it’s possible to move off of a database, if the financial incentives are great enough.
But then the original licensing agreement we had negotiated was coming to an end, and the new deal was going to be massively more expensive.
So we used this as an opportunity to rearchitect our monolith into services built on other open source data stores, that also fixed a lot of the architectural issues and technical debt we had acquired.
So to answer your question, yes it’s possible to move off of a database, if the financial incentives are great enough.
One anecdote: Consider how long it took Amazon to get off of Oracle. They have all the engineering talent, they have a huge budget, they had all the political will, and took ages still for them to migrate.
There is a reason why IBM still makes a lot of money on IMS.
Database refactoring tend to measure in millions of dollars and quarters-years from a consulting perspective.
Some context: this is for expansive legacy application databases that have many years of tenure or large data warehouse applications. If you've ever worked in the enterprise data space, you'll be tripping over this sort of workload left and right.
Some context: this is for expansive legacy application databases that have many years of tenure or large data warehouse applications. If you've ever worked in the enterprise data space, you'll be tripping over this sort of workload left and right.
I've done several database migrations, and it's true, but not because the data is hard to extract. It's moderately easy to dump out CSV or JSON files to S3 and load them back in to another database (only moderately because you have to deal with fiddly encoding issues). So in that sense the data is easily extracted. Tools like AWS Data Migration Services, while not perfect, can also make this a lot easier.
However, at least three issues make a database migration non-trivial:
1. Custom data types that have to be replicated somehow in the new database; this usually involves figuring out how to parse the output representation in a sensible way.
2. User-defined triggers and functions, or custom database features, that may not be available on the new database.
3. There is usually a lot of infrastructure built on top of a database that can be hard to switch over, like if you've used roles substantially in Postgres for access control. Not to mention any business intelligence tool built on it that tends to have a lot of hand-rolled SQL. For example, Periscope can be database-agnostic, but any queries you write might be customized for one particular database vendor.
So in my view, it's not really data concerns (except case 1); there are a lot of operational concerns that make database migrations hard.
However, at least three issues make a database migration non-trivial:
1. Custom data types that have to be replicated somehow in the new database; this usually involves figuring out how to parse the output representation in a sensible way.
2. User-defined triggers and functions, or custom database features, that may not be available on the new database.
3. There is usually a lot of infrastructure built on top of a database that can be hard to switch over, like if you've used roles substantially in Postgres for access control. Not to mention any business intelligence tool built on it that tends to have a lot of hand-rolled SQL. For example, Periscope can be database-agnostic, but any queries you write might be customized for one particular database vendor.
So in my view, it's not really data concerns (except case 1); there are a lot of operational concerns that make database migrations hard.
4. Egress costs from the clouds are non-trivial for large datasets. So now you're locked into both your db and your cloud.
Its not difficult at all. We just moved bunch of usecases from snowflake to pinot. All I had to do was s3 export from snowflake and import into pinot.
I don’t know if this is a corollary, but when I try to come up with an example of someone getting rich selling databases, I think of companies known more for their sales than their engineering.
Or for their predatory anticompetitive behaviour?
If you're thinking of the same company I'm thinking of, I thought it was better known for its legal department than its sales department.
This is exactly why Oracle in the 90s and 00s starting buying up software Application companies (Peoplesoft, JD Edwards, Hyperion, etc).
And then in the old on-premise world, you sell them both the App and require the use of your database which is an additional sale.
This strategy doesn’t work in a Cloud world though. Since customers are no longer buying the individual components (like the database) but are instead just leasing the finished good. By definition, they don’t care what’s under the hood - unless you’re selling to another software company who’s using your database to make their finished good.
And then in the old on-premise world, you sell them both the App and require the use of your database which is an additional sale.
This strategy doesn’t work in a Cloud world though. Since customers are no longer buying the individual components (like the database) but are instead just leasing the finished good. By definition, they don’t care what’s under the hood - unless you’re selling to another software company who’s using your database to make their finished good.
IIRC, Oracle managed to get a huge, well paying customer (DoD) behind their "we build a better database" pitch.
That provided enough funding, users and support to move into applications. In fact even if they didn't expand into other areas, that support could have kept them as a profitable DB company for decades.
That provided enough funding, users and support to move into applications. In fact even if they didn't expand into other areas, that support could have kept them as a profitable DB company for decades.
Was this in their earliest days? Reminds me of what I read recently about IBM getting a significant contract from government, but even more importantly benefits of research, in early days of computing when they were actually behind. After that they got so far ahead they stayed the leader until the industry changed.
Ten years ago, when I still worked in consulting, all my clients (large institutional companies) used Oracle and they'd say they were willing to pay the premium because Oracle was 20% faster than everyone else.
This the key. I'm also a dreamer of a better RDBMs (https://tablam.org) and wish I could live doing it... but working on the sector of small business what they want is a better access/excel. Probably that is what bigger companies want too.
So the internal tech is just a mean to make that possible. I think if a db engine provide the equivalent of the auto-admin of Django it will sell itself easily :)
However this is also hard because DBs engines touch several complex things (storage, concurrency, compilers, interpreters, etc) and is hard to find people for this plus the funding.
So the internal tech is just a mean to make that possible. I think if a db engine provide the equivalent of the auto-admin of Django it will sell itself easily :)
However this is also hard because DBs engines touch several complex things (storage, concurrency, compilers, interpreters, etc) and is hard to find people for this plus the funding.
There's a bit of a chicken and egg problem in that until you have compelling products demonstrating how your “better” RDBMS contributes to better business solutions, there's not something the technical people who understand the theoretical advantages can take to the less technical people that need to approve the decision to justify it over established RDBMSs, that are not only seen as more secure from a business perspective but also easier to hire qualified admins, etc., for.
Yeah, is like build a OS or in fact any infrastructure project. The key is have a clear north so the project know what need to nail at first try...
Yes. The spatial database startups I've had experience with, e.g. CartoDB (now Carto) and GeoSpock, transitioned from selling database tech to selling turnkey GIS (Geographical Information Systems) solutions, often with industry specific specialisations.
On the other hand, any kind of technology in infrastructure is a prime candidate for open source, the incentives are just very much aligned there.
Infrastructure tech makes for a great case that helps everyone while giving nobody a competitive advantage, that's why companies love doing or sponsoring it.
On the other hand, open source tech in infrastructure helps get rid of vendor lock-in as in the worst case, you could still look inside the source code and patch things — also fancy FOSS tech makes for good community building efforts and shiny job ads, both attracting good devs.
Both effects together explain to me why open source makes for such a fierce competitor in any kind of paid infra tech, it's hard to build the momentum to outrun them.
And open core? Pretty much dead by now since as soon as the cloud providers see there's traction around any technology, they will build a hosted version, either directly or in a protocol compatible fashion.
Techies really dig this stuff (I do too), but as a business model, there aren't many worse ones in 2021.
Infrastructure tech makes for a great case that helps everyone while giving nobody a competitive advantage, that's why companies love doing or sponsoring it.
On the other hand, open source tech in infrastructure helps get rid of vendor lock-in as in the worst case, you could still look inside the source code and patch things — also fancy FOSS tech makes for good community building efforts and shiny job ads, both attracting good devs.
Both effects together explain to me why open source makes for such a fierce competitor in any kind of paid infra tech, it's hard to build the momentum to outrun them.
And open core? Pretty much dead by now since as soon as the cloud providers see there's traction around any technology, they will build a hosted version, either directly or in a protocol compatible fashion.
Techies really dig this stuff (I do too), but as a business model, there aren't many worse ones in 2021.
Cloud providers are proprietary infrastructure and their rapid growth shows the opposite.
SQL Server, Oracle just led to AWS DynamoDB/Aurora. You don't have any source code for either. Open-source projects growing in popularity doesn't override the greater trend that companies just want solutions, whether it's open or not.
SQL Server, Oracle just led to AWS DynamoDB/Aurora. You don't have any source code for either. Open-source projects growing in popularity doesn't override the greater trend that companies just want solutions, whether it's open or not.
And if anything, the cloud is an even bigger moat. Which would you rather have to explain to yourself CTO as a cause of outage:
AWS had a problem or “Hosted Super Open Source Database” had a problem.
There is a reason why there is an old adage: “No one got fired for going with (IBM/Microsoft/AWS)”
AWS had a problem or “Hosted Super Open Source Database” had a problem.
There is a reason why there is an old adage: “No one got fired for going with (IBM/Microsoft/AWS)”
Some incentives are aligned for open source data infrastructure but others are clearly misaligned. In practice we just get a different type of less-than-great outcome. Open source data infrastructure is not a solved problem.
A question that lacks a satisfactory answer in open source data infrastructure is this: who is going to pay for the initial architecture to be state-of-the-art? Architecture is forever, if you start with a naive design it will put the platform at a long-term disadvantage. Most open source infrastructure projects are started by well-meaning people that have no idea what a state-of-the-art design even looks like. At the periphery you might be able to burn a giant pile of VC money to make it happen but that is not sustainable without a clear path to profitable exit. The set of people with the critical expertise and the set of people building open source data infrastructure are nearly disjoint.
Today, the money is in state-of-the-art bespoke data infrastructure. If the tradeoff is "open source" and "10-100x better on infrastructure KPIs", companies happily pay millions for the latter. People that know how to design this infrastructure are extremely well paid, 7-figures is common and demand is high. Yes, they could take a year off work and create an open source infrastructure project but this doesn't seem to happen and it is easy to understand why, as there are few incentives and many disincentives. This is roughly my area of business, I see the dynamics from the inside and the trend is not favoring open source.
If all of the cutting edge infrastructure development work is happening in closed source, whether cloud or state-of-the-art bespoke, it doesn't bode well for the long-term relevance of open source.
A question that lacks a satisfactory answer in open source data infrastructure is this: who is going to pay for the initial architecture to be state-of-the-art? Architecture is forever, if you start with a naive design it will put the platform at a long-term disadvantage. Most open source infrastructure projects are started by well-meaning people that have no idea what a state-of-the-art design even looks like. At the periphery you might be able to burn a giant pile of VC money to make it happen but that is not sustainable without a clear path to profitable exit. The set of people with the critical expertise and the set of people building open source data infrastructure are nearly disjoint.
Today, the money is in state-of-the-art bespoke data infrastructure. If the tradeoff is "open source" and "10-100x better on infrastructure KPIs", companies happily pay millions for the latter. People that know how to design this infrastructure are extremely well paid, 7-figures is common and demand is high. Yes, they could take a year off work and create an open source infrastructure project but this doesn't seem to happen and it is easy to understand why, as there are few incentives and many disincentives. This is roughly my area of business, I see the dynamics from the inside and the trend is not favoring open source.
If all of the cutting edge infrastructure development work is happening in closed source, whether cloud or state-of-the-art bespoke, it doesn't bode well for the long-term relevance of open source.
The unspoken alternative in articles like this is that they don't know why they failed, or their view of the market is just not clear enough to form a judgement.
The industry is still in an awkward position where software deals in 'intellectual property', but the concept was never really developed with an understanding of whatever it is that software is. Something like PostgreSQL for example isn't exactly a product, a service or a novel idea. Most of its consumers don't use most of its features. It is almost a perspective on a problem and some codified good design ideas.
It isn't obvious if selling a perspective is profitable. There are a lot of winners in software that don't actually sell software. Eg, Facebook/Google sells eyeballs, Amazon sells infrastructure, Apple sells iPhones.
MongoDB.Inc probably doesn't sell 'a database' if the details of their customer relationships were open for inspection.
The industry is still in an awkward position where software deals in 'intellectual property', but the concept was never really developed with an understanding of whatever it is that software is. Something like PostgreSQL for example isn't exactly a product, a service or a novel idea. Most of its consumers don't use most of its features. It is almost a perspective on a problem and some codified good design ideas.
It isn't obvious if selling a perspective is profitable. There are a lot of winners in software that don't actually sell software. Eg, Facebook/Google sells eyeballs, Amazon sells infrastructure, Apple sells iPhones.
MongoDB.Inc probably doesn't sell 'a database' if the details of their customer relationships were open for inspection.
Indeed there's always some secret sauce in finding product-market fit - but it's always specific to every product! One can only discover it by toiling hard with customers and be opened for unbias evaluation of inputs coming from all the sides. There's no guarantee that you'll discover it for your product before you run out of the inner drive to toil.
This is akin to everyone’s dream to start a cozy, atmospheric coffee shop. These types of coffee bars fold very quickly or at best give founders years of servitude below minimum wage. The reason is that the type of behaviour this kind of establishment encourages - lounging, book reading, laptop work is exactly the opposite of the quick serve model that is conducive to high revenue. The customer loves this model, the owner hates it.
Analogous to it, the nosql DB market is the exact wrong market to enter. Large companies willing to pay big bucks for enterprise features will pay established vendors for the stodgy but battle tested stuff like Oracle or DB2. The hipster startup market pays nobody for anything as there is a myriad of free choices in every common flavour and the few that will pay will mostly do so to purchase managed hosting. And that’s only if their PoC built on your new and untrusted database ever makes it to production. And then it’s probably your cheapest tier. Don’t be selling to paupers!
Analogous to it, the nosql DB market is the exact wrong market to enter. Large companies willing to pay big bucks for enterprise features will pay established vendors for the stodgy but battle tested stuff like Oracle or DB2. The hipster startup market pays nobody for anything as there is a myriad of free choices in every common flavour and the few that will pay will mostly do so to purchase managed hosting. And that’s only if their PoC built on your new and untrusted database ever makes it to production. And then it’s probably your cheapest tier. Don’t be selling to paupers!
Interesting, I think this is why private clubs used to be a thing and may be the only way to get a sustainable cozy, atmospheric coffee shop. Because really that's what you're selling, not coffee, but atmosphere. And membership is a way to charge for your real product.
> private clubs used to be a thing
They are absolutely still a thing in every major city.
They are absolutely still a thing in every major city.
Or charge for time, like Ziferblat do: https://ziferblat.co.uk/
Also Moscow's Catbureau, which has more cats: https://www.youtube.com/watch?v=Ikfbv1enCME
That place looks lovely.
Offering open source software as a service with support seemed like a solution to this problem for a while. Hipsters use your stuff for free, but then pay for support and hosted offerings once they have more money than time.
But AWS is killing this model by just offering popular open source products as a service themselves, cutting out the original developments.
But AWS is killing this model by just offering popular open source products as a service themselves, cutting out the original developments.
> The customer loves this model, the owner hates it.
Only if the owner was trying to sell Coffee in the first place, which allegedly isn't Starbucks' main business: https://archive.is/9TZej
Only if the owner was trying to sell Coffee in the first place, which allegedly isn't Starbucks' main business: https://archive.is/9TZej
> People wanted RethinkDB to be fast on workloads they actually tried, rather than “real world” workloads we suggested
Uh no, It was slow on real world workloads, you just don't want to admit it.
Uh no, It was slow on real world workloads, you just don't want to admit it.
I think that's his point. That's why he put "real world" in scare quotes.
I dont think so because he could have said it plainly since it was generally slow, but instead he implied that the use cases users were trying weren't "real world" use cases that fit within what a database of its class should be able to do. Hes not being totally honest about the performance issues and how much of an impact it had on their failure.
[deleted]
> Developers love building developer tools, often for free. So while there is massive demand, the supply vastly outstrips it.
This is key, and plays out over and over again in different forms. There are no points for difficulty, only supply and demand. PG puts this well [1]:
> That's the essence of a startup: having brilliant people do work that's beneath them. Big companies try to hire the right person for the job. Startups win because they don't—because they take people so smart that they would in a big company be doing "research," and set them to work instead on problems of the most immediate and mundane sort. Think Einstein designing refrigerators.
[1] http://www.paulgraham.com/bronze.html
This is key, and plays out over and over again in different forms. There are no points for difficulty, only supply and demand. PG puts this well [1]:
> That's the essence of a startup: having brilliant people do work that's beneath them. Big companies try to hire the right person for the job. Startups win because they don't—because they take people so smart that they would in a big company be doing "research," and set them to work instead on problems of the most immediate and mundane sort. Think Einstein designing refrigerators.
[1] http://www.paulgraham.com/bronze.html
So what happens then when a polished open source solution solves for a problem that developer tools cannot solve for? That speaks nothing of demand and only signals that there is no alternative supply. It also speaks nothing to the boringness of the hypothetical solution.
I suspect by asking that question any reader starts immediately scratching their head thinking up what problem/solution that could possibly be. Don’t. That confuses product for business.
I suspect by asking that question any reader starts immediately scratching their head thinking up what problem/solution that could possibly be. Don’t. That confuses product for business.
Just because people want a super-wham-o-dyne tool doesn't mean that they are willing to pay for one. The cases of awesome well-polished open-source tools are frequently cases where the market wasn't willing to pay for them. One or few developers wanted it enough to build and release it for free, so they did it anyway.
So yeah, that is speaking of demand. Economically-relevant demand.
So yeah, that is speaking of demand. Economically-relevant demand.
We're experiencing this a bit with our startup, except we're not building developer tools, we've built a (semi-personal) knowledge management platform.
In the era of COVID, it seems like every developer with some extra free time has decided that they want to build their own PKM/note-taking/etc app, so competition has scaled up massively in the last year.
In the era of COVID, it seems like every developer with some extra free time has decided that they want to build their own PKM/note-taking/etc app, so competition has scaled up massively in the last year.
> it seems like every developer with some extra free time has decided that they want to build their own PKM/note-taking/etc app, so competition has scaled up massively in the last year.
Still none with dark mode, native apps, markdown.
Still none with dark mode, native apps, markdown.
What’s with all of this seemingly whitewashing of history? Slava himself described why it failed. I mean as founder you have to be laser focused. Yet he was too busy trying to win over his “hot new employee” than focusing on the database.
There is an archive of his deleted tweets here where he describes it: https://gist.github.com/travisbrown/059310042193a2e143408b05...
And then there was an article that seems to be gone from the web (https://web.archive.org/web/20201130215752/https://threader....). It stands to reason this huge distraction played a role in the mismanagement of the database and why it ultimately didn’t succeed.
There is an archive of his deleted tweets here where he describes it: https://gist.github.com/travisbrown/059310042193a2e143408b05...
And then there was an article that seems to be gone from the web (https://web.archive.org/web/20201130215752/https://threader....). It stands to reason this huge distraction played a role in the mismanagement of the database and why it ultimately didn’t succeed.
lol i didnt realise those tweets were deleted. that guy usually doubles down.
Eh, I think you’re way too quick to criticise him for it. He’s human, after all. And in love on top of it!
Personally, I appreciate the frankness. Too bad it got deleted.
Personally, I appreciate the frankness. Too bad it got deleted.
What a romantic take on a failure of fiduciary duty towards ones employees and investors. Imagine being an employee there and thinking you’re going to change databases forever only to have the founder being distracted and then the whole outfit failing.
Um, you're obviously lying in this post here, contradicting the source material you link to.
People write online way too much about their personal lives.
Bizarre. I don't know how much of a distraction "I hired a hot girl and then fell in love with her" was, but to me tweeting 40-60 times per day says more about a lack of focus (e.g. 56 times on 30th November 2020 where that tweet came from). The deleted tweets don't go back to 2017 but if that level of tweeting is normal I wonder where he found time to do anything, let alone build a database company.
That said, I'm wasting time on HN currently. But I'm also on holiday.
That said, I'm wasting time on HN currently. But I'm also on holiday.
Perhaps I was keying of the wrong thing. I think you’re 100% right.
RethinkDB was great. They had better tech than mongodb. It was fast and consistent, rql was expressive and easy to lean. Sharding and replicas was easy.
I wish they didn’t run out of money and give up. In many ways startups feel like you gotta be a cockroach. The goal is to not die and live long enough to be profitable where you control your own destiny.
Like mongo atlas, rethinkdb’s hosted solution (horizon I think) could have easily taken off.
Like Firebase’s Firestore, it could have been quite successful.
I wish they didn’t run out of money and give up. In many ways startups feel like you gotta be a cockroach. The goal is to not die and live long enough to be profitable where you control your own destiny.
Like mongo atlas, rethinkdb’s hosted solution (horizon I think) could have easily taken off.
Like Firebase’s Firestore, it could have been quite successful.
I distinctly remember their creative hiring tactics:
They anonymously floated mysterious puzzles across the web. The solution was a domain name which had a static website yielding an email address.
If you sent a email to that address they immediately asked you if you were interested in working for them. I know, because I solved one (The lure to solve the mystery was just too tempting to resist)
Google did this in 2002(?). I remember billboards in downtown Seattle with no text or logo other than [math problem].com.
Did you also end up working there? Or just solved the puzzle, got the offer but refused?
I declared to be not interested. Just solved the puzzle to see what was behind that door.
This is gold. I have seen a lot of "cool" startups that failed, and "not-so-cool" startups which do the dirty work for their customers so that their customer don't have to succeeded.
RDB is fantastic, I’m using it today.
You are right! IMHO RDB is a good piece of software. I'm using it on one side project and it is working good for all my needs
[deleted]
If the customer relationship is sticky and product/technology improves over time, it is more important to be quick than "mature" or "feature complete". When the technology matures (roughly simultaneously for all competitors), the size of the user base is what matters.
It failed because megacorps didn't like the idea of a database which could allow startups to scale from zero to hundreds of millions of users out of the box. Anyone who's had any affiliation with projects which aim for 'out of the box' scalability will know that these projects are actively suppressed and starved of funding and customers by corporate interests.
I was baffled when they went down.
MongoDB and ArangoDB did well so there seems to be a valid case for new databases. Rethink was loved by HN commenters. I used it in a startup I was working at in 2015-2017 and it was a pleasure.
MongoDB and ArangoDB did well so there seems to be a valid case for new databases. Rethink was loved by HN commenters. I used it in a startup I was working at in 2015-2017 and it was a pleasure.
I was under the impression that rethink db was not open source (until after it “failed”).
That’s not a reason for failure, of course, but I have mentioned before that we (as in developers) often only choose things with permissive licenses, often to the detriment of the products we “support” (see the recent elastic drama, where we blamed elastic being forked by Amazon for being too permissive!).
That’s not a reason for failure, of course, but I have mentioned before that we (as in developers) often only choose things with permissive licenses, often to the detriment of the products we “support” (see the recent elastic drama, where we blamed elastic being forked by Amazon for being too permissive!).
It was open source from the start, but under AGPL which is a fairly restrictive open source license. I remember diving through the code trying to debug bugs in RethinkDB when using it in production long before they went out of business. See my blog post http://sagemath.blogspot.com/2016/10/rethinkdb-must-relicens...
Yeah, AGPL seems to be one of those licenses that aren’t popular here. I’ve never understood it myself - especially for something like a db which folks are generally less likely to modify.
What would you say is the problem with AGPL? I was thinking of releasing a product under it myself.
What would you say is the problem with AGPL? I was thinking of releasing a product under it myself.
So much of this is familiar. I was at two database startups.
The first one, an object-oriented database company (last 80s, early 90s) suffered from an extreme case of misperceived markets, due to the occasionally intentional blurring of the lines between a "database system" (which means SQL to nearly everyone, and we didn't to SQL), and a persistent storage class added to C/C++ (and later Smalltalk and Java).
The second one (mid 2000s) was a neat physical storage trick, which should have been a feature of a database system, not an excuse for building a new one. My bad for missing this, but I really, really liked the physical storage ideas, so I joined.
Building and selling new database technology is extremely difficult. There are successful, huge, entrenched companies. There is, therefore, no good reason for anyone to gamble on your new technology. Your only chance is to convince the architecture astronauts at some prospect that they just have to have your product, but the odds of such a decision sticking all the way through the delivery of their product is extremely low, no matter how good your technology is.
The first one, an object-oriented database company (last 80s, early 90s) suffered from an extreme case of misperceived markets, due to the occasionally intentional blurring of the lines between a "database system" (which means SQL to nearly everyone, and we didn't to SQL), and a persistent storage class added to C/C++ (and later Smalltalk and Java).
The second one (mid 2000s) was a neat physical storage trick, which should have been a feature of a database system, not an excuse for building a new one. My bad for missing this, but I really, really liked the physical storage ideas, so I joined.
Building and selling new database technology is extremely difficult. There are successful, huge, entrenched companies. There is, therefore, no good reason for anyone to gamble on your new technology. Your only chance is to convince the architecture astronauts at some prospect that they just have to have your product, but the odds of such a decision sticking all the way through the delivery of their product is extremely low, no matter how good your technology is.
As someone using RethinkDB in production for the last 6 years or so: I am really disappointed by how software development world is dominated by fashions and fads.
RethinkDB was done really, really well. It is one of the very few distributed databases that went through Jepsen relatively unscathed and delivered on promises made. Development was done in the public, questions were asked through StackOverflow. You interacted with competent, skillful and experienced developers.
And yet MongoDB was the latest fashion fad, end even though it did NOT deliver on the promises, it was the hot-database-du-jour that kids used.
The article is mostly about the business model, and I also always thought they would have a hard time making money on the database. But I think they would have had a much better shot if it became more popular. It didn't, which accelerated the company's demise.
EDIT: I just realized that I wrote about RethinkDB in the past tense, even though it very much exists as I write it. In fact I run my business on it. But because it fell out of favor, when it was open-sourced, it failed to pick up momentum, and it now seems unlikely that it will.
Which is why I'm working on switching to FoundationDB, and I'm slightly worried that it will suffer the same fate: it is excellent technically (the best transactional guarantees in a distributed database you can get), but difficult to understand and not very user-friendly. It's not the "node.js database for everyone". The only reason I'm considering it is because Apple uses and develops it, which gives me hope for longer-term maintenance.
Going back to fashions — you can have a product which excels technically, but if it's out of fashion, it might as well not exist.
I think we would all be better off if we stopped trying to always pick The One True Database, The One True Programming Language, etc — and instead accepted that there might be multiple tools, each specialized for certain kinds of tasks.
RethinkDB was done really, really well. It is one of the very few distributed databases that went through Jepsen relatively unscathed and delivered on promises made. Development was done in the public, questions were asked through StackOverflow. You interacted with competent, skillful and experienced developers.
And yet MongoDB was the latest fashion fad, end even though it did NOT deliver on the promises, it was the hot-database-du-jour that kids used.
The article is mostly about the business model, and I also always thought they would have a hard time making money on the database. But I think they would have had a much better shot if it became more popular. It didn't, which accelerated the company's demise.
EDIT: I just realized that I wrote about RethinkDB in the past tense, even though it very much exists as I write it. In fact I run my business on it. But because it fell out of favor, when it was open-sourced, it failed to pick up momentum, and it now seems unlikely that it will.
Which is why I'm working on switching to FoundationDB, and I'm slightly worried that it will suffer the same fate: it is excellent technically (the best transactional guarantees in a distributed database you can get), but difficult to understand and not very user-friendly. It's not the "node.js database for everyone". The only reason I'm considering it is because Apple uses and develops it, which gives me hope for longer-term maintenance.
Going back to fashions — you can have a product which excels technically, but if it's out of fashion, it might as well not exist.
I think we would all be better off if we stopped trying to always pick The One True Database, The One True Programming Language, etc — and instead accepted that there might be multiple tools, each specialized for certain kinds of tasks.
The software development world isn't dominated by fashions and fads.
The segment of the market that chases fashions and fads is dominated by fashions and fads. Most development is just using something old and boring like Postgres or MySQL or MSSQL or Oracle.
The problem for RethinkDB is that when you take out the fashions and fads segment and the old and boring segment, there's not much left.
The segment of the market that chases fashions and fads is dominated by fashions and fads. Most development is just using something old and boring like Postgres or MySQL or MSSQL or Oracle.
The problem for RethinkDB is that when you take out the fashions and fads segment and the old and boring segment, there's not much left.
I recommend reading this 3-part series on Mongo's marketing - it goes into a lot of detail why Mongo "won". https://www.nemil.com/mongo/1.html
As a former RethinkDB user (we have migrated to Postgres) I actually don't miss it as much as I would - JSONB in PG does what we need, and the real-time features of RethinkDB never really delivered because of various performance issues in the database itself.
As a former RethinkDB user (we have migrated to Postgres) I actually don't miss it as much as I would - JSONB in PG does what we need, and the real-time features of RethinkDB never really delivered because of various performance issues in the database itself.
> As a former RethinkDB user (we have migrated to Postgres) I actually don't miss it as much
For me, JSON wasn't the important part. I wanted to have a distributed database, where a single server could disappear completely at any time and my customers would not lose any data.
There are remarkably few solutions that deliver this, though many claim to. Even fewer have been verified with Jepsen. And for all the hoopla about distributed computing using Kubernetes, I find it puzzling that so little emphasis is placed on data integrity.
Changefeeds have worked great for me as well, and I'm still trying to work out what to replace them with.
For me, JSON wasn't the important part. I wanted to have a distributed database, where a single server could disappear completely at any time and my customers would not lose any data.
There are remarkably few solutions that deliver this, though many claim to. Even fewer have been verified with Jepsen. And for all the hoopla about distributed computing using Kubernetes, I find it puzzling that so little emphasis is placed on data integrity.
Changefeeds have worked great for me as well, and I'm still trying to work out what to replace them with.
DON'T switch if it works! A working system has nothing todo with "favor" or "momentum".
This is life in general, not just software development.
You can compare selling a database to selling git. Nobody is selling git itself, they sell the hosting around the tech, or create a dev workflow (gitlab) that uses git at its core.
If anyone tried to sell git itself it would be impossible. I never used mongodb for reasons listed in the article. But there is free and open source and resilient postgres.
the companies who succeeded around git did not build git. it would be a tough thing to develop both git and the hosting / workflow business around it.
so I guess the lesson to learn is don't try to build extremely sophisticated software as a startup where there are already good enough open source alternatives.
If anyone tried to sell git itself it would be impossible. I never used mongodb for reasons listed in the article. But there is free and open source and resilient postgres.
the companies who succeeded around git did not build git. it would be a tough thing to develop both git and the hosting / workflow business around it.
so I guess the lesson to learn is don't try to build extremely sophisticated software as a startup where there are already good enough open source alternatives.
> To see how this plays out for other companies consider MongoDB (valued at roughly $1.6B with ~700 employees),
Maybe another, important, lesson the RethinkDB team left out was: raise more capital than you need, and go public fast.
Two years later, MongoDB the company is now valued at $19.491B. Like many other public tech companies, it is yet to turn a single profitable quarter, and is not expected to in the next few years.
Annual YoY revenue growth is flat too at around $500m. Equity is negative.
Perhaps because the market thinks useful product companies will be valuable vs. the US dollar in the future. And its a vital part of many companies infrastructure these days, without much of a solid enterprise alternative.
Maybe another, important, lesson the RethinkDB team left out was: raise more capital than you need, and go public fast.
Two years later, MongoDB the company is now valued at $19.491B. Like many other public tech companies, it is yet to turn a single profitable quarter, and is not expected to in the next few years.
Annual YoY revenue growth is flat too at around $500m. Equity is negative.
Perhaps because the market thinks useful product companies will be valuable vs. the US dollar in the future. And its a vital part of many companies infrastructure these days, without much of a solid enterprise alternative.
>Annual YoY revenue growth is flat too at around $500m.
2018: $154M
2019: $267M
2020: $421M
2021: $590M
Source: https://finance.yahoo.com/quote/MDB/financials?p=MDB
You're right, thanks. 40% YoY revenue growth at size is considerable.
I had such high hopes for RethinkDB as a free and open source alternative to Firebase. FB was a great tool, but was never the same after getting purchased by Google (I quit following its progress shortly thereafter).
This article has less to do with databases or open source software than it does with the fundamentally misaligned incentives of egalitarianism and capitalism.
I wish there was a social open source license that said "you can use this indefinitely as long as you pay us something". The price would be up to the user but the generally accepted polite minimum would be at least a penny ($0.01).
Then businesses could publicly display their level of support (both initially and yearly) for the software they use, to attract customers. They could also be audited by the IRS, so a lack of patronage could correlate to a lack of equality and maybe even reveal corruption and other malfeasance. At the very least it would reveal a lack of internal controls.
This could work kind of like UBI for open source. And I'm definitely not the only one who has ever thought about this. But there is just so much free capital floating around right now that it's a great time to be thinking about how to reform the paradigms that we all depend on.
This article has less to do with databases or open source software than it does with the fundamentally misaligned incentives of egalitarianism and capitalism.
I wish there was a social open source license that said "you can use this indefinitely as long as you pay us something". The price would be up to the user but the generally accepted polite minimum would be at least a penny ($0.01).
Then businesses could publicly display their level of support (both initially and yearly) for the software they use, to attract customers. They could also be audited by the IRS, so a lack of patronage could correlate to a lack of equality and maybe even reveal corruption and other malfeasance. At the very least it would reveal a lack of internal controls.
This could work kind of like UBI for open source. And I'm definitely not the only one who has ever thought about this. But there is just so much free capital floating around right now that it's a great time to be thinking about how to reform the paradigms that we all depend on.
So, is the Economist any good? I'm thinking HN is a better source and it's free.
Why is a 2017 article trending? Also isn't this the dude who had that 'this sounds like a parody but is actually how I hire engineers' article?
developer tool market seems to be the only one I'm familiar with as an engineer. If I were to start a business, I will probably also do something for this market.