Why is everything so scalable?(stavros.io)
stavros.io
Why is everything so scalable?
https://www.stavros.io/posts/why-is-everything-so-scalable/
382 comments
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing servers), but you absolutely need to configure and monitor your specific XaaS you’re renting.
Anyone that says, "they don’t have to manage infrastructure" I would invite them to deal with a multi-environment terraform setup and tell me again that about what they don't have to manage.
While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers.
Most are biased because they like dealing with the kind of issues in on premises.
They like dealing with the performance regressions, heat maps, kernel issues etc. Because why not? You are a developer and you need some way to exercise your skills. AWS takes that away and makes you focus on the product. Issues arising from AWS only requires you talking to support. Most developers get into this industry for the love of solving these problems and not actually solving product requirements.
AWS takes away what devs like and brings in more "actual" work.
Most are biased because they like dealing with the kind of issues in on premises.
They like dealing with the performance regressions, heat maps, kernel issues etc. Because why not? You are a developer and you need some way to exercise your skills. AWS takes that away and makes you focus on the product. Issues arising from AWS only requires you talking to support. Most developers get into this industry for the love of solving these problems and not actually solving product requirements.
AWS takes away what devs like and brings in more "actual" work.
> AWS takes that away and makes you focus on the product. Issues arising from AWS only requires you talking to support.
Not my experience at all. e.g. NLBs don't support ICMP which has broken some clients of the application I work on. When we tried to turn on preserve-client-ip so we could get past the ephemeral port limit, it started causing issues with MSS negotiation, breaking some small fraction of clients. This stuff is insanely hard to debug because you can't get onto the loadbalancer to do packet captures (nor can AWS support). Loadbalancing for long-lived connections works poorly.
Lambda runs into performance issues immediately for a web application server because it's just an entirely broken architecture for that use-case (it's basically the exact opposite of user-mode threads to scale: let's use an entire VM per request!). For some reason they encourage people to do it anyway. Lord help you if you have someone with some political capital in your org that wants to push for that.
RDS also runs into performance issues the moment you actually have some traffic. A baremetal server is orders of magnitude more capable.
ipv6 support is still randomly full of gaps (or has only very recently been fixed, except you might have to do things like recreate your production EKS cluster, oops) which leads to random problems that you have to architect around. Taken with NAT gateway being absurdly expensive, you end up having to invert sensible architectures or go through extra proxy layers that just complicate things.
AWS takes basic skills around how to build/maintain backend systems and makes half of your knowledge useless/impossible to apply, instead upgrading all of your simple tuning tasks into architectural design problems. The summary of my last few years has basically been working around problems that almost entirely originate from trying to move software into EKS and dealing with random constraints that would take minutes to fix baremetal.
Not my experience at all. e.g. NLBs don't support ICMP which has broken some clients of the application I work on. When we tried to turn on preserve-client-ip so we could get past the ephemeral port limit, it started causing issues with MSS negotiation, breaking some small fraction of clients. This stuff is insanely hard to debug because you can't get onto the loadbalancer to do packet captures (nor can AWS support). Loadbalancing for long-lived connections works poorly.
Lambda runs into performance issues immediately for a web application server because it's just an entirely broken architecture for that use-case (it's basically the exact opposite of user-mode threads to scale: let's use an entire VM per request!). For some reason they encourage people to do it anyway. Lord help you if you have someone with some political capital in your org that wants to push for that.
RDS also runs into performance issues the moment you actually have some traffic. A baremetal server is orders of magnitude more capable.
ipv6 support is still randomly full of gaps (or has only very recently been fixed, except you might have to do things like recreate your production EKS cluster, oops) which leads to random problems that you have to architect around. Taken with NAT gateway being absurdly expensive, you end up having to invert sensible architectures or go through extra proxy layers that just complicate things.
AWS takes basic skills around how to build/maintain backend systems and makes half of your knowledge useless/impossible to apply, instead upgrading all of your simple tuning tasks into architectural design problems. The summary of my last few years has basically been working around problems that almost entirely originate from trying to move software into EKS and dealing with random constraints that would take minutes to fix baremetal.
I agree that building your backend on Lambda is terrible for many reasons: slow starts, request / response size restrictions, limitations in "layer" sizes, etc.
RDS, however, I have found to be rock solid. What have you run into?
RDS, however, I have found to be rock solid. What have you run into?
The parent compares RDS to baremetal, which I think isn't a fair comparison at all. Especially since we don't know the specs of either of these.
I found RDS to be rock solid too, although performance issues are often resolved by developers by submitting a PR that bumps the instance size x2, because "why not". On baremetal it's often impossible to upgrade CPU just like that, so people have to fix performance issues elsewhere, which leads to better outcome at the end.
I found RDS to be rock solid too, although performance issues are often resolved by developers by submitting a PR that bumps the instance size x2, because "why not". On baremetal it's often impossible to upgrade CPU just like that, so people have to fix performance issues elsewhere, which leads to better outcome at the end.
RDS works great, but it's far easier to scale a bare metal setup to an extent that makes RDS look like an expensive toy because you have far more hardware options
RDS is a good option if you want convenience and simplicity, though.
RDS is a good option if you want convenience and simplicity, though.
Managing database backups myself is something that gives me nightmares. I would refuse to use bare-metal dbs unless I have a dedicated team just to manage the database (or data that is okay to lose, like caching layers).
Managing database backups is fairly straightforward. Postgres + a base backup + long term wal archiving in a blob store is very easy to set up and monitor. It could be easier, and if you don't want to manage that using RDS is certainly a valid choice, but it's a tradeoff - I often have customers that help addressing performance issues with RDS they simply wouldn't have if they sized a bare metal setup with enough RAM and NVMe and configured it even halfway decently instead, and the end result is often that they end up paying more for devops help to figure out performance bottlenecks than they'd spend putting the same devops consultant on retainer ensuring they have a solid backup setup.
I dunno, it does sound like significant work and way outside my (and most devs) area of expertise. I can definitely supervise a managed RDBMS (like RDS) by myself without help on the side even though I am no dba.
A mismanaged VPS is downtime and churn, a mismanaged DB will insta-kill your business if you have unrecoverable data loss. I would definitely use a managed solution until I can get a dedicated person to babysit the DB, but I would consider managing a VPS myself.
A mismanaged VPS is downtime and churn, a mismanaged DB will insta-kill your business if you have unrecoverable data loss. I would definitely use a managed solution until I can get a dedicated person to babysit the DB, but I would consider managing a VPS myself.
There's no need for a dedicated person. A single operator can easily manage dozens of DB instances unless your needs are extremely complex. Managing these kinds of things are serviced trivially available on retainer.
I don't know too much about the performance side of RDS, but the backup model is absolutely a headache. It's at the point where I'd rather pg_dump into gz and upload to s3.
I never had problems with RDS backup, but I never used it a ton or had really large DBs in it. What problems did you ran into?
> performance regressions, heat maps, kernel issues etc.
> AWS takes that away and makes you focus on the product.
ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions.
AWS is great if your IT/purchasing department is inefficient. Getting a new AWS machine is instant, compared to getting purchasing to approve new machine and IT allocating it to you. But all the low-level stuff is still there.
> AWS takes that away and makes you focus on the product.
ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions.
AWS is great if your IT/purchasing department is inefficient. Getting a new AWS machine is instant, compared to getting purchasing to approve new machine and IT allocating it to you. But all the low-level stuff is still there.
>AWS is great if your IT/purchasing department is inefficient
Fwiw, I think a lot of companies have this problem.
Fwiw, I think a lot of companies have this problem.
My wife is literally consulting for a big bank where it takes them 6+ months to get an on-premise VM setup and configured.
I think the conversation has turned from "Can we spend more?" to "Can you please try and spend less?"
"Can you please try and spend less?"
"Sure, we can get some on-prem machines. They'll pay for themselves in 6 months. I just need permissions from Finance to spend some CAPEX, and get IT and Facilities to cooperate"
"Ugh, actually please keep using AWS. But try and spend less.. if you can and this does not compromise deadlines"
"Sure, we can get some on-prem machines. They'll pay for themselves in 6 months. I just need permissions from Finance to spend some CAPEX, and get IT and Facilities to cooperate"
"Ugh, actually please keep using AWS. But try and spend less.. if you can and this does not compromise deadlines"
We have both AWS and colocated servers. The server workload mostly scales with the machine count not the user count. And you can get a lot done with very few servers these days.
I have literally never met a dev who wanted to deal with any kind of infrastructure.
That was barely true a decade ago. It's total nonsense today when it's trivial to ensure all your servers have IPMI or similar and it cann all be automated apart from the couple-of-times a year component swap outs.
But it's also the wrong comparison: there's rarely a reason to go on premises, and need to take responsibility for the hardware yourself - renting bare metal servers is usually the sweet spot and means someone else does the annoying bits for you but you still have the simplicity and lower cost.
As someone contracted to manage systems for people, I consistently make more money from people who overengineer their cloud setups than from people with bare metal servers. It tends to require far more maintenance to keep an AWS setup same, secure, and not bankrupting you.
But it's also the wrong comparison: there's rarely a reason to go on premises, and need to take responsibility for the hardware yourself - renting bare metal servers is usually the sweet spot and means someone else does the annoying bits for you but you still have the simplicity and lower cost.
As someone contracted to manage systems for people, I consistently make more money from people who overengineer their cloud setups than from people with bare metal servers. It tends to require far more maintenance to keep an AWS setup same, secure, and not bankrupting you.
Uh, no I don't in fact like dealing with any of that. And I've never had to in 20 years of managing some mid scale service on bare metal. (Though of course there have been issues of various sorts.)
I think you may have it backwards: people like tinkering with complex cloud stuff, even if they don't need it.
I think you may have it backwards: people like tinkering with complex cloud stuff, even if they don't need it.
> While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers.
If you have one or two baremetal servers it is not, but yes once you start having a lot of infra it is way better.
But you can get really, really far with one or two baremetal servers and some SaaS solutions...
If you have one or two baremetal servers it is not, but yes once you start having a lot of infra it is way better.
But you can get really, really far with one or two baremetal servers and some SaaS solutions...
Those are the ones that also usually tell you you can just stitch together a few SaaS products and it's magic.
It's much the same mindset as: "Vibe-coding can do it for you so you don't have to program"
Yep. Low-effort, shallow knowledge, risk-taking guys.
You are an outdate Boomer!!! I have 37 agents doing that for me!!!!!11^
LOL
LOL
I've certainly done some things where outsourcing hosting meant I didn't have to manage infrastructure. For services running on vm instances in gcp vs services running on bare metal managed hosts, there's not a whole lot of difference in terms of management IMHO.
But any infrastructure that the product I support use is infrastructure I need to manage; having it outside my control just makes it that much harder to manage. If it's outside my control, the people who control it better do a much better job than I would at managing it, otherwise it's going to be a much bigger pain.
But any infrastructure that the product I support use is infrastructure I need to manage; having it outside my control just makes it that much harder to manage. If it's outside my control, the people who control it better do a much better job than I would at managing it, otherwise it's going to be a much bigger pain.
I do consulting in this space, and I'm torn: I make much more money managing infrastructure from clients who insist on AWS. But it's much more enjoyable to work with people who knows how to keep it simple.
I worked on a project for my company (a low volume basic web app) and I suggested we could just start the whole thing on one server. They brought on some Azure consultants and the project ballooned out to months of work and all kinds of services. I’m convinced most of the consultants were just piling on services so they could make more money.
If you hire hammer experts then you're going to end up using a lot of hammers in your construction. The Azure experts aren't pitching Azure because they're trying to sell more Azure products. They do it because that's all they know and most likely because you don't know it so you'll be likely to come back to them for support when things inevitably need to evolve.
Also, the more you use your cloud vendor's various services in your code, the more subject you are to vendor lock-in.
I won't name any names, but I'm pretty sure this is a big part of the reason why a specific cloud vendor pushed so very hard for us to push a bunch of data into their highly advanced NoSQL big data solution, when the data in question was perfectly happy continuing indefinitely to exist as a few tens of megabytes of CSV files that were growing at a rate of a couple kilobytes per day.
I won't name any names, but I'm pretty sure this is a big part of the reason why a specific cloud vendor pushed so very hard for us to push a bunch of data into their highly advanced NoSQL big data solution, when the data in question was perfectly happy continuing indefinitely to exist as a few tens of megabytes of CSV files that were growing at a rate of a couple kilobytes per day.
CosmosDB?
It’s okay, this is the Internet, you can name names.
It’s okay, this is the Internet, you can name names.
Naming names would not be helpful. The problem isn't any one company's business practices. The problem isn't necessarily even vendor lock-in. The problem is familiarity bias. When all you have is a hammer, everything looks like a nail. And, as a corollary, when you ask a hammer vendor what you should do their answer will always be to treat your problems like nails.
It's probably true. The biggest challenge with doing the right thing in this space is that the sales job is hard, time consuming and so expensive that it's a lot easier to make it profitable if you make projects balloon like that. The sales effort is much the same.
I've been offering to help people cut costs for a while, and it's a shockingly hard sell even with offers of guarantees, so we're deemphasizing it to focus more on selling more complex DevOps assistance and AI advice instead... Got to eat (well, I do much better than that, but anyway), but I refuse to over engineer things just to make more money.
I've been offering to help people cut costs for a while, and it's a shockingly hard sell even with offers of guarantees, so we're deemphasizing it to focus more on selling more complex DevOps assistance and AI advice instead... Got to eat (well, I do much better than that, but anyway), but I refuse to over engineer things just to make more money.
I don't necessarily even blame the contractors. When the bosses look askance at simple solutions what can you do? It's weirdly harder to sell people on something simple than on something complex. They assume the simple solution must be missing something important.
What I’ve always found concerning about managed cases is that the “platform” teams could never explain, in simple terms, how the application was actually deployed.
It was so complex I gave up after a while. That’s never a good sign.
It was so complex I gave up after a while. That’s never a good sign.
I'll play devil's advocate a little bit here. But to be clear, I hate AWS and all of their crazy concepts and exorbitant pricing, so ultimately I think I'm on your side.
OS maintenance honestly is a bit hard for me. I need to know what to install for monitoring, I need to maintain scripts or Ansible playbooks. I need to update these and make sure they don't break my setup.
And the big kicker is compliance. I always work under SOC2, ISO27001, PCI-DSS, HIPAA, you name it. These require even more things like intrusion detection, antivirus, very detailed logging, backups, backup testing, web application firewall. When you just use AWS Lambda with DynamoDB, the compliance burden goes down a lot.
OS maintenance honestly is a bit hard for me. I need to know what to install for monitoring, I need to maintain scripts or Ansible playbooks. I need to update these and make sure they don't break my setup.
And the big kicker is compliance. I always work under SOC2, ISO27001, PCI-DSS, HIPAA, you name it. These require even more things like intrusion detection, antivirus, very detailed logging, backups, backup testing, web application firewall. When you just use AWS Lambda with DynamoDB, the compliance burden goes down a lot.
Yes, you need to write Ansible initially. But honestly, it’s not that much for your average application server. Turn on unattended-upgrades with anything critical to your application blacklisted, and you won’t have to touch it other than to bump version pins whenever you make a new golden image.
Re: compliance, other than SOC2 being a giant theater of bullshit, agreed that it adds additional work. My point is that the claims of “not having to manage infrastructure” is highly misleading. You get to skip some stuff, yes, but you are paying through the nose in order to avoid writing some additional config files.
Re: compliance, other than SOC2 being a giant theater of bullshit, agreed that it adds additional work. My point is that the claims of “not having to manage infrastructure” is highly misleading. You get to skip some stuff, yes, but you are paying through the nose in order to avoid writing some additional config files.
Working on various teams operating on infrastructure that ranged from a rack in the back of the office, a few beefy servers in a colo, a fleet of Chef-managed VMs, GKE, ECS, and various PaaSes, what I've liked the most about the cloud and containerized workflows is that they wind up being a forcing function for reproducibility, at least to a degree.
While it's absolutely 100% possible to have a "big beefy server architecture" that's reasonably portable, reproducible, and documented, it takes discipline and policy to avoid the "there's a small issue preventing {something important}, I can fix it over SSH with this one-liner and totally document it/add it to the config management tooling later once we've finished with {something else important}" pattern, and once people have been doing that for a while it's a total nightmare to unwind down the line.
Sometimes I want to smash my face into my monitor the 37th time I push an update to some CI code and wait 5 minutes for it to error out, wishing I could just make that band-aid fix, but at the end of the day I can't forget to write down what I did, since it's in my Dockerfile or deploy.yaml or entrypoint.sh or Terraform or whatever.
While it's absolutely 100% possible to have a "big beefy server architecture" that's reasonably portable, reproducible, and documented, it takes discipline and policy to avoid the "there's a small issue preventing {something important}, I can fix it over SSH with this one-liner and totally document it/add it to the config management tooling later once we've finished with {something else important}" pattern, and once people have been doing that for a while it's a total nightmare to unwind down the line.
Sometimes I want to smash my face into my monitor the 37th time I push an update to some CI code and wait 5 minutes for it to error out, wishing I could just make that band-aid fix, but at the end of the day I can't forget to write down what I did, since it's in my Dockerfile or deploy.yaml or entrypoint.sh or Terraform or whatever.
You have to remove admin rights to your admins then, because scrappy enough DevOps/platform engineers/whatever will totally hand-edit your AWS infra or Kubernetes deployments. I suffered that first hand. And it's even worse that in the old days, because at least back in the day it was expected.
Or at least you have to automatically destroy and recreate all nodes / VMs / similar every N days, so that nobody can pretend that any truly unavoidable hand-edits during emergency situations will persist. Possibly also control access to the ability to do hand edits behind a break-glass feature that also notifies executives or schedules a postmortem meeting about why it was necessary to do that.
I know of at least one organisation that'd automatically wipe every instance on (ssh-)user logout, so you could log in to debug, but nothing you did would persist at all. I quite like that idea, though sometimes being able to e.g. delay the wipe for up to X hours might be slightly easier to deal with for genuinely critical emergency fixes.
But, yes, gating it behind notifications would also be great.
But, yes, gating it behind notifications would also be great.
Oh no it ran out of disk space because of bug! I will run a command on that instance to free it rather than fix bug. Oh no error now happens half of the time better debug for hours only to find out someone only fixed a single instance…
I will never understand the argument for cloud other than bragging rights about burning money and saving money which never shoulda been burning to begin with.
I will never understand the argument for cloud other than bragging rights about burning money and saving money which never shoulda been burning to begin with.
Nah, just run Puppet or similar. You’re welcome to run your command to validate what you already tested in stage, but if you don’t also push a PR that changes the IaC, it’s getting wiped out in a few minutes.
I hate not having root access. I don’t want to have to request permission from someone who has no idea how to do what I want to do. Log everything, make everything auditable, and hold everyone accountable - if I fuck up prod, my name will be in logs, and there will be a retro, which I will lead - but don’t make me jump through hoops to do what I want, because odds are I’ll instead find a way around them, because you didn’t know what you were doing when you set up your security system.
I hate not having root access. I don’t want to have to request permission from someone who has no idea how to do what I want to do. Log everything, make everything auditable, and hold everyone accountable - if I fuck up prod, my name will be in logs, and there will be a retro, which I will lead - but don’t make me jump through hoops to do what I want, because odds are I’ll instead find a way around them, because you didn’t know what you were doing when you set up your security system.
But then your next deployment goes, and it all rolls back, right?
And then it their fault, right?
I might have mild trauma from people complaining their artisanal changes to our environment weren’t preserved.
And then it their fault, right?
I might have mild trauma from people complaining their artisanal changes to our environment weren’t preserved.
Weeeeell, if you use Helm the manual change might be preserved, which makes investigations even more... interesting.
In my org nobody has admin rights with the exception of emergencies, but we are ending up with a directory full of Github workflows and nobody knows, which of them are currently supposed to work.
Nothing beats people knowing what they are doing and cleaning up behind them.
Nothing beats people knowing what they are doing and cleaning up behind them.
I'm still a pretty big fan of Docker (compose) behind Caddy as a reverse-proxy... I think that containers do offer a lot in terms of application support... even if it's a slightly bigger hoop to get started with in some ways.
I'm working on an app server that's auto deploying itself behind Caddy + DNS/SSL aut config. Caddy is amazing, and there really should be no reason for complex setups for most people these days... I've worked on some huge systems, but most systems can run in trivially simple setups given modern hardware.
Have always felt the same.
I’ve seen an entire company proudly proclaim a modern multicore Xeon with 32GB RAM can do basic monitoring tasks that should have been possible with little more than an Arduino.
Except the 32GB Xeon was far too slow for their implementation...
I’ve seen an entire company proudly proclaim a modern multicore Xeon with 32GB RAM can do basic monitoring tasks that should have been possible with little more than an Arduino.
Except the 32GB Xeon was far too slow for their implementation...
I swear, before I finished reading your comment, this thought jumped into my mind: ‘oh my, they do host everything with a computer similar to my [pretty old by the way, but still beefy] for-work computer! Impressive!’
Which is, I still believe is perfectly possible to do.
Then, I was ‘what?!’
Which is, I still believe is perfectly possible to do.
Then, I was ‘what?!’
Let me guess: database tables with no indexes, full scans everywhere?
How did they implement it? That's horrendous.
Java - but done so horribly even the devil would be aghast.
There were thousands of threads, murmuring an incessant hum fully occupying a few cores when absolutely nothing was happening. Over 20GB RAM actively used at idle.
I think booting the application took almost half an hour at one point, using a local SSD.
I’m fairly certain at no point in my career could I ever have replicated such a monstrosity.
There were thousands of threads, murmuring an incessant hum fully occupying a few cores when absolutely nothing was happening. Over 20GB RAM actively used at idle.
I think booting the application took almost half an hour at one point, using a local SSD.
I’m fairly certain at no point in my career could I ever have replicated such a monstrosity.
I totally agree. So much complexity for generally no good reason [0]. I saw so much of this that I ended up starting a company doing the exact opposite. I figured I could do it better and cheaper, so that that's now what we do!
If anyone wants to bail out of AWS et al and onto a few beefy servers, save some money, and gain a DevOps team in the process, then drop us an email (adam at domain in bio).
[0] My pet theory about the real reason: the hyper-scalers hire all the engineers who have the skills to deploy-to-a-few-beefy-servers, and then charge a 10x multiplier for compute. Companies can then choose between impossible hiring, or paying more. Paying more is easier to stomach, and plenty of rationalisations are available.
If anyone wants to bail out of AWS et al and onto a few beefy servers, save some money, and gain a DevOps team in the process, then drop us an email (adam at domain in bio).
[0] My pet theory about the real reason: the hyper-scalers hire all the engineers who have the skills to deploy-to-a-few-beefy-servers, and then charge a 10x multiplier for compute. Companies can then choose between impossible hiring, or paying more. Paying more is easier to stomach, and plenty of rationalisations are available.
> My pet theory about the real reason: the hyper-scalers hire all the engineers who have the skills to deploy-to-a-few-beefy-servers, and then charge a 10x multiplier for compute.
This is also my pet theory, and it’s maddening. They’ve successfully convinced an entire generation of devs that physical servers are super scary and they shouldn’t ever have to look at them.
This is also my pet theory, and it’s maddening. They’ve successfully convinced an entire generation of devs that physical servers are super scary and they shouldn’t ever have to look at them.
Docker compose on a couple nice VPS’s can do a LOT
On the other hand, I know a lot of people who spend more time / salary messing around with their infra than the couple hundred bucks they've saved from not pressing a couple of buttons on vercel / cloudfare
There's a time and place for just deploying quickly to a cloud provider versus trying to manage your infra. It's a nuanced tradeoff that rarely has a clear winner.
There's a time and place for just deploying quickly to a cloud provider versus trying to manage your infra. It's a nuanced tradeoff that rarely has a clear winner.
I look at what I can do with an old mac mini (2011) and it’s quite good. I think the only issue with hardware is technical maintenance, but at the scale of a small companies, that would probably be having a support contract with Dell and co.
Small companies should never forget to ask Dell, etc for discounts. The list prices at many of these companies are aspirational and, even at very small scale, huge discounts are available.
I think it depends on what you are optimizing for. If you are a VC funded startup trying to get to product market fit, spending a bit more on say AWS probably makes sense so you can be “agile”. The opportunity cost there might outweigh infrastructure cost. If you are bootstrapped and cost matters a lot, then different story.
You can get a server now with, like, five hundred cores and a fifty terabytes of RAM. It's expensive, but you can get one.
A used server with sixty cores and one terabyte of RAM is a lot cheaper. Couple thousand bucks. I mean, that's still a lot of bucks, but a terabyte for only four digits?
A used server with sixty cores and one terabyte of RAM is a lot cheaper. Couple thousand bucks. I mean, that's still a lot of bucks, but a terabyte for only four digits?
You can get a server now with, like, five hundred cores and a fifty terabytes of RAM. It's expensive, but you can get one.
A used server with sixty cores and one terabyte of RAM is a lot cheaper. Couple thousand bucks. I mean, that's a lot of bucks, but a terabyte for only four digits?
A used server with sixty cores and one terabyte of RAM is a lot cheaper. Couple thousand bucks. I mean, that's a lot of bucks, but a terabyte for only four digits?
The problem with onsite or colo is always the same. You have to keep fighting the same battle again and again and again. In 5 years when the servers need replaced even though you have already proven it saves orders of magnitude in costs.
I've never once been rewarded for saving 100k+ a month even though I have done exactly that. I have been punished by having to constantly re justify the decision though. I just don't care anymore. I let the "BIG BRAIN MBA's" go ahead and set money on fire in the cloud. It's easier for me. Now I get to hire a team of "cloud architects" to do the infra. At eye bleeding cost increases for a system that will never ever see more than a few thousand users.
I've never once been rewarded for saving 100k+ a month even though I have done exactly that. I have been punished by having to constantly re justify the decision though. I just don't care anymore. I let the "BIG BRAIN MBA's" go ahead and set money on fire in the cloud. It's easier for me. Now I get to hire a team of "cloud architects" to do the infra. At eye bleeding cost increases for a system that will never ever see more than a few thousand users.
What I say is that we massively underestimate just how fast computers are these days
On the other hand, there is a real crossroad that pops up that HNers tend to dismiss.
A common story is that since day one you just have lightweight app servers handling http requests doing 99% I/O. And your app servers can be deployed on a cheap box anywhere since they're just doing I/O. Maybe they're on Google Cloud Run or a small cluster of $5 VPS. You've built them so that they have zero deps on the machine they're running on.
But then one day you need to do some sort of computations.
One incremental option is to create a worker that can sit on a machine that can crunch the tasks and a pipeline to feed it. This can be seen as operationally complex compared to one machine, but it's also simple in other ways.
Another option is to do everything on one beefy server where your app servers just shell out the work on the same machine. This can be operationally simple in some ways, but not necessarily in all ways.
A common story is that since day one you just have lightweight app servers handling http requests doing 99% I/O. And your app servers can be deployed on a cheap box anywhere since they're just doing I/O. Maybe they're on Google Cloud Run or a small cluster of $5 VPS. You've built them so that they have zero deps on the machine they're running on.
But then one day you need to do some sort of computations.
One incremental option is to create a worker that can sit on a machine that can crunch the tasks and a pipeline to feed it. This can be seen as operationally complex compared to one machine, but it's also simple in other ways.
Another option is to do everything on one beefy server where your app servers just shell out the work on the same machine. This can be operationally simple in some ways, but not necessarily in all ways.
In 2010 I was managing 100 servers, with many Oracle and Postgres DB, PHP, Apache, all on Solaris and Sun HW. I was constantly impressed by how people were unable to do more or less correct estimations. I had a discussion with my boss, he wanted to buy 8 servers, I argued one was more than enough. The system, after growing massively, was still in 2020 managing the load with just 3 servers. So I would argue, not only today, but 15 years ago already.
Most younger devs just have no concept on how limited hardware we ran services on...
I used to run a webmail system with 2m accounts on hardware with less total capacity (ram, disk, CPU throughput) than my laptop...
What's more: It was a CGI (so new process for every request), and the storage backend spawned separate processes per user.
I used to run a webmail system with 2m accounts on hardware with less total capacity (ram, disk, CPU throughput) than my laptop...
What's more: It was a CGI (so new process for every request), and the storage backend spawned separate processes per user.
If you know anything about hardware and look at the typical instances AWS is serving up (other than the ludicrously expensive ones) it's Skylake and older.
I think people have a warped perception of performance, if only because the cloud providers are serving up a shared VM on equipment I'd practically class as vintage computing. You could throw some of the same parts together from eBay and buy the whole system with less than a few months worth of the hourly on-demand cost.
I think people have a warped perception of performance, if only because the cloud providers are serving up a shared VM on equipment I'd practically class as vintage computing. You could throw some of the same parts together from eBay and buy the whole system with less than a few months worth of the hourly on-demand cost.
Indeed - they are incredibly fast, it's just buried under layers upon layers of stuff
No worries, another fifteen layers of software abstraction will soak that up pronto.
Depending on your regulatory environment, it can be cost-effective to not have to maintain your own data center with 24/7 security response, environmental monitoring, fire suppression systems, etc. (of course, the majority of businesses are probably not interested in things like SOC 2)
This argument comes up a lot, but it feels a bit silly to me. If you want a beefy server you start out with renting one. $150/month will give you a server with 24 core Xeon and 256GB of RAM, in a data center with everything you mentined plus a 24/7 hands-on technician you can book. Preferably rent two servers, because reliablity. Once you outgrow renting servers you start renting rack space in a certified data center with all the same amenities. Once you outgrow that you start renting entire racks, then rows of racks or small rooms inside the DC. Then you start renting portions of the DC. Once you have outgrown that you have to seriously worry about maintaining your own data center. But at that point you have so much scale that this will be the least of your worries
> This argument comes up a lot, but it feels a bit silly to me. If you want a beefy server you start out with renting one. $150/month will give you a server with 24 core Xeon and 256GB of RAM, in a data center with everything you mentined plus a 24/7 hands-on technician you can book.
What's the bandwidth and where can I rent one of these??
What's the bandwidth and where can I rent one of these??
Hetzner [1]. Bandwidth is 1 GBit/s. You can also get 10 GBit/s, that's hidden away a bit instead of being mentioned on the order page [2]
1: https://www.hetzner.com/dedicated-rootserver/matrix-ex
2: https://docs.hetzner.com/robot/dedicated-server/network/10g-...
1: https://www.hetzner.com/dedicated-rootserver/matrix-ex
2: https://docs.hetzner.com/robot/dedicated-server/network/10g-...
I have wished for years that Hetzner would offer their bare metal servers in the U.S., and not just Hetzner Cloud.
Here is US Hetzner: https://ioflood.com/
Their prices have come down a lot. I used them when the servers still cost $200 a piece, but their support at the time was fantastic.
Their prices have come down a lot. I used them when the servers still cost $200 a piece, but their support at the time was fantastic.
Wow. No joke. I haven’t heard of them, but I like their blurb, and those are Hetzner like prices. Now, I just need to find a use for that much beef.
How is that any different from cloud?
This whole thread was a response to
> Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.
suggesting to use a few beefy servers but if we are renting them from cloud we're back where we started.
This whole thread was a response to
> Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.
suggesting to use a few beefy servers but if we are renting them from cloud we're back where we started.
The difference from the big clouds is that an equivalent instance at AWS costs 10x as much. If you go with few beefy servers AWS offers very little value for the money they charge, they only make sense for "cloud native" architectures. But if you rent raw servers from traditional hosters you can get prices much closer to the amortized costs of running them yourself, with the added convenience of having them in a certified data center with 24/7 security, backup power, etc.
If you want more control than that, colo is also pretty cheap [1]. But I'd consider that a step above what 95% of people need
https://www.hetzner.com/colocation
If you want more control than that, colo is also pretty cheap [1]. But I'd consider that a step above what 95% of people need
https://www.hetzner.com/colocation
For me the comparison was not against the specific instance of AWS but cloud in general, and AWS was a for instance. Which was the whole reason why I brought up compliance and stuff—it is much cheaper to have someone else handle that for you (even if it is hetzner!). That was my whole point.
Not ideal when a large part of your userbase is in APAC.
https://us.ovhcloud.com/bare-metal/prices/?display=list
also pretty sure 24 cores is like 48 cloud “cores” which are usually just hyper threads right?
also pretty sure 24 cores is like 48 cloud “cores” which are usually just hyper threads right?
[deleted]
I'm a lot less concerned about CPU and ram and a lot more concerned about replicated object storage (across data centers). High end GPUs are also pretty important.
The only companies directly dealing with that type of stuff are the ones already at such a scale where they need to actually build their own data centers. Everyone else is just renting space somewhere that already takes care of those things and you just need to review their ISO/SOC reports.
This kind of argument comes from the cloud provider marketing playbook, not reality.
This kind of argument comes from the cloud provider marketing playbook, not reality.
This is handled by colo.
Exactly this! The educational product I work on is used by hundreds of thousands of students a day, and the secret to our success is how simple our architecture is. PHP monoliths + Cache (Redis/Memcached) scale super wide basically for free. We don't really think about scalability, it just happens.
I have a friend whose startup had a super complicated architecture that was falling apart at 20 requests per second. I used to be his boss a lifetime ago and he brought me in for a meeting with his team to talk about it. I was just there flabbergasted at "Why is any of this so complicated?!" It was hundreds of microservices, many of them black boxes they'd paid for but had no access to the source. Your app is essentially an async chat app, a fancy forum. It could have been a simple CRUD app.
I basically told my friend I couldn't help, if I can't get to the source of the problematic nodes. They'll need to talk to the vendor. I explained that I'd probably rewrite it from the ground up. They ran out of runway and shut down. He's an AI influencer now...
I have a friend whose startup had a super complicated architecture that was falling apart at 20 requests per second. I used to be his boss a lifetime ago and he brought me in for a meeting with his team to talk about it. I was just there flabbergasted at "Why is any of this so complicated?!" It was hundreds of microservices, many of them black boxes they'd paid for but had no access to the source. Your app is essentially an async chat app, a fancy forum. It could have been a simple CRUD app.
I basically told my friend I couldn't help, if I can't get to the source of the problematic nodes. They'll need to talk to the vendor. I explained that I'd probably rewrite it from the ground up. They ran out of runway and shut down. He's an AI influencer now...
crud app for async chat app works only that far. when you start getting a lot of customers (companies, etc), big chat rooms, etc - things get complicated.
i saw this kind of system that started as simple crud app, and many years later developers still try to resolve some of the originals sins.
i saw this kind of system that started as simple crud app, and many years later developers still try to resolve some of the originals sins.
If it takes about long to resolve "original sins" it wasn't simple enough to begin with.
"original sins" happened way before i joined the company. so i don't know the exact chronology, but app was started as very simple crud app. new features were added as client base increased. client base became more enterprisy. mobile apps + desktop apps appeared. new features were added to support them.
so something that started as simple node.js + nosql crud app written by not that experienced developers became over span of few years little shop of horror that was failing to scale without all kinds of crazy workarounds
so something that started as simple node.js + nosql crud app written by not that experienced developers became over span of few years little shop of horror that was failing to scale without all kinds of crazy workarounds
Around 2013 I was handling bursts up to thousands of requests per second for multi-megabyte file downloads with dynamic authentication using just PHP5, Apache2, and Haproxy, with single-node MySQL (or may have been MariaDB, by then?) as the database, and Redis for caching. On a single mid-range rented server. And Haproxy was only there for operational convenience, you could cut it out and it'd work just as well. No CDN. Rock solid.
My joke but not-actually-a-joke is that the Cloud is where you send a workload that's fast on your laptop, if you need it to be way slower. The performance of these fussy, over-complicated, hard-to-administer[1] systems is truly awful for the price.
[1] They're hypothetically simpler and easier to administer, but I've never seen this in the wild. If anything, we always seem to end up with more hours dedicated to care & feeding of this crap, and more glitchiness and failures, than we would with a handful of rented servers with maybe a CDN in front.
My joke but not-actually-a-joke is that the Cloud is where you send a workload that's fast on your laptop, if you need it to be way slower. The performance of these fussy, over-complicated, hard-to-administer[1] systems is truly awful for the price.
[1] They're hypothetically simpler and easier to administer, but I've never seen this in the wild. If anything, we always seem to end up with more hours dedicated to care & feeding of this crap, and more glitchiness and failures, than we would with a handful of rented servers with maybe a CDN in front.
> My joke but not-actually-a-joke is that the Cloud is where you send a workload that's fast on your laptop, if you need it to be way slower.
Not to forget: where you send a workload that is free on your laptop, in order to be charged for it.
Not to forget: where you send a workload that is free on your laptop, in order to be charged for it.
> The general setup for the largest players in that space was haproxy in front of nginx in front of several PHP servers in front of a MySQL database that had one primary r/w with one read only replica.
You'd be surprised that the most stable setups today are run this way. The problem is that this way it's hard to attract investors; they'll assume you are running on old or outdated tech. Everything should be serverless, agentic and, at least on paper, hyperscalable, because that sells further.
> Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.
That is actually the goal of hyperscalers: they are charging you premium for way inferior results. Also, the article stated a very cold truth: "every engineer wants a fashionable CV that will help her get the next job" and you won't definitely get a job if you said: "I moved everything from AWS and put it behind haproxy on one bare-metal box for $100/mo infra bill".
You'd be surprised that the most stable setups today are run this way. The problem is that this way it's hard to attract investors; they'll assume you are running on old or outdated tech. Everything should be serverless, agentic and, at least on paper, hyperscalable, because that sells further.
> Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.
That is actually the goal of hyperscalers: they are charging you premium for way inferior results. Also, the article stated a very cold truth: "every engineer wants a fashionable CV that will help her get the next job" and you won't definitely get a job if you said: "I moved everything from AWS and put it behind haproxy on one bare-metal box for $100/mo infra bill".
> The problem is that this way it's hard to attract investors; they'll assume you are running on old or outdated tech. Everything should be serverless, agentic and, at least on paper, hyperscalable, because that sells further.
Investors don't give a shit about your stack
Investors don't give a shit about your stack
Many do. For most it's not the biggest concern (that would be quite weird). AFAIK it's mostly about reducing risk (avoiding complete garbage/duck taped setups)
Source: I know a person who does tech DD for investors, and I've also been asked this question in DD processes.
Source: I know a person who does tech DD for investors, and I've also been asked this question in DD processes.
[deleted]
Are those over engineered systems even actually scalable? I know teams who designed a CQRS architecture using messages queues and a distributed NoSQL database and fail to sustain 10req/s for a read in something that is basically a CRUD application. Heck once someone literally said "But we use Kafka, why aren't we fast?!".
Exactly this, every time I see kafka or similar its a web of 10M microprocesses that take more time in invocation alone than if you just ran the program in one go.
[deleted]
Eh, they scale between $1000 and $10000 per month fairly easily. I’m not sure about the requests though.
I watched in amusement as the architecture team at $JOB eagerly did a PoC of a distributed RDBMS, only to eventually conclude that the latency was too high. Gee… if only someone had told you that would happen when you mentioned the idea. Oh wait.
I've seen an application, a 95% CRUD application, which had about 100-1000 users across the UK, users who would only be using it from 9am-5:30pm, and at that - barely interacting with it. This was backed by literally the most sophisticated and complex architecture I have ever seen in my entire life.
There were 3 instances of cognito. RDS, DynamoDB and S3. The entire architecture diagram would only be legible on an A2 (heck, maybe even A1) page. And that was the high level diagram. The central A4 part of that diagram was a bunch of micro-services for handling different portions of this CRUD application.
This company could afford a system architect as well as a team of developers to work on this full time.
I was genuinely baffled, but this company was in an extremely lucrative industry, so I guess in this case it's fine to just take some of your profits and burn them.
There were 3 instances of cognito. RDS, DynamoDB and S3. The entire architecture diagram would only be legible on an A2 (heck, maybe even A1) page. And that was the high level diagram. The central A4 part of that diagram was a bunch of micro-services for handling different portions of this CRUD application.
This company could afford a system architect as well as a team of developers to work on this full time.
I was genuinely baffled, but this company was in an extremely lucrative industry, so I guess in this case it's fine to just take some of your profits and burn them.
Akin to buying a high performance sports car and never driving it. Maybe it has social value, maybe you just feel good having it.
Exactly.. it was a lot different when a typical server was 2-4 CPUs and costs more than a luxury car... today you get hundreds of simultaneous threads and upwards of a terabyte of ram for even less, not counting inflation.
You can go a very, very, very long way on 2-3 modern servers with a fast internet connection and a good backup strategy.
Even with a traditional RDBMS like MS-SQL/PostgreSQL, you aren't bottlenecked by the 1-2ghz cpu and spinning rust hard drives. You can easily get to millions of users for a typical site/app with a couple servers just for a read replica/redundancy. As much as I happen to like some of the ergonomics of Mongo from a developer standpoint, or appreciate the scale of Cassandra/.ScyllaDB or even Cockroach... it's just not always necessary early on, or ever.
I've historically been more than happy to reach for RabbitMQ or Redis when you need queueing or caching... but that's still so much simpler than where some microservice architectures have gone. And while I appreciate what Apollo and GraphQL bring to the table, it's over the top for the vast majority of applications.
You can go a very, very, very long way on 2-3 modern servers with a fast internet connection and a good backup strategy.
Even with a traditional RDBMS like MS-SQL/PostgreSQL, you aren't bottlenecked by the 1-2ghz cpu and spinning rust hard drives. You can easily get to millions of users for a typical site/app with a couple servers just for a read replica/redundancy. As much as I happen to like some of the ergonomics of Mongo from a developer standpoint, or appreciate the scale of Cassandra/.ScyllaDB or even Cockroach... it's just not always necessary early on, or ever.
I've historically been more than happy to reach for RabbitMQ or Redis when you need queueing or caching... but that's still so much simpler than where some microservice architectures have gone. And while I appreciate what Apollo and GraphQL bring to the table, it's over the top for the vast majority of applications.
I’m as likely to talk about human scale as hardware scale, and one of the big issues with human scale is what the consequences are of having the wrong team size in either direction.
When you reduce the man hours per customer you can get farther down your backlog. You can carve people off for new prospective business units. You can absorb the effects of a huge sale or bad press better because you aren’t trying to violate Brooks’ Law nor doing giant layoffs that screw your business numbers.
You have time for people to speculate on big features or more work on reducing the costs further. If you don’t tackle this work early you end up in the armed Queen Problem: running as fast as you can just to stay still.
When you reduce the man hours per customer you can get farther down your backlog. You can carve people off for new prospective business units. You can absorb the effects of a huge sale or bad press better because you aren’t trying to violate Brooks’ Law nor doing giant layoffs that screw your business numbers.
You have time for people to speculate on big features or more work on reducing the costs further. If you don’t tackle this work early you end up in the armed Queen Problem: running as fast as you can just to stay still.
Preach. Architects need to get back to base principles, stop reading so many blog posts, start writing more code, and actually try things for themselves. Almost every system I work on is architecturally obese and while it "could scale" it has never and will realistically never need to. The first thing I'd do to claw back performance (and productivity) in most instances is make the system less distributed.
Around 5 years ago the metagame was to make everything horizontally scalable.
Now it seems things are swinging back the other direction and articles like "Use One Big Server" are getting re-discussed: https://news.ycombinator.com/item?id=45085029
Now it seems things are swinging back the other direction and articles like "Use One Big Server" are getting re-discussed: https://news.ycombinator.com/item?id=45085029
"La mode, c'est ce qui se démode" -- J Cocteau
("fashion is what goes out of fashion")
("fashion is what goes out of fashion")
I thought i knew about scaled deployments before i started working where i do now. After staring here, i realized i had no idea what an environment of huuuuge scale actually was. Id been part of multi site deployments and scaled infra, but it was basically potatoes comparatively. We have a team whose platform we, on IT, call the DoS'er of the company. Its responsible for processing hundreds of thousands of test runs a day, and data is fed to a plethora of services after. The scale is so large that they are able to take down critical services, or deeply impact them, purely due to throughput, if a developer goes too far (like say uploading a million small logs to an s3 bucket every minute).
We also have been contacted by AWS having them ask us what the hell we are doing, for a specific set of operations. We do a huge prep for some operations, and the prep feeds massive amounts of data through some AWS services, so much so, they thought we were under attack or had been compromised. Nope, just doin data ingestion!
We also have been contacted by AWS having them ask us what the hell we are doing, for a specific set of operations. We do a huge prep for some operations, and the prep feeds massive amounts of data through some AWS services, so much so, they thought we were under attack or had been compromised. Nope, just doin data ingestion!
You are spot on. Most developers have no clue how powerful servers are nowadays.
It is an issue to mistake scalability with resiliency
Yes, we can run twitter on a single server (https://thume.ca/2023/01/02/one-machine-twitter/) No, we do not want to run twitter on a single server
Yes, we can run twitter on a single server (https://thume.ca/2023/01/02/one-machine-twitter/) No, we do not want to run twitter on a single server
I would argue that even resiliency is a metric that should not be overemphasized in early stages of development. I would rather have a system that suffers occasional outages than one that has perfect resiliency but has added complexity with tradeoffs in costs, complexity and thus development velocity. I think the risks of not being quick enough to product market fit in early stages is bigger than losing customers over short duration outages - except of course if the selling point is resiliency.
Of course this should not be overdone, but there is something to be said for single server + backup setups, and reweriting to scale + resiliency once traction has been established.
Of course this should not be overdone, but there is something to be said for single server + backup setups, and reweriting to scale + resiliency once traction has been established.
It's much easier to build a resilient system with a simple architecture. E.g. run the application on a decent VM or even bare metal server and mirror the whole system between a few different data centers.
I have been advocating similar setups to this for years, no one wants to hear it. I get downright puzzled looks from people who think you need moon-landing monitorability of every little service and database...
We have 100k paying users and most of them don't actively use our service, yet we have microservice slop with k8 and load balancers galore running in big tech cloud provider and document databases.
From our whole setup I can only think of _maybe_ one thing that should be split into a separate dedicated service, everything else could happily live in a monolith backed by a relational database and a caching layer and a storage solution.
We have 100k paying users and most of them don't actively use our service, yet we have microservice slop with k8 and load balancers galore running in big tech cloud provider and document databases.
From our whole setup I can only think of _maybe_ one thing that should be split into a separate dedicated service, everything else could happily live in a monolith backed by a relational database and a caching layer and a storage solution.
The architecture you describe is ok because in the end it is a fairly simple website. Little user interaction, limited amount of content (at most a few million records), few content changes per day. The most complex part is probably to have some kind of search engine but even with 10 million videos an ElasticSearch index is probably no larger than 1GB.
The only problem is that there is a lot of video data.
The only problem is that there is a lot of video data.
This is probably also true for 98% of startups.
I think most people don't realise that "10 million" records is small, for a computer.
(That said, I have had to deal with code that included an O(n^2) de-duplication where the test data had n ~= 20,000, causing app startup to take 20 minutes; the other developer insisted there was no possible way to speed this up, later that day I found the problem, asked the CTO if there was a business reason for that de-duplication, removed the de-duplication, and the following morning's stand-up was "you know that 20 minute startup you said couldn't possibly be sped up? Yeah, well, I sped it up and now it takes 200ms")
I think most people don't realise that "10 million" records is small, for a computer.
(That said, I have had to deal with code that included an O(n^2) de-duplication where the test data had n ~= 20,000, causing app startup to take 20 minutes; the other developer insisted there was no possible way to speed this up, later that day I found the problem, asked the CTO if there was a business reason for that de-duplication, removed the de-duplication, and the following morning's stand-up was "you know that 20 minute startup you said couldn't possibly be sped up? Yeah, well, I sped it up and now it takes 200ms")
I thought you were going to say to reduced O(n^2) to O(n*log(n)), but you just deleted the operation. Normally I'd say that's great, but just how much duplicate data is being left around now? Is that OK?
Each element was about, oh I can't remember exactly, perhaps 50 bytes? It wasn't a constant value, there could in theory be a string in there, but those needed to be added manually and when you have 20,000 of them, nobody would.
Also, it was overwhelmingly likely that none of the elements were duplicates in the first place, and the few exceptions were probably exactly one duplicate.
Also, it was overwhelmingly likely that none of the elements were duplicates in the first place, and the few exceptions were probably exactly one duplicate.
I'm kind of surprised no one just searched for "deduplication algorithm". If it was absolutely necessary to get this 1MB dataset to be smaller (when was this? Did it need to fit in L2 on a pentium 3 something?), then it could probably have been deduped + loaded in 300-400ms.
Most engineers that I've worked with that die on a premature optimization molehill like you describe also make that molehill as complicated as possible. Replacing the inside of the nested loop with a hashtable probe certainly fits the stereotype.
Most engineers that I've worked with that die on a premature optimization molehill like you describe also make that molehill as complicated as possible. Replacing the inside of the nested loop with a hashtable probe certainly fits the stereotype.
> I'm kind of surprised no one just searched for "deduplication algorithm".
Fair.
To set the scene a bit: the other developer at this point was arrogant, not at all up to date with even the developments of his preferred language, did not listen to or take advice from anyone.
I think a full quarter of my time there was just fire-fighting yet another weird thing he'd done.
> If it was absolutely necessary to get this 1MB dataset to be smaller
It was not, which is why my conversation with the CTO to check on if it was still needed was approximately one or two sentences from each of us. It's possible this might have been important on a previous pivot of the thing, at least one platform shift before I got there, but not when I got to it.
Fair.
To set the scene a bit: the other developer at this point was arrogant, not at all up to date with even the developments of his preferred language, did not listen to or take advice from anyone.
I think a full quarter of my time there was just fire-fighting yet another weird thing he'd done.
> If it was absolutely necessary to get this 1MB dataset to be smaller
It was not, which is why my conversation with the CTO to check on if it was still needed was approximately one or two sentences from each of us. It's possible this might have been important on a previous pivot of the thing, at least one platform shift before I got there, but not when I got to it.
As opposed to what problem?
Like I can honestly have trouble listing too many business problems/areas that would fail to scale with their expected user count, given reasonable hardware and technical competence.
Like YouTube and Facebook are absolute outliers. Famously, stackoverflow used to run on a single beefy machine (and the reason they changed their architecture was not due to scaling issues), and "your" startup ain't needing more scale than SO.
Like I can honestly have trouble listing too many business problems/areas that would fail to scale with their expected user count, given reasonable hardware and technical competence.
Like YouTube and Facebook are absolute outliers. Famously, stackoverflow used to run on a single beefy machine (and the reason they changed their architecture was not due to scaling issues), and "your" startup ain't needing more scale than SO.
Scaling to a lot of reads is relatively easy, but you get into weird architectural territory once you hit a certain volume of writes. Anything involving monitoring or real-time event analysis can get hairy. That's when stuff like kafka becomes really valuable.
In streaming your website is typically totally divorced from your media serving. Media serving is just a question of cloud storage and pointing at an hls/dash manifest in that object store. Once it starts playing the website itself does almost nothing. Live streaming adds more complexity but it's still not much of a website problem.
Maintaining the media lifecycle, receiving, transcoding, making it available and removing it, is the big task but that's not real-time, it's batch/event processing at best efforts.
The biggest challenges with streaming are maintaining the content catalogue, which aren't just a few million records but rich metadata about the lifecycle and content relationships. Then user management and payments tends to also have a significant overhead, especially when you're talking about international payment processing.
Maintaining the media lifecycle, receiving, transcoding, making it available and removing it, is the big task but that's not real-time, it's batch/event processing at best efforts.
The biggest challenges with streaming are maintaining the content catalogue, which aren't just a few million records but rich metadata about the lifecycle and content relationships. Then user management and payments tends to also have a significant overhead, especially when you're talking about international payment processing.
This was before HTML5 and before the browser magically handled a lot of this… so there was definitely a bit more to it. Every company also wanted to have statistics of where people scrub to and all of that. It wasn’t super simple, but yeah, it also wasn’t crazy complex. The point is, scale is achievable without complex inf.
I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the perfect vision the lead(s) had.
Cue programmers blaming the product team for "always changing their mind" as they discover what users actually need, and the product team blaming developers for being hesitant to do changes, and when programmers agree, it takes a long time to undo the perfect architecture they've spent weeks fine-tuning against some imaginary future user-base.
Cue programmers blaming the product team for "always changing their mind" as they discover what users actually need, and the product team blaming developers for being hesitant to do changes, and when programmers agree, it takes a long time to undo the perfect architecture they've spent weeks fine-tuning against some imaginary future user-base.
In my opinion, if those influential programmers actually architected around some concrete metrics like 1,000 TPS and 10K daily users, they would end up with much simpler systems.
The problem I see is much more about extremely vague notions of scalability, trends, best practices, clean code, and so on. For example we need Kafka, because Kafka is for the big boys like us. Not because the alternatives couldn’t handle the actual numbers.
CV-driven development is a much bigger issue than people picking overly ambitious target numbers.
The problem I see is much more about extremely vague notions of scalability, trends, best practices, clean code, and so on. For example we need Kafka, because Kafka is for the big boys like us. Not because the alternatives couldn’t handle the actual numbers.
CV-driven development is a much bigger issue than people picking overly ambitious target numbers.
I was part of a small team that built a $300M company on Ruby and MySQL that made every scaling mistake you can possibly make. This was also the right decision because it forced us to stay lean and focus on what we needed right now, as opposed to getting starry-eyed about what it was going to be like when we had 10 million users. At every order of magnitude, we had sudden emergencies where some new part of the system had become a bottleneck, and we scrambled like crazy to rearchitect things to accommodate. It was hard, and it was fun. And it was frugal. We eventually hit over 10 million users before I left, and I can’t say I regret the painful approach one bit.
I also imagine you were pretty agile by not having tons of complexity to grapple with every time you wanted to add a new feature.
Haha, we were tech debt city. It certainly didn’t feel agile.
> 1000TPS and 10K daily users
I absolutely agree with your point, but I want to point out, like other commenters here, that the numbers should be much larger. We think that, because 10k daily users is a big deal for a product, they're also a big deal for a small server, but they really aren't.
It's fantastic that our servers nowadays can easily handle multiple tens of thousands of daily users on $100/mo.
I absolutely agree with your point, but I want to point out, like other commenters here, that the numbers should be much larger. We think that, because 10k daily users is a big deal for a product, they're also a big deal for a small server, but they really aren't.
It's fantastic that our servers nowadays can easily handle multiple tens of thousands of daily users on $100/mo.
Users/TPS aren't the right metric in the first place. I have a webhook glue side project that I didn't even realize had ~8k daily users/~300tps until I set up Cloudflare analytics. As a go program doing trivial work, the load is dwarfed by the cpu/memory usage of all my seedbox related software (which has 1 user, not even every day).
> Users/TPS aren't the right metric in the first place.
This was my initial point :) Don't focus on trying to achieve some metrics, focus on making sure to build the right thing.
This was my initial point :) Don't focus on trying to achieve some metrics, focus on making sure to build the right thing.
> We think that, because 10k daily users is a big deal for a product, they're also a big deal for a small server, but they really aren't.
Yeah we seem to forget just how fast computers are now a days. Obviously varies with complexity of the app & what other tech you are using, but for simpler things 10k daily users could be handled by a reasonbly powerful desktop sitting under my desk without breaking a sweat.
Yeah we seem to forget just how fast computers are now a days. Obviously varies with complexity of the app & what other tech you are using, but for simpler things 10k daily users could be handled by a reasonbly powerful desktop sitting under my desk without breaking a sweat.
I've seen senior engineers get fired and the business suffer a setback because they didn't have any way to scale beyond a single low spec VPS from a budget provider, and their system crashed when a hall full of students tried to sign up together during a demo and each triggered 200ms of bcrypt CPU activity.
This seems weird. I have a lot of experience with rails which is considered super slow. But the scenario you describe is trivial. Just get a bigger VPS and change a single env var. even if you fucked up everything else like file storage etc you can still to that. If you build your whole application in way where you can’t scale anything you should be fired. That is not even that easy
People screw up the bcrypt thing all the time. Pick a single threaded server stack (and run on one core, because Kubernetes), then configure bcrypt so brute forcing 8 character passwords is slow on an A100. Configure kubernetes to run on a medium range CPU because you have no load. Finally, leave your cloud provider's HTTP proxy's timeout set to default.
The result is 100% of auth requests timeout once the login queue depth gets above a hundred or so. At that point, the users retry their login attempts, so you need to scale out fast. If you haven't tested scale out, then it's time to implement a bcrypt thread pool, or reimplement your application.
But at least the architecture I described "scales".
The result is 100% of auth requests timeout once the login queue depth gets above a hundred or so. At that point, the users retry their login attempts, so you need to scale out fast. If you haven't tested scale out, then it's time to implement a bcrypt thread pool, or reimplement your application.
But at least the architecture I described "scales".
Fond memories of a job circa 2013 on a very large Rails app where CI times were sped up by a factor of 10 when someone realized bcrypt was misconfigured when running tests and slowing things down every time a user was created through a factory.
"because Kubernetes"? Is this assuming that you're running your server inside of a Kubernetes instance (and if so, is Kubernetes going to have problems with more than one thread?), or is there some other reason why it comes into this?
Of course you should be fired for doing that! I meant the example as an illustration of how "you don't need to scale" thinking turns into A-grade bullshit.
You do, in fact, need to scale to trivial numbers of users. You may even need to scale to a small number of users in the near future.
You do, in fact, need to scale to trivial numbers of users. You may even need to scale to a small number of users in the near future.
I'm not seeing how your example proves that a beefy server/cloud free architecture cannot handle the workload that most companies will encounter. The example you give of an under specified VPS is not what is being discussed in the article.
I was responding to CaptainOfColt, who was writing about premature optimisation killing companies. The article's proposed architecture seems fine and is similar to things I've done, but it's not an excuse to completely avoid thinking about future traffic patterns.
I will never forget the time my university's home-grown Web-based registration system crashed at the beginning of the semester, and the entirety of the university's student body had to form a line in order to have their registration entered manually. I waited a whole day, and they did not get round to me by night, so I had to wait the next day too.
“Knowing what’s reasonable” matters.
If you have a product that’s being deployed for a new school year, yeah you should be prepared for any one-time load for that time period.
Many products don’t have the “school year just started” spikes. But some do.
It requires careful thought, pragmatism, and business sense to balance everything and achieve the most with the available resources.
If you have a product that’s being deployed for a new school year, yeah you should be prepared for any one-time load for that time period.
Many products don’t have the “school year just started” spikes. But some do.
It requires careful thought, pragmatism, and business sense to balance everything and achieve the most with the available resources.
Wonder which one happens more often? Personally I haven't worked in that kind of "find the person to blame" culture which would led to something like that, so I haven't witnessed what you're talking about, but I believe you it does happen in some places.
That’s a skill issue, not an indictment on the limitations of the architecture. You can spin up N servers and load-balance them, as TFA points out. If the server is a snowflake and has nothing in IaC, again, not an architectural issue, but a personnel / knowledge issue.
The architecture in TFA is fine, and sounds preferable to microservices for most use cases.
I am worried by the talk of 10k daily users and a peak of 1000TPS being too much premature optimisation. Those numbers are quite low. You should know your expected traffic patterns, add a margin of error, and stress test your system to make sure it can handle the traffic.
I disagree that self-inflicted architectural issues and personnel issues are different.
I am worried by the talk of 10k daily users and a peak of 1000TPS being too much premature optimisation. Those numbers are quite low. You should know your expected traffic patterns, add a margin of error, and stress test your system to make sure it can handle the traffic.
I disagree that self-inflicted architectural issues and personnel issues are different.
> they didn't have any way to scale beyond a single low spec VPS from a budget provider
they couldn't redeploy to a high-spec VPS instead?
they couldn't redeploy to a high-spec VPS instead?
I frankly don't believe that in a workplace where an userbase can be characterized as a "hall full of students" anyone was fired overnight.
Doesn't happen at these places.
Reprimanded, maybe
More frequently, anyone that sounded the alarm about this was let go months ago, so the one that'd be fired is the one in charge of the firing.
Instead, they celebrate "learning from running at scale" or some nonsense.
Instead, they celebrate "learning from running at scale" or some nonsense.
Something that does not scale to 10k users is likely so badly architected, it would be faster to iterate on it if it was more scalable hence better architected and more maintainable.
For reference, in 1999 10K was still considered a (doable) challenge ... but they were talking "simultaneous" not "per day".
The modern equivalent challenge is 10 million simultaneous users per machine.
The modern equivalent challenge is 10 million simultaneous users per machine.
On the flip side, I've seen a project fail because it was built on the unvalidated assumption that the naive architecture would scale to real world loads only to find that a modest real world workload was exceeding targets by a factor of 100X. You really do need technical leadership with good judgment and experience; we can't substitute it with facile "assume low scale" or "assume large scale" axioms.
1000TPS isn't that much? Engineer for low latency and with a 10ms budget that'd be 10 cores if it were CPU-bound, less in practice since usually part of the time is spent in IO wait.
> 1000TPS isn't that much?
Why does that matter? My argument is: Engineer for what you know, leave the rest for when you know better, which isn't before you have lots of users.
Why does that matter? My argument is: Engineer for what you know, leave the rest for when you know better, which isn't before you have lots of users.
What I'm saying is that "building for 1000TPS" is not what gets you an overengineered 5-layer microservice architecture. If you build for a good user experience (which includes low latency) you get that not-that-big scale without sharding.
And with CPUs now being shipped with 100+ cores, you can brute force that sucker a long way.
I doubt much time would be in I/O wait if this was really a scale up architecture. Ignoring the 100's of GB of page cache, it should be sitting on NVMe drives, where a write is just a PCIe round trip, and a read is < 1ms.
Clearly this project failed for either
1. scaling for a very specific use case, or because
2. it hasn't even found product-market fit
> 1000TPS and 10K daily users
That is not a lot. You can host that on a Raspberry Pi.
That is not a lot. You can host that on a Raspberry Pi.
That entirely depends on what these transactions are meant to do.
I always find these debate weird. How can you compare one app’s TPS with another?
I always find these debate weird. How can you compare one app’s TPS with another?
Not if you’re going to be “web scale” (tm) you can’t.
You can host it on 8 raspberry pi's: Three for etcd, three for minio/ceph, and two for Kubernetes workers.
(16 if you need geo replication.)
(16 if you need geo replication.)
You put one Mongo shard on each Pi, they are the secret ingredient in the web scale sauce.
You simply can't get the software or support for a lot of smaller solutions. It can be sometimes easier to do the seemingly more difficult thing, and sometimes because all the money goes to those more difficult seeming technical problems and solutions.
Many startup business models have no chance of becoming profitable unless they reach a certain scale, but they might have less than 1% probability of reaching that scale. Making it scalable is easy work since it is deterministic, but growing customers is not.
Another perspective is that the defacto purpose of startups (and projects at random companies) may actually be work experience and rehearsal for the day the founders and friends get to interview at an actual FAANG.
I think the author's “dress for the job you want, not the job you have” nails it.
Another perspective is that the defacto purpose of startups (and projects at random companies) may actually be work experience and rehearsal for the day the founders and friends get to interview at an actual FAANG.
I think the author's “dress for the job you want, not the job you have” nails it.
Unfortunately, you can't really get experience from solving hypothetical problems. The actual problems you'll encounter are different, and while you can get experience in a particular "scalable" stack, it won't be worth its maintenance cost for a company that doesn't need it.
I guess the work is deterministic, but it often (unintentionally) makes the systems being developed non-deterministic!
Ah yes. I once worked at a startup that insisted on Mongo despite not having anywhere near the data volume for it to make any sense at all. Like, we're talking 5 orders of magnitude off of what one would reasonably expect to need a Mongo deployment.
I was but a baby engineer then, and the leads would not countenance anything as pedestrian as MySQL/Postgres.
Anyway, fast forward a bit and we were tasked with building an in-house messaging service. And at that point Mongo's eventual consistency became a roaring problem. Users would get notifications that they had a new message, and then when they tried to read it it was... well... not yet consistent.
We ended up implementing all kinds of ugly UX hacks to work around this, but really we could've run the entire thing off of sqlite on a single box and users would've been able to read messages instantaneously, so...
I was but a baby engineer then, and the leads would not countenance anything as pedestrian as MySQL/Postgres.
Anyway, fast forward a bit and we were tasked with building an in-house messaging service. And at that point Mongo's eventual consistency became a roaring problem. Users would get notifications that they had a new message, and then when they tried to read it it was... well... not yet consistent.
We ended up implementing all kinds of ugly UX hacks to work around this, but really we could've run the entire thing off of sqlite on a single box and users would've been able to read messages instantaneously, so...
I've seen similar with Firebase. Luckily I took over as tech lead at this company, so I was able to migrate us to Postgres. Amusingly, as well as being more reliable, the Postgres version (on a single small database instance) was also much faster than the previous Firebase-based version (due to it enabling JOINs in the database rather than in application code).
Funnily enough prior to this startup I had worked at a rainforest-themed big tech co where we ran all kinds of stuff on MySQL without issue, at scales that dwarfed what this startup was up to by 3-4 orders of magnitude.
I feel like that's kind of the other arm of this whole argument: on the one hand, you ain't gonna need that "scalable" thing. On the other hand, the "unscalable" thing scales waaaaaay higher than you are led to believe.
A single primary instance with a few read-only mirrors gets you a reaaaaaaally long way before you have to seriously think about doing something else.
I feel like that's kind of the other arm of this whole argument: on the one hand, you ain't gonna need that "scalable" thing. On the other hand, the "unscalable" thing scales waaaaaay higher than you are led to believe.
A single primary instance with a few read-only mirrors gets you a reaaaaaaally long way before you have to seriously think about doing something else.
> On the other hand, the "unscalable" thing scales waaaaaay higher than you are led to believe.
Agreeing with you... Any reasonable database will scale pretty far if you put in a machine with 160 cores and 3 TB of RAM. And that's just a single socket board.
There's no reason to do anything other than get bigger machines until you're near or at the limits of single socket. Dual socket and cpu generations should cover you for long enough to move to something else if you need to. Sharding a traditional database works pretty well in a lot of cases, and it mostly feels like the regular database.
Agreeing with you... Any reasonable database will scale pretty far if you put in a machine with 160 cores and 3 TB of RAM. And that's just a single socket board.
There's no reason to do anything other than get bigger machines until you're near or at the limits of single socket. Dual socket and cpu generations should cover you for long enough to move to something else if you need to. Sharding a traditional database works pretty well in a lot of cases, and it mostly feels like the regular database.
That, and a lot of companies don't have the scale they think they have.
The Postgres database for a company I worked for (that was very concerned about scaling when they interviewed me because their inefficient "nosql" solution was slow) ran very happily on a machine with 2 shared CPU cores and 4GB RAM.
The Postgres database for a company I worked for (that was very concerned about scaling when they interviewed me because their inefficient "nosql" solution was slow) ran very happily on a machine with 2 shared CPU cores and 4GB RAM.
Dawg tinder was operating at 20M DAU with all its DB based on Dynamo. Probably still is.
And yeah there was ton of those issues but yolo
And yeah there was ton of those issues but yolo
I watched a company you've probably heard of burn stupid amounts of money because one guy there was trying to build a personal brand as a Graph Database Expert, and another had fallen hard for Neo4j's marketing. Stability issues, stupid bugs, weak featureset, mediocre performance for most of the stuff they wanted to do (Neo4j, at least at this time, was tuned to perform some graph-related operations very fast, but it was extremely easy to find other graph-related operations that it's terrible at, and they're weren't exactly obscure things) all stretching out project development times to like 2x what they needed to be, with absolutely zero benefits for it. So fucking dumb.
Meanwhile all they needed was... frankly, probably SQLite, for their particular use case, having each client of theirs based around a single portable file actually would have been a big win for them. Their data for each client were tiny, like put-it-all-in-memory-on-an-RPi2 tiny. But no, "it's graphs so we need a graph database! Everything's graphs when you think about it, really! (So says Neo4j's marketing material, anyway)"
Meanwhile all they needed was... frankly, probably SQLite, for their particular use case, having each client of theirs based around a single portable file actually would have been a big win for them. Their data for each client were tiny, like put-it-all-in-memory-on-an-RPi2 tiny. But no, "it's graphs so we need a graph database! Everything's graphs when you think about it, really! (So says Neo4j's marketing material, anyway)"
>“dress for the job you want, not the job you have”
I don't think I should dress down any further :>
I don't think I should dress down any further :>
> Making it scalable is easy work since it is deterministic
I don't think that necessarily follows. Especially the language choice is almost impossible to change - look at Facebook, Dropbox, etc. Facebook ended up creating an entirely new language that only they use, because it was impossible to rewrite in another language.
Language choice (and probably database choice too) are essentially locked in from the start, and they do affect scaling.
Growing customers is probably harder, but I don't buy "do everything in hacky Bash scripts because you can fix it later". Nor do I think having solid foundations means you need to be less agile. Would Dropbox have been less successful if they wrote their backend in Typescript? I doubt it.
I don't think that necessarily follows. Especially the language choice is almost impossible to change - look at Facebook, Dropbox, etc. Facebook ended up creating an entirely new language that only they use, because it was impossible to rewrite in another language.
Language choice (and probably database choice too) are essentially locked in from the start, and they do affect scaling.
Growing customers is probably harder, but I don't buy "do everything in hacky Bash scripts because you can fix it later". Nor do I think having solid foundations means you need to be less agile. Would Dropbox have been less successful if they wrote their backend in Typescript? I doubt it.
I don't get this scalability craze either. Computers are stupid fast these days and unless you are doing something silly, it's difficult to run into CPU speed limitations.
I've been running a SaaS for 10 years now. Initially on a single server, after a couple of years moved to a distributed database (RethinkDB) and a 3-server setup, not for "scalability" but to get redundancy and prevent data loss. Haven't felt a need for more servers yet. No microservices, no Kubernetes, no AWS, just plain bare-metal servers managed through ansible.
I guess things look different if you're using somebody else's money.
I've been running a SaaS for 10 years now. Initially on a single server, after a couple of years moved to a distributed database (RethinkDB) and a 3-server setup, not for "scalability" but to get redundancy and prevent data loss. Haven't felt a need for more servers yet. No microservices, no Kubernetes, no AWS, just plain bare-metal servers managed through ansible.
I guess things look different if you're using somebody else's money.
It's not about scalability. It's about copying what the leaders in a space do, regardless of whether it makes sense or not. It's pervasive in most areas of life.
One of the silliest things you can do to cripple your performance is build something that is artificially over distributed, injecting lots of network delays between components, all of which have to be transited to fulfill a single user request. Monoliths are fast. Yes, sometimes you absolutely have to break something into a standalone service, but that’s rare.
I've notice a strong correlation between artificially over-distributing, and not understanding things like the CAP theorem. So, you end up with a slow system that's added a bunch of unsolvable distributed systems problems on its fast path.
(Most distributed systems problems are solvable, but only if the person that architected the system knows what they're doing. If they know what they're doing, they won't over-distribute stuff.)
(Most distributed systems problems are solvable, but only if the person that architected the system knows what they're doing. If they know what they're doing, they won't over-distribute stuff.)
Yes, that too. If you look at the commits for Heisenbugs associated with the system, you have a good chance of seeing artificial waits injected to “fix” things.
You can solve just about any distributed systems problem by accepting latency, but nobody wants to accept latency :)
...despite the vast majority of latency issues being extremely low-hanging fruit, like "maybe don't have tens of megabytes of data required to do first paint on your website" or "hey maybe have an index in that database?".
...despite the vast majority of latency issues being extremely low-hanging fruit, like "maybe don't have tens of megabytes of data required to do first paint on your website" or "hey maybe have an index in that database?".
Well, yeah, but the people that create the issues typically solve them by just corrupting the crap out of app state and adding manual ops procedures.
There's no need to deploy separate service on separate machines.
I ran a SaaS for 10 years. Two products. Profitable from day 1 as customers paid $500/month and it ran on a couple of EC2 instances as well as a small RDS database.
Another thing one has to consider is the market size and timeframe window of your SaaS. No sense in building for scalability if the business opportunity is only 100 customers and only for a few years.
Another thing one has to consider is the market size and timeframe window of your SaaS. No sense in building for scalability if the business opportunity is only 100 customers and only for a few years.
> unless you are doing something silly, it's difficult to run into CPU speed limitations.
Yes, but it's not difficult to do something silly without even noticing until too late. Implicitly (and unintentionally) calling something with the wrong big-O, for example.
That said, anyone know what's up with the slow deletion of Safari history? Clearly O(n), but as shown in this blog post still only deleted at a rate of 22 items in 10 seconds: https://benwheatley.github.io/blog/2025/06/19-15.56.44.html
Yes, but it's not difficult to do something silly without even noticing until too late. Implicitly (and unintentionally) calling something with the wrong big-O, for example.
That said, anyone know what's up with the slow deletion of Safari history? Clearly O(n), but as shown in this blog post still only deleted at a rate of 22 items in 10 seconds: https://benwheatley.github.io/blog/2025/06/19-15.56.44.html
>> Yes, but it's not difficult to do something silly without even noticing until too late. Implicitly (and unintentionally) calling something with the wrong big-O, for example.
On a non-scalable system you're going to notice that big-O problem and correct it quickly. On a scalable system you're not going to notice it until you get your AWS bill.
On a non-scalable system you're going to notice that big-O problem and correct it quickly. On a scalable system you're not going to notice it until you get your AWS bill.
Also, instead of having a small team of people to fight scalable infrastructure configuration, you could put 1-2 full time engineers on performance engineering. They'd find big-O and constant factor problems way before they mattered in production.
Of course, those people's weekly status reports would always be "we spent all week tracking down a dumb mistake, wrote one line of code and solved a scaling problem we'd hit at 100x our current scale".
That's equivalent to waving a "fire me" flag at the bean counters and any borderline engineering managers.
Of course, those people's weekly status reports would always be "we spent all week tracking down a dumb mistake, wrote one line of code and solved a scaling problem we'd hit at 100x our current scale".
That's equivalent to waving a "fire me" flag at the bean counters and any borderline engineering managers.
For how many users, and at what transaction rate?
Not disagreeing that you can do a lot on a lot less than in the old days, but your story would be much more impactful with that information. :)
Not disagreeing that you can do a lot on a lot less than in the old days, but your story would be much more impactful with that information. :)
Scalability isn't just about CPU.
It's just as much about storage and IO and memory and bandwidth.
Different types of sites have completely different resource profiles.
It's just as much about storage and IO and memory and bandwidth.
Different types of sites have completely different resource profiles.
Microservice is not a solution for scalability. There are multiple options for building scalable software, even a monolith or a modular monolith with proper loadbalanced setup will drastically reduce the complexity of microservice and get massive scale. Only bottleneck will be db.
Microservices take an organizational problem:
The teams don't talk, and always blame each other
and adds distributed systems and additional organizational problems:
Each team implements one half of dozens of bespoke network protocols, but they still don't talk, and still always blame each other. Also, now they have access to weaponizable uptime and latency metrics, since because each team "owns" the server half of one network endpoint, but not the client half.
The teams don't talk, and always blame each other
and adds distributed systems and additional organizational problems:
Each team implements one half of dozens of bespoke network protocols, but they still don't talk, and still always blame each other. Also, now they have access to weaponizable uptime and latency metrics, since because each team "owns" the server half of one network endpoint, but not the client half.
Is it about scalability, or about resiliency ?
Or is it about outsourcing problems?
There's a lot of off the shelf microservices that can solve difficult problems for me. Like keycloak for user management. Isn't that a good reason?
Or Grafana for log visualization?
Should I build that into the monolith too? Or should I just skip it?
There's a lot of off the shelf microservices that can solve difficult problems for me. Like keycloak for user management. Isn't that a good reason?
Or Grafana for log visualization?
Should I build that into the monolith too? Or should I just skip it?
>Modules cannot call each other, except through specific interfaces (for our Python monolith, we put those in a file called some_module/api.py, so other modules can do from some_module.api import some_function, SomeClass and call things that way.
This is a solution to a large chunk of what people want out of microservices. There are just two problems, both of which feel tractable to a language/runtime that really wanted to solve them:
1. If the code implementing the module API is private, it must all be colocated in one package. If it is public, then anyone can import it, breaking the module boundary. You need a visibility system that can say "this subtree of packages can cooperate with each other, but code outside the subtree can only use the top-level package."
2. If a change module A has a problem, you must roll back the entire monolith, preventing a good change in module B from reaching users. You need a way to change the deployed version of different modules independently. Short of microservices, they could be separate processes doing some kind of IPC, or you need a runtime with hot reloading (and a reasonably careful backwards compatibility story).
This is a solution to a large chunk of what people want out of microservices. There are just two problems, both of which feel tractable to a language/runtime that really wanted to solve them:
1. If the code implementing the module API is private, it must all be colocated in one package. If it is public, then anyone can import it, breaking the module boundary. You need a visibility system that can say "this subtree of packages can cooperate with each other, but code outside the subtree can only use the top-level package."
2. If a change module A has a problem, you must roll back the entire monolith, preventing a good change in module B from reaching users. You need a way to change the deployed version of different modules independently. Short of microservices, they could be separate processes doing some kind of IPC, or you need a runtime with hot reloading (and a reasonably careful backwards compatibility story).
Though even the solution to 1 doesn't solve "ACLs", which distribution does. If you want to ensure your module is only called by an approved list of upstream modules, public / private isn't granular enough. (You can solve it with attributes or some build tools, but it's ad-hoc and complex, doesn't integrate with the editor, etc. I've always thought something more granular and configurable should've been built into Java/OOP theory from the start).
That said, 2 is really the big problem. As things really scale, this tends to cause problems on every deployment and slow the whole company down, cause important new features to get blocked by minor unrelated changes, a lot of extra feature flag maintenance, etc. 90% of the time, that should be the gating factor by which you decide went to split a service into multiple physical components.
As the author said, an additional reason for distribution is sometimes it's prudent to distribute because of physical scale reasons (conflicts between subservice A needing high throughput, B needing low latency, C needing high availability, and D needing high IOPS that blows your budget to have VMs that satisfy every characteristic, or impossible in memory-managed languages), but usually I see this being done way too early, based more on ideals than numbers.
That said, 2 is really the big problem. As things really scale, this tends to cause problems on every deployment and slow the whole company down, cause important new features to get blocked by minor unrelated changes, a lot of extra feature flag maintenance, etc. 90% of the time, that should be the gating factor by which you decide went to split a service into multiple physical components.
As the author said, an additional reason for distribution is sometimes it's prudent to distribute because of physical scale reasons (conflicts between subservice A needing high throughput, B needing low latency, C needing high availability, and D needing high IOPS that blows your budget to have VMs that satisfy every characteristic, or impossible in memory-managed languages), but usually I see this being done way too early, based more on ideals than numbers.
Exactly. There should be no problem having tens of thousands of stateless, I/O bound database-transaction-wrapper endpoints in the same service. You're not going to run out of memory to hold the binary or something. If you want to partition physical capacity between groups of endpoints, you can probably accomplish that at the load balancer.
Having the latent capability to serve endpoint A in the binary is not interfering with endpoint B's QPS unless it implies some kind of crazy background job or huge in-memory dataset. Even in this case, monoliths normally have a few different components according to function: API, DB, cache, queue, background worker, etc. You can group workloads by their structure even if their business purposes are diverse.
Having the latent capability to serve endpoint A in the binary is not interfering with endpoint B's QPS unless it implies some kind of crazy background job or huge in-memory dataset. Even in this case, monoliths normally have a few different components according to function: API, DB, cache, queue, background worker, etc. You can group workloads by their structure even if their business purposes are diverse.
> 1. If the code implementing the module API is private, it must all be colocated in one package. If it is public, then anyone can import it, breaking the module boundary. You need a visibility system that can say "this subtree of packages can cooperate with each other, but code outside the subtree can only use the top-level package."
This is easily achieved in Scala with a particular use of package paths. You are allowed to make some symbols only visible under a package path.
This is easily achieved in Scala with a particular use of package paths. You are allowed to make some symbols only visible under a package path.
Unfortunately the thought pattern “modern software development is way too complex, we need something radically simpler… I know! I’ll use Scala!” is not a thing.
> This is a solution to a large chunk of what people want out of microservices.
Yeah, just put some gRPC services in a single pod and have them communicate over unix domain socket. You now have a single unit of modules that can only communicate over IPC using a well-defined type-safe boundary. As a bonus you can set resource quotas separately according to the needs of the module.
Want to scale? Rewrite some config and expose a few internal services and have them communicate over TCP.
Yeah, just put some gRPC services in a single pod and have them communicate over unix domain socket. You now have a single unit of modules that can only communicate over IPC using a well-defined type-safe boundary. As a bonus you can set resource quotas separately according to the needs of the module.
Want to scale? Rewrite some config and expose a few internal services and have them communicate over TCP.
Visibility systems are great!
> If a change module A has a problem, you must roll back the entire monolith, preventing a good change in module B from reaching users.
eh. In these setups you really want to be fixing forward for the reason you describe - so you revert the commit for feature A or turn off the feature flag for it or something. You don't really want to be reverting deployments. If you have to, well, then it's probably worth the small cost of feature B being delayed. But there are good solutions to shipping multiple features at the same time without conflicting.
> If a change module A has a problem, you must roll back the entire monolith, preventing a good change in module B from reaching users.
eh. In these setups you really want to be fixing forward for the reason you describe - so you revert the commit for feature A or turn off the feature flag for it or something. You don't really want to be reverting deployments. If you have to, well, then it's probably worth the small cost of feature B being delayed. But there are good solutions to shipping multiple features at the same time without conflicting.
There is a point in commit throughput at which finding a working combination of reverts becomes unsustainable. I've seen it. Feature flagging can delay this point but probably not prevent it unless you’re isolating literally every change with its own flag (at which point you have a weird VCS).
> scalability needs a whole bunch of complexity
I am not sure this is true. Complexity is a function of architecture. Scalability can be achieved by abstraction, it doesn't necessarily imply highly coupled architecture, in fact scalability benefits from decoupling as much as possible, which effectively reduces complexity.
If you have a simple job to do that fits in an AWS Lambda, why not deploy it that way, scalability is essentially free. But the real advantage is that by writing it as a Lambda you are forced to think of it in stateless terms. On the other hand if suddenly it needs to coordinate with 50 other Lambdas or services, then you have complexity -- usually scalability will suffer in this case, as things become more and more synchronous and interdependent.
> The monolith is composed of separate modules (modules which all run together in the same process).
It's of course great to have a modular architecture, but whether or not they run in the same process should be an implementation detail. Barriers should be explicit. By writing it all depending on local, synchronous, same-process logic, you are likely building in all sorts of implicit barriers that will become hidden dangers when suddenly you do need to scale. And by the way that's one of the reasons we think about scaling in advance, is that when the need comes, it comes quickly.
It's not that you should scale early. But if you're designing a system architecture, I think it's better to think about scaling, not because you need it, but because doing so forces you to modularize, decouple, and make synchronization barriers explicit. If done correctly, this will lead to a better, more robust system even when it's small.
Just like premature optimization -- it's better not to get caught up doing it too early, but you still want to design your system so that you'll be able to do it later when needed, because that time will come, and the opportunity to start over is not going to come as easily as you might imagine.
I am not sure this is true. Complexity is a function of architecture. Scalability can be achieved by abstraction, it doesn't necessarily imply highly coupled architecture, in fact scalability benefits from decoupling as much as possible, which effectively reduces complexity.
If you have a simple job to do that fits in an AWS Lambda, why not deploy it that way, scalability is essentially free. But the real advantage is that by writing it as a Lambda you are forced to think of it in stateless terms. On the other hand if suddenly it needs to coordinate with 50 other Lambdas or services, then you have complexity -- usually scalability will suffer in this case, as things become more and more synchronous and interdependent.
> The monolith is composed of separate modules (modules which all run together in the same process).
It's of course great to have a modular architecture, but whether or not they run in the same process should be an implementation detail. Barriers should be explicit. By writing it all depending on local, synchronous, same-process logic, you are likely building in all sorts of implicit barriers that will become hidden dangers when suddenly you do need to scale. And by the way that's one of the reasons we think about scaling in advance, is that when the need comes, it comes quickly.
It's not that you should scale early. But if you're designing a system architecture, I think it's better to think about scaling, not because you need it, but because doing so forces you to modularize, decouple, and make synchronization barriers explicit. If done correctly, this will lead to a better, more robust system even when it's small.
Just like premature optimization -- it's better not to get caught up doing it too early, but you still want to design your system so that you'll be able to do it later when needed, because that time will come, and the opportunity to start over is not going to come as easily as you might imagine.
> If you have a simple job to do that fits in an AWS Lambda, why not deploy it that way, scalability is essentially free. But the real advantage is that by writing it as a Lambda you are forced to think of it in stateless terms.
What you are describing is already the example of premature optimization. The moment you are thinking of a job in terms of "fits in an AWS Lambda" you are automatically stuck with "Use S3 to store the results" and "use a queue to manage the jobs" decisions.
You don't even know if that job is the bottleneck that needs to scale. For all you know, writing a simple monolithic script to deploy onto a VM/server would be a lot simpler deployment. Just use the ram/filesystem as the cache. Write the results to the filesystem/database. When the time comes to scale you know exactly which parts of your monolith are the bottleneck that need to be split. For all you know - you can simply replicate your monolith, shard the inputs and the scaling is already done. Or just use the DB's replication functionality.
To put things into perspective, even a cheap raspberry pi/entry level cloud VM gives you thousands of postgres queries per second. Most startups I worked at NEVER hit that number. Yet their deployment stories started off with "let's use lambdas, s3, etc..". That's just added complexity. And a lot of bills - if it weren't for the "free cloud credits".
What you are describing is already the example of premature optimization. The moment you are thinking of a job in terms of "fits in an AWS Lambda" you are automatically stuck with "Use S3 to store the results" and "use a queue to manage the jobs" decisions.
You don't even know if that job is the bottleneck that needs to scale. For all you know, writing a simple monolithic script to deploy onto a VM/server would be a lot simpler deployment. Just use the ram/filesystem as the cache. Write the results to the filesystem/database. When the time comes to scale you know exactly which parts of your monolith are the bottleneck that need to be split. For all you know - you can simply replicate your monolith, shard the inputs and the scaling is already done. Or just use the DB's replication functionality.
To put things into perspective, even a cheap raspberry pi/entry level cloud VM gives you thousands of postgres queries per second. Most startups I worked at NEVER hit that number. Yet their deployment stories started off with "let's use lambdas, s3, etc..". That's just added complexity. And a lot of bills - if it weren't for the "free cloud credits".
> The moment you are thinking of a job in terms of "fits in an AWS Lambda" you are automatically stuck with "Use S3 to store the results" and "use a queue to manage the jobs" decisions.
I think the most important one you get is that inputs/outputs must always be < 6mb in size. It makes sense as a limitation for Lambda's scalability, but you will definitely dread it the moment a 6.1mb use case makes sense for your application.
I think the most important one you get is that inputs/outputs must always be < 6mb in size. It makes sense as a limitation for Lambda's scalability, but you will definitely dread it the moment a 6.1mb use case makes sense for your application.
The counterargument to this point is also incredibly weak: It forces you to have clean interfaces to your functions, and to think about where the application state lives, and how it's passed around inside your application.
That's equivalent to paying attention in software engineering 101. If you can't get those things right on one machine, you're going to be in world of hurt dealing with something like lambda.
That's equivalent to paying attention in software engineering 101. If you can't get those things right on one machine, you're going to be in world of hurt dealing with something like lambda.
I'd say the real advantage is that if you need to change it you don't have to deploy your monolith. Of course, the relative benefit of that is situationally dependent, but I was recently burned by a team that built a new replication handler we needed into their monolith, and every time it had a bug, and the monolith only got deployed once a week. I begged them to put it into a lambda but every week was "we'll get it right next week", for months. So it does happen.
That’s orthogonal to microservices. They could deploy the monolith multiple times a day.
Of course, that’d require CI, which clearly wasn’t working well in your example.
Of course, that’d require CI, which clearly wasn’t working well in your example.
Correct. It's orthogonal in an ideal world, but in a real world where there's tech debt and competing priorities, it can very much come into play.
> It's of course great to have a modular architecture, but whether or not they run in the same process should be an implementation detail
It should be, but I think "microservices" somehow screwed up that. Many developers think "modular architecture == separate services communicating via HTTP/network that can be swapped", failing to realize you can do exactly what you're talking about. It doesn't really matter what the barrier is, as long as it's clear, and more often than not, network seems to be the default barrier when it doesn't have to be.
It should be, but I think "microservices" somehow screwed up that. Many developers think "modular architecture == separate services communicating via HTTP/network that can be swapped", failing to realize you can do exactly what you're talking about. It doesn't really matter what the barrier is, as long as it's clear, and more often than not, network seems to be the default barrier when it doesn't have to be.
> network seems to be the default barrier when it doesn't have to be.
But if you want to use off the shelf solutions to your problems it often is. You can't very well do 'from keycloak import login_page'.
But if you want to use off the shelf solutions to your problems it often is. You can't very well do 'from keycloak import login_page'.
The complexity that makes money is all the essential complexity of the problem domain. The "complexity in the architecture" can only add to that (and often does).
This is the part that is about math as a language for patterns as well as research for finding counter-examples. It’s not an engineering problem yet.
Once you have product market fit, then it becomes and engineering problem.
This is the part that is about math as a language for patterns as well as research for finding counter-examples. It’s not an engineering problem yet.
Once you have product market fit, then it becomes and engineering problem.
> The first problem every startup solves is scalability. The first problem every startup should solve is “how do we have enough money to not go bust in two months”, but that’s a hard problem, whereas scalability is trivially solvable by reading a few engineering blogs, and anyway it’s not like anyone will ever call you out on it, since you’ll go bust in two months.
I laughed. I cried. Having a back full of microservices scars, I can attest that everything said here is true. Just build an effin monolith and get it done.
I laughed. I cried. Having a back full of microservices scars, I can attest that everything said here is true. Just build an effin monolith and get it done.
Ugh, there is just something so satisfying about developer cynicism. It gives me that warm, fuzzy feeling.
I basically agree with most of what the author is saying here, and I think that my feeling is that most developers are at least aware that they should resist technical self-pleasure in pursuit of making sure the business/product they're attached to is actually performing. Are there really people out there who still reach for Meta-scale by default? Who start with microservices?
I basically agree with most of what the author is saying here, and I think that my feeling is that most developers are at least aware that they should resist technical self-pleasure in pursuit of making sure the business/product they're attached to is actually performing. Are there really people out there who still reach for Meta-scale by default? Who start with microservices?
> Are there really people out there who still reach for Meta-scale by default? Who start with microservices?
Anecdotally, the last three greenfield projects I was a part of, the Architects (distinct people in every case) began the project along the lines of "let us define the microservices to handle our domains".
Every one of those projects failed, in my opinion not primarily owing to bad technical decisions - but they surely didn't help either by making things harder to pivot, extend and change.
Clean Code ruined a generation of engineers IMO.
Anecdotally, the last three greenfield projects I was a part of, the Architects (distinct people in every case) began the project along the lines of "let us define the microservices to handle our domains".
Every one of those projects failed, in my opinion not primarily owing to bad technical decisions - but they surely didn't help either by making things harder to pivot, extend and change.
Clean Code ruined a generation of engineers IMO.
I think this sounds more like Domain Driven Design than Clean Code.
It kinda started with Clean Code. I remember some old colleagues walking around with the book in their hand and deleting ten year old comments in every commit they made: "You see, we don't need that anymore, because the code describes itself". It made a generation (generations?) of software developers think that all the architectural patterns were found now, we can finally do real engineering and just have to find the one that fits for the problem at hand! Everyone asked the SOLID principles during interviews, because that's how real engineers design! I think "cargo cult" was getting used at that time too to describe this phenomenon.
It was (is) bad. The worst part is they the majority of people pushing it haven’t even read Clean Code. They’ve read a blog post by a guy who read a blog post by a guy who skimmed the book.
I don't buy the idea that people mainly reach for microservices for scalability or "pleasure" reasons though.
I personally reach for it to outsource some problems by using off the shelf solutions. I don't want to reinvent the wheel. And if everyone else is doing it in a certain way I want to do it in the same way to try to stand on the shoulders of giants and not reinvent everything.
But that's probably the wrong approach then...
I personally reach for it to outsource some problems by using off the shelf solutions. I don't want to reinvent the wheel. And if everyone else is doing it in a certain way I want to do it in the same way to try to stand on the shoulders of giants and not reinvent everything.
But that's probably the wrong approach then...
Yes, there are still people who start with microservices, unfortunately. There are where I work.
I needed to build an internal admin console, not super-scalable, just a handful of business users to start. The SQL database it would access was on-premises, but might move to the cloud in future. Authorized users needed single sign-on to their Azure-based active directory accounts for login. I wanted to do tracing of user requests with OpenTelemetry or something like.
At this point in my career, why wouldn't I reach for microservices to supply the endpoints that my frontend calls out to? Microservices are straightforward to implement with NodeJS (or any other language, for that matter.) I get very straightforward tracing and Azure SSO support in NodeJS. For my admin console, I figured I would need one backend-for-frontend microservice that the frontend would connect to and a domain service for each domain that needed to be represented (with only one domain to start). We picked server technologies and frameworks that could easily port to the cloud.
So two microservices to implement a secure admin console from scratch, is that too many? I guess I lack the imagination to do the project differently. I do enjoy the "API First" approach and the way it lets me engage meaningfully with the business folks to come up with a design before we write any code. I like how it's easy to unit/functional test with microservices, very tidy.
Perhaps what makes a lot/most of microservice development so gross is misguided architectural and deployment goals. Like, having a server/cluster per deployed service is insane. I deploy all of my services monolithically until a service has some unique security or scaling needs that require it to separate from the others.
Similarly, it seems common for microservices teams to keep multiple git repos, one for each service. Why?! Some strange separation-of-concerns/purity ideals. Code reuse, testing, pull requests, and atomic releases suffer needless friction unless everything is kept in a monorepo, as the OP implied.
Also, when teams build microservices in such a way that services must call other services completely misses the point of services - that's just creating a distributed monolith (slow!)
I made a rule on my team that the only service type that can call another service is aggregation services like my backend-for-frontend which could launch downstream calls in parallel and aggregate the results for the caller. This made the architecture very flat with the minimum number of network hops and with as much parallelism as possible so it would stay performant. Domain services owned their data sources, no drama with backend data.
I see a lot of distributed monolith drama and abuse of NoSQL data sources giving microservices a bad reputation.
At this point in my career, why wouldn't I reach for microservices to supply the endpoints that my frontend calls out to? Microservices are straightforward to implement with NodeJS (or any other language, for that matter.) I get very straightforward tracing and Azure SSO support in NodeJS. For my admin console, I figured I would need one backend-for-frontend microservice that the frontend would connect to and a domain service for each domain that needed to be represented (with only one domain to start). We picked server technologies and frameworks that could easily port to the cloud.
So two microservices to implement a secure admin console from scratch, is that too many? I guess I lack the imagination to do the project differently. I do enjoy the "API First" approach and the way it lets me engage meaningfully with the business folks to come up with a design before we write any code. I like how it's easy to unit/functional test with microservices, very tidy.
Perhaps what makes a lot/most of microservice development so gross is misguided architectural and deployment goals. Like, having a server/cluster per deployed service is insane. I deploy all of my services monolithically until a service has some unique security or scaling needs that require it to separate from the others.
Similarly, it seems common for microservices teams to keep multiple git repos, one for each service. Why?! Some strange separation-of-concerns/purity ideals. Code reuse, testing, pull requests, and atomic releases suffer needless friction unless everything is kept in a monorepo, as the OP implied.
Also, when teams build microservices in such a way that services must call other services completely misses the point of services - that's just creating a distributed monolith (slow!)
I made a rule on my team that the only service type that can call another service is aggregation services like my backend-for-frontend which could launch downstream calls in parallel and aggregate the results for the caller. This made the architecture very flat with the minimum number of network hops and with as much parallelism as possible so it would stay performant. Domain services owned their data sources, no drama with backend data.
I see a lot of distributed monolith drama and abuse of NoSQL data sources giving microservices a bad reputation.
ThoughtWorks gathers this phenomenon under the term "envy": Web Scale envy [1] or Big Data envy [2] are two relevant blips on their technology radar. It is typically better to keep things simple.
[1] https://www.thoughtworks.com/radar/techniques/high-performan...
[2] https://www.thoughtworks.com/radar/techniques/big-data-envy
[1] https://www.thoughtworks.com/radar/techniques/high-performan...
[2] https://www.thoughtworks.com/radar/techniques/big-data-envy
I've seen my share of insanely over-engineered Azure locked-in applications that could easily have been run on an open source stack on a $20 VM.
But what if payroll grows to 100M internal users?
I sense sarcasm but just to add on, most software problems actually have very predictable natural upper bounds. There are only X number of people in the world growing at a well-defined rate. There are only X number of people in your country, or in your TAM (e.g. if you're a restaurant) etc. This is especially true for B2B.
The need to accommodate runaway scale (unbounded N and unbounded rate of growth of N) is actually quite rare.
The need to accommodate runaway scale (unbounded N and unbounded rate of growth of N) is actually quite rare.
Yea, and what if my country suddenly grows by 10000%?
Seems like a great problem to have. Surely one of those millions of employees can be used to change the current system at that point. Until then, no reason to overengineer it.
Hilariously written but also too true.
One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time.
Another place I worked at we were paying for 2 dev ops engineers (and those guys don't come cheap) to maintain our deployment cluster for 3 apps which each had a single customer (with a handful of users). This whole operation had like 20 people and an engineering team of 8.
One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time.
Another place I worked at we were paying for 2 dev ops engineers (and those guys don't come cheap) to maintain our deployment cluster for 3 apps which each had a single customer (with a handful of users). This whole operation had like 20 people and an engineering team of 8.
This sounds just about right: I have read that Kubernetes is the greek term for "more containers than customers".
What were these dev ops engineers doing all day? Surely you can only polish a cluster so much before it's done and there is nothing left to do?
You should have seen the architecture they came up with... it had ALL the bells and whistles you could possibly imagine and cost an absolute fortune.
Of course they eventually got bored and quit. And then it became really annoying since no one else understood anything about it.
Of course they eventually got bored and quit. And then it became really annoying since no one else understood anything about it.
It takes approximately 3 months to get it "just right". Luckily, k8s releases (and CNI and auth sidecars, and...) release every 2.8 months.
I work at a place with 8 k8s clusters. We needed to evolve from generation 2 to generation 3 because of "manageability" or something. Gen 3 needed two clusters instead of one. Now we have 8 * (1 + 2) = 24 clusters.
Happy days.
Happy days.
Are they aware that namespaces exist? Surely you're getting more out of your clusters by having few clusters running many pods instead of many clusters running few pods?
At my job we also have some redundant clusters but that's because we're in the middle of a transition (really two transitions, the first of which was never completed), of the 10 clusters that fall under my responsibility 6 will hopefully be gone by the end of this year.
At my job we also have some redundant clusters but that's because we're in the middle of a transition (really two transitions, the first of which was never completed), of the 10 clusters that fall under my responsibility 6 will hopefully be gone by the end of this year.
Ah yes, transitions. Ours will also finish soon. Enjoy. :)
We have the same shit and its super annoying too cause in addition I cant do shit without going through the dev ops team even though were 5 engineers.
> No, no you don’t, you can deploy your modules to their own repos and work on them that way, though then you lose the nice property of being able to atomically deploy changes across your entire codebase when an API changes.
That's not all you lose. You also lose being able to have a single git SHA to describe the state of the entire system at any time. And, lose the naturalness of running CI on the entire system at a given state (and knowing what CI ran on what state), although you can rig that up unnaturally of course.
That's not all you lose. You also lose being able to have a single git SHA to describe the state of the entire system at any time. And, lose the naturalness of running CI on the entire system at a given state (and knowing what CI ran on what state), although you can rig that up unnaturally of course.
I hear you about rigging those repos together artificially.
I used to work somewhere that had several different systems written as monoliths that eventually needed to interact more closely with each other. There was an in-house ticket system (since made a support service wrapped around ZenDesk, but ZD wouldn’t replicate all its functionality). There was an in-house employee management system. There was a first-party CRM. There was a homegrown e-commerce store. They’d built their own licensing servers for their software. Eventually the CRM was managing large customer licenses, the ticket system linked to the store to sell priority support, and the store was selling licenses to smaller customers.
So instead of making these things all support APIs that supported the other applications, people started copying libraries around from service to service. Then to make sure those libraries didn’t fall out of date, the (at the time rsync) deployment process for each of those apps was changed to require a pull from every one of those repos, then a push from that staging server to the production servers. Then security did a PCI-DSS internal audit, and the developers couldn’t just get onto a staging server and make direct changes to production.
So I, as the lead SRE at the time, wrote a builder web app that takes a config file per project. It holds the data on the repos and the default tag to pull from each. The web app allows the developer to update to a different commit or tag for any particular repo involved. Then a single button pulls everything and serially takes production servers out of rotation, updates them, and puts them back into production. It’s something that could have been avoided many different ways including using a monorepo for those systems.
I used to work somewhere that had several different systems written as monoliths that eventually needed to interact more closely with each other. There was an in-house ticket system (since made a support service wrapped around ZenDesk, but ZD wouldn’t replicate all its functionality). There was an in-house employee management system. There was a first-party CRM. There was a homegrown e-commerce store. They’d built their own licensing servers for their software. Eventually the CRM was managing large customer licenses, the ticket system linked to the store to sell priority support, and the store was selling licenses to smaller customers.
So instead of making these things all support APIs that supported the other applications, people started copying libraries around from service to service. Then to make sure those libraries didn’t fall out of date, the (at the time rsync) deployment process for each of those apps was changed to require a pull from every one of those repos, then a push from that staging server to the production servers. Then security did a PCI-DSS internal audit, and the developers couldn’t just get onto a staging server and make direct changes to production.
So I, as the lead SRE at the time, wrote a builder web app that takes a config file per project. It holds the data on the repos and the default tag to pull from each. The web app allows the developer to update to a different commit or tag for any particular repo involved. Then a single button pulls everything and serially takes production servers out of rotation, updates them, and puts them back into production. It’s something that could have been avoided many different ways including using a monorepo for those systems.
There's a now famous Harvard lecture video on YouTube of Zuckerberg earlier in the Facebook days, where he walks through the issues they hit early on.
https://www.youtube.com/watch?v=xFFs9UgOAlE
I watched it ages ago, but I seem to remember one thing that I liked was that each time they changed the architecture, it was to solve a problem they had, or were beginning to have. They seemed to be staying away from pre-optimization and instead took the approach of tackling problems as they had as they appeared, rather than imagining problems long before/if they occurred.
It's a bit like the "perfect is the enemy of done" concept - you could spend 2-3x the time making it much more scalable, but that might have an opportunity cost which weakens you somewhere else or makes it harder/more expensive to maintain and support.
Take it with a pinch of salt, but I thought it seemed like quite a good level-headed approach to choosing how to spend time/money early on, when there's a lot of financial/time constraints.
https://www.youtube.com/watch?v=xFFs9UgOAlE
I watched it ages ago, but I seem to remember one thing that I liked was that each time they changed the architecture, it was to solve a problem they had, or were beginning to have. They seemed to be staying away from pre-optimization and instead took the approach of tackling problems as they had as they appeared, rather than imagining problems long before/if they occurred.
It's a bit like the "perfect is the enemy of done" concept - you could spend 2-3x the time making it much more scalable, but that might have an opportunity cost which weakens you somewhere else or makes it harder/more expensive to maintain and support.
Take it with a pinch of salt, but I thought it seemed like quite a good level-headed approach to choosing how to spend time/money early on, when there's a lot of financial/time constraints.
Actually, scalability is cheap. Our AWS bill until recently was around $160-$200 a month. To get the level of HA and performance would require at least 20 boxes in two data centers.
Dev/test/prod with an HA db and a backend that never dies. I’ve built those on bare iron and they’re expensive.
If you’re going for saas and customers that don’t care about your infrastructure then a hetzner box is fine.
But really, creating resilient infrastructure is super cheap now.
Agree with “make what your customers want,” but many customers actually want a service that doesn’t barf.
Dev/test/prod with an HA db and a backend that never dies. I’ve built those on bare iron and they’re expensive.
If you’re going for saas and customers that don’t care about your infrastructure then a hetzner box is fine.
But really, creating resilient infrastructure is super cheap now.
Agree with “make what your customers want,” but many customers actually want a service that doesn’t barf.
Could you explain this some more? How are your costs so low in comparison? Are you using serverless?
That's how the cloudy platforms get you. They're very cheap on the low end, until they're not.
No, it's because we changed the way we process our metrics.
Previously we processed our metrics by consolidating them into multidimensional entries on a minute basis.
We moved to single metric second-based collection, because it was getting too complicated to process and because we wanted second-by-second measurement to measure engagement more granularly. That increased our data retention tremendously. We're still under the cost for the other timestream products, but we'll be adjusting how we do that in a quarter or two.
Previously we processed our metrics by consolidating them into multidimensional entries on a minute basis.
We moved to single metric second-based collection, because it was getting too complicated to process and because we wanted second-by-second measurement to measure engagement more granularly. That increased our data retention tremendously. We're still under the cost for the other timestream products, but we'll be adjusting how we do that in a quarter or two.
I've heard this a few times. Can you explain a bit more why you think that's a problem.
I've always made the assumption that once they become "not cheap" you now have the cost to offset investment against.
I've always made the assumption that once they become "not cheap" you now have the cost to offset investment against.
It depends on you understanding your app and how things need to be structured. We have what essentially is a video CMS, so we have two parts: a management UI that end-users use and a backend that actually delivers the video and collects metrics.
They are essentially two products, and are designed that way; if the management UI barfed the backend would continue along forever.
You can combine management and delivery in one app, but that makes delivery more fragile and will be slower because presumably it has to invoke a lot of useless stuff just to deliver bytes. I remember working with a spring app that essentially built and destroyed the whole spring runtime just to serve a request, which was an unbelievably dumb thing to do. Spring became the bottleneck, and for most requests there was actually no work done; 99% of the time was in spring doing spring things.
So really, once you separate the delivery and management it becomes easier to figure out the minimum amount of stuff you need. Redis, because you need to cache a bunch of metadata and handle lots of connections. Mysql, because you need a persistent store. Lambda, as a thin layer between everything. And a CDN, because you don't want to serve stuff out of AWS if you can help it. SQS for what essentially becomes job control. And for metric collection we use fastly with synthetic logging.
To be fair, our AWS cost was low but our CDN cost is like $1800/mo for some number of PB/mo (5? 10? I forget).
In the old days this would require at least (2 DB + 2 App server + 2 NAS) * 2 locations = 8 boxes. If we were going to do the networking ourselves we'd add 4 f5s. Ideally we'd have the app server, redis, and the various lambdas on different boxes, so 2 redis + 2 runners = 8 more servers. If we didn't use f5s we'd have 2 reverse proxies as the front end at each location. Each box would have 2 PSUs, at least a raid 1, dual NICs, and ECC. I think the lowest end Dell boxes with those features are like $5k each? Today I'd probably just stuff some 1TB SSDs in them and mirror them instead of going SAS. The NAS would be hard to spec because you have to figure out how much storage you need and they can be a pain to reconfigure. You don't want to spend too much up front, but you also don't want to have downtime while you add some more drive space.
Having built this out, it's not as easy as you'd think. I've been lucky enough to have built this sort of thing a few times. It's fun to do, but maintaining it can be a PITA. If you don't believe in documentation your deployment will fail miserably because you did something out of order.
They are essentially two products, and are designed that way; if the management UI barfed the backend would continue along forever.
You can combine management and delivery in one app, but that makes delivery more fragile and will be slower because presumably it has to invoke a lot of useless stuff just to deliver bytes. I remember working with a spring app that essentially built and destroyed the whole spring runtime just to serve a request, which was an unbelievably dumb thing to do. Spring became the bottleneck, and for most requests there was actually no work done; 99% of the time was in spring doing spring things.
So really, once you separate the delivery and management it becomes easier to figure out the minimum amount of stuff you need. Redis, because you need to cache a bunch of metadata and handle lots of connections. Mysql, because you need a persistent store. Lambda, as a thin layer between everything. And a CDN, because you don't want to serve stuff out of AWS if you can help it. SQS for what essentially becomes job control. And for metric collection we use fastly with synthetic logging.
To be fair, our AWS cost was low but our CDN cost is like $1800/mo for some number of PB/mo (5? 10? I forget).
In the old days this would require at least (2 DB + 2 App server + 2 NAS) * 2 locations = 8 boxes. If we were going to do the networking ourselves we'd add 4 f5s. Ideally we'd have the app server, redis, and the various lambdas on different boxes, so 2 redis + 2 runners = 8 more servers. If we didn't use f5s we'd have 2 reverse proxies as the front end at each location. Each box would have 2 PSUs, at least a raid 1, dual NICs, and ECC. I think the lowest end Dell boxes with those features are like $5k each? Today I'd probably just stuff some 1TB SSDs in them and mirror them instead of going SAS. The NAS would be hard to spec because you have to figure out how much storage you need and they can be a pain to reconfigure. You don't want to spend too much up front, but you also don't want to have downtime while you add some more drive space.
Having built this out, it's not as easy as you'd think. I've been lucky enough to have built this sort of thing a few times. It's fun to do, but maintaining it can be a PITA. If you don't believe in documentation your deployment will fail miserably because you did something out of order.
A bit of an alternative take on this, but I talk to a lot of folks at small start-ups (in Toronto, if that matters), but it seems like most people actually get this right and understand not to bring in complexity until later. Things like microservices seems like they are mostly understood as a tool that's not really meant to solve a real scalibility problem and is massive liability early on.
The exceptions are usually just inexperienced people at the helm. My feeling is, hire someone with adequate experience and this is likely not an issue.
I do think architecture astronauts tend to talk a lot more about their houses of cards, which makes it seem like these set ups are more popular than they are.
The exceptions are usually just inexperienced people at the helm. My feeling is, hire someone with adequate experience and this is likely not an issue.
I do think architecture astronauts tend to talk a lot more about their houses of cards, which makes it seem like these set ups are more popular than they are.
When I'm working with new developers I always have to convince them to simplify their setup. Why are we on autoscaled, pay by the query infra when we are serving a few people. Then they complain how expensive it is. I had someone tell me that their costs were $1500/mon when they were in demo stages. I asked them why they aren't hosting on a single small server for $20. And they responded that it didn't matter because they were using free credits.
Except that those free credits will go away and you'll find yourself not wanting to do all the work to move it over when it would've been easier to do so when you just had that first monolith server up.
I think free credits and hyped up technology is to blame. So, basically a gamed onboarding process that gets people to over-engineer and spend more.
Except that those free credits will go away and you'll find yourself not wanting to do all the work to move it over when it would've been easier to do so when you just had that first monolith server up.
I think free credits and hyped up technology is to blame. So, basically a gamed onboarding process that gets people to over-engineer and spend more.
I'm not totally following the cost analysis from some of these comments. I agree that there's no reason to make your architecture overly complicated, especially if it will cost dev time. I'm not saying this as someone that does cloud very much. We normally do a single server. More as an outsider looking in.
If you load balance 4 smaller ec2 instances vs a larger one, it's significantly cheaper. There is overhead when you run the same app on 4 machines, but not as much as people think. An idle system uses 300-600mb of ram, leaving roughly 3.5gb vram usable, per machine.
4 t3.medium instances are about $119/mo, vs. an equivalent t3.2xlarge which is $238/mo.
Also people will often say Digital Ocean is cheaper, but they compare it to non-equivalent AWS services. A Digital Ocean VPS compared to an equivalent EC2 is about the same.
So I understand when people argue about complicating architecture or optimizing prematurely, but I don't think the math on actual server cost really checks out. I would prefer to do dedicated hosting, but Hetzner doesn't have that in the U.S.
If you load balance 4 smaller ec2 instances vs a larger one, it's significantly cheaper. There is overhead when you run the same app on 4 machines, but not as much as people think. An idle system uses 300-600mb of ram, leaving roughly 3.5gb vram usable, per machine.
4 t3.medium instances are about $119/mo, vs. an equivalent t3.2xlarge which is $238/mo.
Also people will often say Digital Ocean is cheaper, but they compare it to non-equivalent AWS services. A Digital Ocean VPS compared to an equivalent EC2 is about the same.
So I understand when people argue about complicating architecture or optimizing prematurely, but I don't think the math on actual server cost really checks out. I would prefer to do dedicated hosting, but Hetzner doesn't have that in the U.S.
I have read and watched these articles and videos where people seem to have a problem with Microservice, Kubernetes, cloud providers, or anything that's not a PHP server sitting behind an nginx running on a $5 VPS. I have also seen the front-end analogy of these types of posts, where anything that is not written using HTML, CSS, and jQuery is unnecessary bloat. I will soon write a blog, which I think will cover more points and nuances of both sides. For now, here are some of my scattered thoughts.
- If deploying your MVP to EKS is overengineering, then signing a year-long lease for bare metal is hubris. Both think one day they will need it, but only one of them can undo that decision.
- Don't compare your JBOD to a multi-region replicated, CDN-enabled object store that can shrug off a DDoS attack. One protects you from those egress fees, and the other protects you from a disaster. They are not comparable.
- A year from now, the startup you work for may not exist. Being able to write that you have experience with that trendy technology on your resume sure sounds nice. Given the layoffs we are seeing right now, putting our interest above the company's may be a good idea.
- Yes, everyone knows modern CPUs are very fast, and paying $300/mo for an 8-core machine feels like a ripoff, but unless you are business of renting GPUs and selling tokens. Compute was never your cost center; it was always humans. For some companies, not being able to meet your SLA due to talent attrition is scarier than the cloud bill.
I know these are one-sided arguments, and I said I would cover both sides with more nuance. I need some time to think through all the arguments, especially on the frontend side. I will soon write a blog.
- If deploying your MVP to EKS is overengineering, then signing a year-long lease for bare metal is hubris. Both think one day they will need it, but only one of them can undo that decision.
- Don't compare your JBOD to a multi-region replicated, CDN-enabled object store that can shrug off a DDoS attack. One protects you from those egress fees, and the other protects you from a disaster. They are not comparable.
- A year from now, the startup you work for may not exist. Being able to write that you have experience with that trendy technology on your resume sure sounds nice. Given the layoffs we are seeing right now, putting our interest above the company's may be a good idea.
- Yes, everyone knows modern CPUs are very fast, and paying $300/mo for an 8-core machine feels like a ripoff, but unless you are business of renting GPUs and selling tokens. Compute was never your cost center; it was always humans. For some companies, not being able to meet your SLA due to talent attrition is scarier than the cloud bill.
I know these are one-sided arguments, and I said I would cover both sides with more nuance. I need some time to think through all the arguments, especially on the frontend side. I will soon write a blog.
I thought capitalism was about adding value, not conflict of interest
My friend is the first dev hire at a startup where they prematurely overengineered for scalability. The technical founders had recently exited a previous startup and their rationale was that it makes a future acquisition easier, since a potential acquirer will weigh scalability in their evaluation of the code (and maybe even conflate it with quality). In fact it was a regret from their first startup that they hadn't baked in scalability earlier. I remain skeptical of the decision, but curious if there's any truth to the fact that acquirers weigh scalability in their scorecard?
Sure, if the acquirer thinks the product is going to sell a lot.
A relatively common plan (it doesn't always work) for large enterprise software companies is to buy a product and then use their very large sales force to sell it into all their existing customers. If thats the plan, you have to make sure the product will work with all the increased usage.
I'd still suggest it's far better to optimize for building the right product - the "is this going to scale" problem is one of the nicest problems you can face.
A relatively common plan (it doesn't always work) for large enterprise software companies is to buy a product and then use their very large sales force to sell it into all their existing customers. If thats the plan, you have to make sure the product will work with all the increased usage.
I'd still suggest it's far better to optimize for building the right product - the "is this going to scale" problem is one of the nicest problems you can face.
You can get an awful lot done with Go, MySQL or MariaDB, FCGI, Apache or Ngnix.
You start with one server on shared hosting for a few dollars a month. Then scale up to a dedicated server, if the load appears. Then scale up to multiple servers with a load balancer. Replicate the database.
On the load side, if you're accumulating "statistics" about user behavior, do you really need them for every user? Maybe only one user in a hundred. Or a thousand.
When you exceed the limits of that, you're a big company and can afford AWS.
A few years ago, we had those guys with the liquid meal (not Jucero, the Soylent guys) boasting about their "infrastructure". Not for making the product, for their web site. From their financials, you could calculate that they were doing about four sales a minute. Their "infrastructure" could run on an Raspberry Pi.
On the load side, if you're accumulating "statistics" about user behavior, do you really need them for every user? Maybe only one user in a hundred. Or a thousand.
When you exceed the limits of that, you're a big company and can afford AWS.
A few years ago, we had those guys with the liquid meal (not Jucero, the Soylent guys) boasting about their "infrastructure". Not for making the product, for their web site. From their financials, you could calculate that they were doing about four sales a minute. Their "infrastructure" could run on an Raspberry Pi.
The problem with a VPS/baremetal solution is always high availability. Once you introduce HA, you may as well go into the cloud since HA'ing stateful stuff using VPS is a major pain.
Also, you can get a lot done with a serverless (FaaS/PaaS) solution and a simple DB like DynamoDB.
Also, you can get a lot done with a serverless (FaaS/PaaS) solution and a simple DB like DynamoDB.
I don't disagree with this, but I think people sometimes miss the fact that even cheap commodity hardware is way more reliable than most software. Your site may go down more often due to bugs in your HA configuration than it would have gone down because of your VPS dying once a year or so. If you are not a huge operation, the sweet spot for HA is probably something very simple (e.g. two servers behind a load balancer, or something like that).
Two servers, load balancer, replicated Postgres database on the back end, offsite backups?
Everything is scalable because for an enterprising executive/manager/employee the most important thing they are trying to solve is scaling their empire. Pay is based on how many reports you have and how big is your budget. Instead of 2 x 20K beefy servers, you get to spend tens or hundreds of million on cloud spend, then dozens of $500K/year SRE/DevOps/Cloud 'experts' running multi-cloud hybrid running kubernetes thingies. There will be SO many internal automation tools and big bonuses for developing those tools (wins all around), yet more and more hiring!
Once other executives understand that you can scale your team massively, they will hire you to scale even greater heights! More #winning!
There is a human element of everything and perverse incentives. Understanding these explains most things that seem baffling at first.
Once other executives understand that you can scale your team massively, they will hire you to scale even greater heights! More #winning!
There is a human element of everything and perverse incentives. Understanding these explains most things that seem baffling at first.
Exactly. Once you become a people manager, the only metric that matters in your career is "number of people below you".
> The first problem every startup should solve is “how do we have enough money to not go bust in two months”, but that’s a hard problem, whereas scalability is trivially solvable by reading a few engineering blogs [...] Do you know what the difference between Google and your startup is? It’s definitely not scalability, you’ve solved that problem. It’s that Google has billions upon billions with which to pay for that scalability, which is really good because scalability is expensive.
Too true. Now that I've stepped into an "engineering leadership" role and spend as much time looking at finances as I do at code, I've formed the opinion that in 99.999% of cases, engineering problems are really business problems. If you could throw infinite time and money at the technical challenges, they'd no longer be challenging. But businesses, especially startups, don't have infinite (or even "some") money and time, so the challenge is doing the best engineering work you can, given time and budget constraints.
> The downsides [of the monolith approach]
I like the article's suggestion of using explicitly defined API boundaries between modules, and that's a good approach for a monolith. However one massive downside that cannot be ignored -- by having a single monolith you now have an implicit dependency on the same runtime working on all parts of your code. What I mean by this is, all your code is going to share the same Python version and same libraries (particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies). This means that if you're working on Module A, and you realize you need a new feature from Pandas 2.x, but the rest of the code is on Pandas 1.x... well, you can't upgrade unless you go and fix Modules B, C, D ... Z to work with Pandas 2.
This won't be an issue at the start, but it's worth pointing out. Being forced to upgrade a core library or language runtime and finding out it's a multi-month disruptive project can be brutal.
Too true. Now that I've stepped into an "engineering leadership" role and spend as much time looking at finances as I do at code, I've formed the opinion that in 99.999% of cases, engineering problems are really business problems. If you could throw infinite time and money at the technical challenges, they'd no longer be challenging. But businesses, especially startups, don't have infinite (or even "some") money and time, so the challenge is doing the best engineering work you can, given time and budget constraints.
> The downsides [of the monolith approach]
I like the article's suggestion of using explicitly defined API boundaries between modules, and that's a good approach for a monolith. However one massive downside that cannot be ignored -- by having a single monolith you now have an implicit dependency on the same runtime working on all parts of your code. What I mean by this is, all your code is going to share the same Python version and same libraries (particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies). This means that if you're working on Module A, and you realize you need a new feature from Pandas 2.x, but the rest of the code is on Pandas 1.x... well, you can't upgrade unless you go and fix Modules B, C, D ... Z to work with Pandas 2.
This won't be an issue at the start, but it's worth pointing out. Being forced to upgrade a core library or language runtime and finding out it's a multi-month disruptive project can be brutal.
> by having a single monolith you now have an implicit dependency on the same runtime working on all parts of your code
I think many people would not consider what I am going to suggest a monolith, but docker compose all hosted in the same server acts a lot like a monolith.
> particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies
You can use virtual environments each with its python version and packages.
I think many people would not consider what I am going to suggest a monolith, but docker compose all hosted in the same server acts a lot like a monolith.
> particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies
You can use virtual environments each with its python version and packages.
Early in my career I was given the opportunity (in between consulting jobs) to make an MVP for a revamp of our internal employee management system.
I seized the opportunity to deploy my own Kubernetes cluster, and create a sidecar to help with user authentication (because of course we'd need a common way to do this for the multi-language suite of microservices we'd be building). I used up pretty much the entirety of my time designing and architecting how this colossus was going to work, and by the end I realized how foolish the whole endeavor was.
That was really an instructive failure - at my next job, I got everyone behind turning our team's microservices back into a modular monolith, and it worked very well.
I seized the opportunity to deploy my own Kubernetes cluster, and create a sidecar to help with user authentication (because of course we'd need a common way to do this for the multi-language suite of microservices we'd be building). I used up pretty much the entirety of my time designing and architecting how this colossus was going to work, and by the end I realized how foolish the whole endeavor was.
That was really an instructive failure - at my next job, I got everyone behind turning our team's microservices back into a modular monolith, and it worked very well.
Isn't it simple as the following?
Break your code into modules/components that have a defined interface between them. That interface only passes data - not code with behaviour - and signal the method calls may fail to complete ( ie throw exceptions ).
ie the interface could be a network call in the future.
Allow easy swapping of interface implementations by passing them into constructors/ using factories or dependency injection frameworks if you must.
That's it - you can then start with everything in-process and the rapid development that allows, but if you need to you can add splitting into networked microservices - any complexity that arises from the network aspect is hidden behind the proxy, with the ultimate escape hatch of the exception.
Have I missed something?
Break your code into modules/components that have a defined interface between them. That interface only passes data - not code with behaviour - and signal the method calls may fail to complete ( ie throw exceptions ).
ie the interface could be a network call in the future.
Allow easy swapping of interface implementations by passing them into constructors/ using factories or dependency injection frameworks if you must.
That's it - you can then start with everything in-process and the rapid development that allows, but if you need to you can add splitting into networked microservices - any complexity that arises from the network aspect is hidden behind the proxy, with the ultimate escape hatch of the exception.
Have I missed something?
You're not missing something, but you're assuming that it's easy to know ahead of time where the module boundaries should be and what the interfaces should look like. This is very far from easy, if possible at all (eg google "abstraction boundaries are optimization boundaries").
Also, most of these interfaces you'll likely never need. It's a cost of initial development, and the indirection is a cost on maintainability of your code. It's probably (although not certainly) cheaper to refactor to introduce interfaces as needed, rather than always anticipate a need that might never come.
Also, most of these interfaces you'll likely never need. It's a cost of initial development, and the indirection is a cost on maintainability of your code. It's probably (although not certainly) cheaper to refactor to introduce interfaces as needed, rather than always anticipate a need that might never come.
I think it is more intuitive if we think of side-effects. By specifying the interface you are explicitly defining inputs and outputs. If you want to add this later, it can be very difficult to make sure you can find all the side-effects. The whole point of the interface is to explicitly limit those side-effects and extra inputs outputs to happen, so it makes sense to define in advance.
You're not missing much, but I don't understand why you're just basically repeating what the article already says. Except the article also says to use a monorepo.
No, I'm saying you don't need to use a monorepo! The repo discussion is a bit orthogonal, and up to you to decide whether you want a single repo or multiple repos with modules/libraries that get deployed together.
I think I've added a couple of elements to make it possible to scale your auth service if you need to. Easily swappable implementations and making sure the interfaces advertise that calls may simply fail.
Even so it's still very simple.
To scale your auth service you just write a proxy to a remote implementation and pass that in - any load balancing etc is hidden behind that same interface and none of the rest of the code cares.
Even so it's still very simple.
To scale your auth service you just write a proxy to a remote implementation and pass that in - any load balancing etc is hidden behind that same interface and none of the rest of the code cares.
Good point! Sorry if I was being ungenerous.
I like the idea of the remote implementation being proxied -- not sure I've come across that pattern before.
I like the idea of the remote implementation being proxied -- not sure I've come across that pattern before.
the swap from interface to network call is still non-trivial.
you get to have new problems that are qualitatively different from before like timeouts, which can break the adsumptions in the rest of your code about say, whether state was updated or not, and in what order. you also then get to deal with thundering herds and circuit breakers and so on.
you get to have new problems that are qualitatively different from before like timeouts, which can break the adsumptions in the rest of your code about say, whether state was updated or not, and in what order. you also then get to deal with thundering herds and circuit breakers and so on.
Sure is more complex - but as I said key thing is to define those interfaces in a way that can be networked - you are just passing data not behaviour and the calls could fail to complete.
In terms of timing the call is synchronous and either succeeds or fails - the details like timeouts/ asynch underhood etc are hidden by the proxy - in the end the call succeeds or fails and if you surface that as a synchronous call you hide the underlying complexity from the caller.
A bit like opening a file and writing to it - most platform apis throw exceptions - and your code has to deal with it.
In terms of timing the call is synchronous and either succeeds or fails - the details like timeouts/ asynch underhood etc are hidden by the proxy - in the end the call succeeds or fails and if you surface that as a synchronous call you hide the underlying complexity from the caller.
A bit like opening a file and writing to it - most platform apis throw exceptions - and your code has to deal with it.
Yes, you are missing the cost of complexity and network calls. You are describing a distributed monolith. It does not help.
Not sure I understand. What is a distributed monolith?
I'm not suggesting that the distributed bit is still coupled behind the scenes ( ie via a data backend that requires distributed transactions ) - the interaction is through the interface.
In the end you are always going to have code calling code - the key point is to assume these key calls are simply data passing, not behaviour passing, and that they can fail.
What else is need to make something network friendly? ( I'm suggesting that things like retries, load-balancing etc can be hidden as a detail in the network implementation - all you need to surface is succeed or fail ).
I'm not suggesting that the distributed bit is still coupled behind the scenes ( ie via a data backend that requires distributed transactions ) - the interaction is through the interface.
In the end you are always going to have code calling code - the key point is to assume these key calls are simply data passing, not behaviour passing, and that they can fail.
What else is need to make something network friendly? ( I'm suggesting that things like retries, load-balancing etc can be hidden as a detail in the network implementation - all you need to surface is succeed or fail ).
Scalability is not purely technical. It's also organizational. For all its drawbacks, the microservices architecture is easier to scale from an organizational perspective.
Only when the service boundaries and interfaces are built with this in mind.
A service that is isolated enough it could be another company? Sure, this scales. But do company hierarchies and organization practices help this happen? I haven't seen it outside of places like Amazon where there was a mandate for it to be that way.
What companies end up with in practice are services so tightly coupled with the rest of the company that they requiring a mishmash of API requests in both directions and endless coordination. Aka a distributed monolith. All the problems with zero the advantages.
A service that is isolated enough it could be another company? Sure, this scales. But do company hierarchies and organization practices help this happen? I haven't seen it outside of places like Amazon where there was a mandate for it to be that way.
What companies end up with in practice are services so tightly coupled with the rest of the company that they requiring a mishmash of API requests in both directions and endless coordination. Aka a distributed monolith. All the problems with zero the advantages.
My experience is it adds a multiplier 10x to the work that needs to be done, which is then a good fit for the 9x extra developers upper management decided to hire.
Sure, it keeps everyone busy with work they feel is meaningful -- but it really is work that was not really needed in the first place with a monolith.
Much simpler to try to stay fewer developers as long as possible.
Also in my experience it is impossible to get people along with organizing the company along software architecture division lines, thus creating a lot of political problems.
Sure, it keeps everyone busy with work they feel is meaningful -- but it really is work that was not really needed in the first place with a monolith.
Much simpler to try to stay fewer developers as long as possible.
Also in my experience it is impossible to get people along with organizing the company along software architecture division lines, thus creating a lot of political problems.
In lot of contexts scaling down is far more important than scaling up. In that sense scalability is cost-optimization; instead of provisioning fixed capacity that is enough for (predicted) peak loads, you can scale based on actual demand and save money or have higher utilization.
I think it was around 2015 when everything was basically AWS and Kubernetes
The turning point might have been Heroku? Prior to Heroku, I think people just assumed you deploy to a VPS. Heroku taught people to stop thinking about the production environment so much.
I think people were so inspired by it and wanted to mimic it for other languages. It got more people curios about AWS.
Ironically, while the point of Heroku was to make deployment easy and done with a single command, the modern deployment story on cloud infrastructure is so complicated most teams need to hold a one hour meeting with several developers "hands on deck" and going through a very manual process.
So it might seem counter intuitive to suggest that the trend was started by Heroku, because the result is the exact opposite of the inspiration.
The turning point might have been Heroku? Prior to Heroku, I think people just assumed you deploy to a VPS. Heroku taught people to stop thinking about the production environment so much.
I think people were so inspired by it and wanted to mimic it for other languages. It got more people curios about AWS.
Ironically, while the point of Heroku was to make deployment easy and done with a single command, the modern deployment story on cloud infrastructure is so complicated most teams need to hold a one hour meeting with several developers "hands on deck" and going through a very manual process.
So it might seem counter intuitive to suggest that the trend was started by Heroku, because the result is the exact opposite of the inspiration.
[deleted]
I think part of the problem is (some) programmers being unable to draw clear encapsulation boundaries when writing a monolith. I'm not even referring to imposing a discipline for a whole team, but the ability to design a clean internal API and stick to it oneself.
I've found that building my side projects to be "scalable" is a practical side effect of choosing the most cost-effective hosting.
When a project has little to no traffic, the on-demand pricing of serverless is unbeatable. A static site on S3 or a backend on Lambda with DynamoDB will cost nothing under the AWS free tier. A dedicated server, even a cheap one, is an immediate and fixed $8-10/month liability.
The cost to run a monolith on a VPS only becomes competitive once you have enough users to burn through the very generous free tiers, which for many side projects is a long way off. The primary driver here is minimizing cost and operational overhead from day one.
When a project has little to no traffic, the on-demand pricing of serverless is unbeatable. A static site on S3 or a backend on Lambda with DynamoDB will cost nothing under the AWS free tier. A dedicated server, even a cheap one, is an immediate and fixed $8-10/month liability.
The cost to run a monolith on a VPS only becomes competitive once you have enough users to burn through the very generous free tiers, which for many side projects is a long way off. The primary driver here is minimizing cost and operational overhead from day one.
> A dedicated server, even a cheap one, is an immediate and fixed $8-10/month liability.
Personally, I am more worried about the infinitely-scalable service potentially (liability) sending a huge bill after the fact. This "liability" of $8-10 is predictable, like a Netflix subscription.
Personally, I am more worried about the infinitely-scalable service potentially (liability) sending a huge bill after the fact. This "liability" of $8-10 is predictable, like a Netflix subscription.
I agree, of course. Choose an architecture for the pragmatic reality, not some fantastical non-reality.
However, I appreciate the craft. Some of these unnecessary optimizations (rather, “introduced complexities”) are vestigial accoutrements that come alongside generally good software design. Not all, but some. So I tolerate a fair amount of fanciness in myself and others when it coincides with solid intent and healthy output.
That said, we should absolutely not tolerate the presence of appurtenances of complexity at the architectural layer – that is a place reserved for pure 100% pragmatism.
However, I appreciate the craft. Some of these unnecessary optimizations (rather, “introduced complexities”) are vestigial accoutrements that come alongside generally good software design. Not all, but some. So I tolerate a fair amount of fanciness in myself and others when it coincides with solid intent and healthy output.
That said, we should absolutely not tolerate the presence of appurtenances of complexity at the architectural layer – that is a place reserved for pure 100% pragmatism.
If you’ve ever been in a situation where you do suddenly face scale and have to rip apart a legacy monolith that was built without scale in mind, you’ll chuckle at this article. It’s extremely painful.
I'm not saying "you're never going to scale", I'm saying "you aren't going to scale right now". There's a happy medium between overengineering everything and making everything so shoddily that you can never grow.
[deleted]
Legitimately asking, how? The only bottleneck should be the DB, and if you can saturate a 128-core DB, I want to see your queries and working set size. Not saying it can’t happen, but it’s rare that someone has actually maxed out MySQL or Postgres without there being some serious schema and query flaws, or just poor / absent tuning.
You’re thinking purely in terms of app performance. have you ever seen a terrible db schema? Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+.
I’ve seen monoliths because of their sheer size and how much crap and debt is packed into them, build and deploy processes taking several hours if not an entire day for some fix that could be ci/cd’d in seconds if it wasn’t such a ball of mud. Then, what tends to happen, is the infrastructure around it tends to compensate heavily for it, which turns into its own ball of mud. Nothing wrong with properly scaled monoliths but it’s a bit naive, in my personal experience, to just scoff at scale when your business succeeding relies on scale at some point. Don’t prematurely optimize, but don’t be oblivious to future scenarios, because they can happen quicker than you think
I’ve seen monoliths because of their sheer size and how much crap and debt is packed into them, build and deploy processes taking several hours if not an entire day for some fix that could be ci/cd’d in seconds if it wasn’t such a ball of mud. Then, what tends to happen, is the infrastructure around it tends to compensate heavily for it, which turns into its own ball of mud. Nothing wrong with properly scaled monoliths but it’s a bit naive, in my personal experience, to just scoff at scale when your business succeeding relies on scale at some point. Don’t prematurely optimize, but don’t be oblivious to future scenarios, because they can happen quicker than you think
That was the reality of the fintech I worked at.
The schema wasn't really a problem, but the sheer amount of queries per request. Often a user opening a page or clicking a button would cause 100-200 database queries, including updates. This would prevent strategies such as "just replicating the data somewhere". It was so badly architected that every morning the app would stop responding due to users doing their morning routine operations. And they only had around 300 employees.
And this was just an internal app, the B2C part was already isolated because we couldn't afford to be offline.
The solution I started working on was doing similar to the strangler fig pattern and replacing parts of the API with new code that talked directly to the ORM. Naturally this didn't made the people who wrote the legacy code happy, but at least the outages stopped.
The schema wasn't really a problem, but the sheer amount of queries per request. Often a user opening a page or clicking a button would cause 100-200 database queries, including updates. This would prevent strategies such as "just replicating the data somewhere". It was so badly architected that every morning the app would stop responding due to users doing their morning routine operations. And they only had around 300 employees.
And this was just an internal app, the B2C part was already isolated because we couldn't afford to be offline.
The solution I started working on was doing similar to the strangler fig pattern and replacing parts of the API with new code that talked directly to the ORM. Naturally this didn't made the people who wrote the legacy code happy, but at least the outages stopped.
sounds so similar to a fintech situation I’ve been in once I swear I was gonna say we worked at the same place, but the size of company is wrong. I’ve now seen pretty similar things since, enough to say it’s probably everywhere
I know some people working at very popular EU fintech-banks and the description I get is that it’s just a shitshow engineering wise.
Good engineering is unfortunately at odds with modern development practices.
Good engineering is unfortunately at odds with modern development practices.
> have you ever seen a terrible db schema?
I am a DBRE, so yes, unfortunately most days I see terrible schemata.
> Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+.
IME, the “let’s move fast” mindset causes further problems, because it’s rare that a dev has any inkling about proper data modeling, let alone RDBMS internals. What I usually see are heavily denormalized tables, UUIDs everywhere, and JSON taking the place of good modeling practices. Then they’re surprised when I tell them the issue can’t be fixed with yet another index, or a query rewrite. Turns out when you have the largest instance the cloud provider has, and your working set still doesn’t fit into memory, you’re gonna have a bad time.
I am a DBRE, so yes, unfortunately most days I see terrible schemata.
> Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+.
IME, the “let’s move fast” mindset causes further problems, because it’s rare that a dev has any inkling about proper data modeling, let alone RDBMS internals. What I usually see are heavily denormalized tables, UUIDs everywhere, and JSON taking the place of good modeling practices. Then they’re surprised when I tell them the issue can’t be fixed with yet another index, or a query rewrite. Turns out when you have the largest instance the cloud provider has, and your working set still doesn’t fit into memory, you’re gonna have a bad time.
Did you literally just say your database does not scale because the largest instance isnt enough and your response was to "fix the schema"? Haha, this is the classic example of why things must scale and you just made the parent comments point.
For e-commerce, sure. But for telecom or IoT, it doesn’t take a large company to easily overrun the limits if what Postgres can do.
Man is 100% right; the insanity we put ourselves through for all kinds of hypotheticals is beyond mental
I love this blog post so much. <3
Stop solving problems you don't have. You have no users, you don't need to support a million of them. You make no money, you don't need to burn through all your cloud credits so fast. Just chill, start with a couple Linodes, Droplets, or Lightsail instances and see how far you can get with a simple app, api, load balancer, and a database. You'd be shocked how far that stretches when you're talking about servicing real paying customers.
Stop solving problems you don't have. You have no users, you don't need to support a million of them. You make no money, you don't need to burn through all your cloud credits so fast. Just chill, start with a couple Linodes, Droplets, or Lightsail instances and see how far you can get with a simple app, api, load balancer, and a database. You'd be shocked how far that stretches when you're talking about servicing real paying customers.
A VPS, caddy, sqlite and livestream is really all you need in my experience. Help's if your language has decent support for real and green threads (java/go/clojure).
I think more engineers would benefit from "you can just build stuff" thinking. Like you don't have to use all the complicated whizz-bang tech that everyone else is using.
You can build a boring backend on Linux VMs without containers using open-source software - it's simpler or at least a different level of complexity compared to the big clouds and orchestration systems like k8s, and honestly, it's just more fun to work on - I almost never write yaml - it's a joy.
I wrote my own deployment system using this idea - machines, roles, software and services that map to those roles, idempotent operations, and a constantly-connected async rpc system to orchestrate it all. Written from scratch in a language I like with a config language I like. My deploys are often < 10s (if I'm not waiting on webpack to build the UI) and all connect up to a chatops channel in Slack. I understand it because I wrote it all. Will it scale to infinity? Definitely not, but it's good enough for my uses.
So, I guess - just build stuff using simple primitives. Write simple software - modules and functions and a lot of stateless code. Use postgres for persistence - it's really that good. Use nginx and dns load balancing - tried and true simple architecture.
You can build a boring backend on Linux VMs without containers using open-source software - it's simpler or at least a different level of complexity compared to the big clouds and orchestration systems like k8s, and honestly, it's just more fun to work on - I almost never write yaml - it's a joy.
I wrote my own deployment system using this idea - machines, roles, software and services that map to those roles, idempotent operations, and a constantly-connected async rpc system to orchestrate it all. Written from scratch in a language I like with a config language I like. My deploys are often < 10s (if I'm not waiting on webpack to build the UI) and all connect up to a chatops channel in Slack. I understand it because I wrote it all. Will it scale to infinity? Definitely not, but it's good enough for my uses.
So, I guess - just build stuff using simple primitives. Write simple software - modules and functions and a lot of stateless code. Use postgres for persistence - it's really that good. Use nginx and dns load balancing - tried and true simple architecture.
Honestly, in my experience, the only good reason to have microservices in a "software solution" is to be able to match 1 service -> 1 mantainer/team and have a big (read "nested", with multiple level of middle-managers) group of teams, each that may have different goals.
In this way it's very easy to "map" a manager/team to a "place" in the solution map, with very explicit and documented interactions between them
Plot twist: It's not actually scalable because no amount of tools and buzzwords can compensate for the lack of experience in proper architecture for scaling.
Author talks about enforcing internal code boundaries within their monolith.
There is a very simple solution. Ask your AI to write you a little python script that checks which modules are imported by which other ones, and validates against a set of rules.
Prompts go something like: “we should prevent code in web route handlers from directly importing the database, make it print an error saying only repository classes can use the database directly”. It will write code that visits the python AST or maybe that uses wild regexes. The result is the same: your prompt, codified, turned into a repeatable deterministic tool you can use any time for zero tokens and that runs in under a second.
Put the script in pre-commit and husky. Keep adding rules over time. If it gets too slow, ask your AI to optimise it for you.
These tools codify the tacit knowledge needed to work in your codebase. Which is a boon for onboarding (new colleagues get blocked by it during development instead of code review.) And because it runs just before everyone commits, you have one less thing to think about when reviewing.
There is a very simple solution. Ask your AI to write you a little python script that checks which modules are imported by which other ones, and validates against a set of rules.
Prompts go something like: “we should prevent code in web route handlers from directly importing the database, make it print an error saying only repository classes can use the database directly”. It will write code that visits the python AST or maybe that uses wild regexes. The result is the same: your prompt, codified, turned into a repeatable deterministic tool you can use any time for zero tokens and that runs in under a second.
Put the script in pre-commit and husky. Keep adding rules over time. If it gets too slow, ask your AI to optimise it for you.
These tools codify the tacit knowledge needed to work in your codebase. Which is a boon for onboarding (new colleagues get blocked by it during development instead of code review.) And because it runs just before everyone commits, you have one less thing to think about when reviewing.
In 2015, a single server with 40 cores and 128 GB of RAM was able to handle 2 million WebSocket connections running an Elixir program[0].
One of my hot takes is that a gaming PC has fast enough hardware to serve thousands of clients with a static Rust binary and SQLite. Pair with Litestream, and you have easy-to-test, continuous backups. It's nice being able to test backups by just running `litestream restore` and then running a single binary on my development machine. Addtionally, when the backend is a single static binary, you gain the opportunity to test the entire system in CI without maintaining an ad-hoc cloud environment or heirarchy of mock services.
The points of contention, for me personally, would be managing deployments and observability.
Of course, at my workplace, I wouldn't dare to suggest this kind of architecture, but as others have mentioned, a single machine can go a long way, and I doubt most my projects will ever need to scale beyond 40 cores and 128 GB of RAM.
[0] https://x.com/chris_mccord/status/659430661942550528
One of my hot takes is that a gaming PC has fast enough hardware to serve thousands of clients with a static Rust binary and SQLite. Pair with Litestream, and you have easy-to-test, continuous backups. It's nice being able to test backups by just running `litestream restore` and then running a single binary on my development machine. Addtionally, when the backend is a single static binary, you gain the opportunity to test the entire system in CI without maintaining an ad-hoc cloud environment or heirarchy of mock services.
The points of contention, for me personally, would be managing deployments and observability.
Of course, at my workplace, I wouldn't dare to suggest this kind of architecture, but as others have mentioned, a single machine can go a long way, and I doubt most my projects will ever need to scale beyond 40 cores and 128 GB of RAM.
[0] https://x.com/chris_mccord/status/659430661942550528
Can’t argue with that. Machines are pretty capable, and elixir is awesome as well
I had a client with a system just like this. EBS, S3, RDS, Cognito, the lot. It cost $00s per month under almost no load, and was a maintenance nightmare - which was the real problem, not the cost, as it stopped working altogether eventually. A bit of hacking later, it all fits on a single VM that costs ~$10/month to run and is far easier to build, deploy and maintain.
A problem not addressed by the article is customer expectations. If you work for a company that does contracting, you have to deal with customers who have enough knowledge to know all the buzz words but not enough knowledge to actually know what they're asking for. If you don't give them Kubernetes and micro services they don't want to pay you.
Working in modern architectures that can scale is pretty important for developers that want to have attractive resumes. Given that your startup has a 9 out of 10 chance of failing you're going to need another job. If you want people to stay you have to give them the security of keeping up with at least some of the latest fashions.
I can relate - running a small hosting business. People come up with too complex solutions. They solve problems that they'd wish to have. For instance: HA setups are complex. If not done correctly, like in most cases, people don't gain the additional '9' from the SLA.
Scale articles are too focused on architecture. What about business problems that come with scale. At a certain scale rare events are common many cases cease to be fixable by some random process that involves humans you have to handle a lot more business scenarios with your code.
I agree with this post, but how do we convince decision makers to use simpler solutions? Here's what happened to me not long ago:
A person contacted me to build a tool that will have a single user. After some questioning, the person said they would likely have up to 3 users for this tool at some point, and no more.
So I suggested we use Python and a SQlite database. When I reached out after not having heard from them for 2 weeks, they said that they went with a more experienced programmer because the latter suggested MongoDB, a separate ReactJS front-end, and micro services with Golang. All deployed to AWS, of course.
In the end, I lost the gig because I wanted to save the client time and money.
A person contacted me to build a tool that will have a single user. After some questioning, the person said they would likely have up to 3 users for this tool at some point, and no more.
So I suggested we use Python and a SQlite database. When I reached out after not having heard from them for 2 weeks, they said that they went with a more experienced programmer because the latter suggested MongoDB, a separate ReactJS front-end, and micro services with Golang. All deployed to AWS, of course.
In the end, I lost the gig because I wanted to save the client time and money.
This piece is written with a pretty cliche dismissive tone that assumes that everything everyone else does is driven by cargo-culting if not outright ignorance. That people make these choices because they're just rushing to chase the latest trend.
They're just trying to be cool, you see.
Here's the thing, though: Almost every choice that leads to scalability also leads to reliability. These two patterns are effectively interchangeable. Having your infra costs be "$100 per month" (a claim that usually comes with a massive disclaimer, as an aside) but then falling over for a day because your DB server crashed is a really, really bad place to be.
They're just trying to be cool, you see.
Here's the thing, though: Almost every choice that leads to scalability also leads to reliability. These two patterns are effectively interchangeable. Having your infra costs be "$100 per month" (a claim that usually comes with a massive disclaimer, as an aside) but then falling over for a day because your DB server crashed is a really, really bad place to be.
> Almost every choice that leads to scalability also leads to reliability.
Empirically, that does not seem to be the case. Large scalable systems also go offline for hours at a time. There are so many more potential points of failure due to the complexity.
And even with a single regular server, it's very easy to keep a live replica backup of the database and point to that if the main one goes down. Which is a common practice. That's not scaling, just redundancy.
Empirically, that does not seem to be the case. Large scalable systems also go offline for hours at a time. There are so many more potential points of failure due to the complexity.
And even with a single regular server, it's very easy to keep a live replica backup of the database and point to that if the main one goes down. Which is a common practice. That's not scaling, just redundancy.
>Empirically, that does not seem to be the case.
Failures are astonishingly, vanishingly rare. Like it's amazing at this point how reliable almost every system is. There are a tiny number of failures at enormous scale operations (almost always due to network misconfigurations, FWIW), but in the grand scheme of things we've architected an outrageously reliable set of platforms.
>That's not scaling, just redundancy.
In practice it almost always is scaling. No one wants to pay for a whole n server just to apply shipped logs to. I mean, the whole premise of this article is that you should get the most out of your spend, so in that case much better is two hot servers. And once you have two hot...why not four, distributed across data centers. And so on.
Failures are astonishingly, vanishingly rare. Like it's amazing at this point how reliable almost every system is. There are a tiny number of failures at enormous scale operations (almost always due to network misconfigurations, FWIW), but in the grand scheme of things we've architected an outrageously reliable set of platforms.
>That's not scaling, just redundancy.
In practice it almost always is scaling. No one wants to pay for a whole n server just to apply shipped logs to. I mean, the whole premise of this article is that you should get the most out of your spend, so in that case much better is two hot servers. And once you have two hot...why not four, distributed across data centers. And so on.
> Failures are astonishingly, vanishingly rare
You and I must be using different sites and different clouds.
There's a reason isitdownrightnow.com exists. And why HN'ers are always complaining about service status pages being hosted on the same services.
By your logic, AWS and Azure should fail once in a millennium, yet they regularly bring down large chunks of the internet.
Literally last week: https://cyberpress.org/microsoft-azure-faces-global-outage-i...
You and I must be using different sites and different clouds.
There's a reason isitdownrightnow.com exists. And why HN'ers are always complaining about service status pages being hosted on the same services.
By your logic, AWS and Azure should fail once in a millennium, yet they regularly bring down large chunks of the internet.
Literally last week: https://cyberpress.org/microsoft-azure-faces-global-outage-i...
llm_nerd(1)
A distributed monolith - which is what nearly all places claiming to run microservices actually have - has N^m uptime.
Even if you do truly have a microservices architecture, you’ve also now introduced a great deal of complexity, and unless you have some extremely competent infra / SRE folk on staff, that’s going to bite you. I have seen this over and over and over again.
People make these choices because they don’t understand computing fundamentals, let alone distributed systems, but the Medium blogs and ChatGPT have assured them that they do.
Even if you do truly have a microservices architecture, you’ve also now introduced a great deal of complexity, and unless you have some extremely competent infra / SRE folk on staff, that’s going to bite you. I have seen this over and over and over again.
People make these choices because they don’t understand computing fundamentals, let alone distributed systems, but the Medium blogs and ChatGPT have assured them that they do.
This is the truth. I work with an application that has nearly 100 microservices and it seems like at any given point in time at least one is busted. Is it going to impact what you’re doing? Maybe. Maybe not.
But if it was just a monolith and had proper startup checks, when they roll out a new version and it fails, just kill it right there. Leave the old working version up.
Monoliths have their issues too. But doing microservices correctly is quite the job.
But if it was just a monolith and had proper startup checks, when they roll out a new version and it fails, just kill it right there. Leave the old working version up.
Monoliths have their issues too. But doing microservices correctly is quite the job.
> Almost every choice that leads to scalability also leads to reliability.
Yes, dealing with skew for every single change and hunting down bugs across network boundaries that could have been a function call is peak reliability.
Yes, dealing with skew for every single change and hunting down bugs across network boundaries that could have been a function call is peak reliability.
Yes, reliability comes from the same ground the scalability does, and yes people are mostly chasing the latest trend. One does not contradict the other.
>yes people are mostly chasing the latest trend
https://www.youtube.com/watch?v=b2F-DItXtZs
15 years ago people were making the same "chasing trends" complaints. In that case there absolutely were people cargo culting, but to still be whining about this a decade and a half later, when it's quite literally just absolutely basic best practices.
https://www.youtube.com/watch?v=b2F-DItXtZs
15 years ago people were making the same "chasing trends" complaints. In that case there absolutely were people cargo culting, but to still be whining about this a decade and a half later, when it's quite literally just absolutely basic best practices.
> Here's the thing, though: Almost every choice that leads to scalability also leads to reliability.
How is that supposed to happen. Without k8 involved somehow?
How is that supposed to happen. Without k8 involved somehow?
There is a lot of instruments, that don't need k8s to be scalable and reliable. Starting from stateless services and simple load balancers and ending with actor systems like in Erlang or Akka.
The only bit that I didn't agree with was the downsides. You can scale individual modules.
You can set your load balancer to make all calls to specific endpoints go to specific servers. These can scale independently.
If you have background workers (ActiveJobs/Oban for example), these can be on different queues, that you can scale. It's actually really easy to build out a mono-repo system allow for scale.
If you organise your workers into folders based on their purpose (reporting, exporting, ... ), and you're careful about feature flags, you can drastically reduce git-conflicts and CICD issues.
If you have background workers (ActiveJobs/Oban for example), these can be on different queues, that you can scale. It's actually really easy to build out a mono-repo system allow for scale.
If you organise your workers into folders based on their purpose (reporting, exporting, ... ), and you're careful about feature flags, you can drastically reduce git-conflicts and CICD issues.
Comes down to knowing when to stop. You don’t really want to DIY your own orchestrator etc. So better off just using kubernetes. But then not going too far down that rabbit hole.
ie yes kubernetes but the simplest vanilla version of it you can manage
ie yes kubernetes but the simplest vanilla version of it you can manage
I wouldn’t start with K8s, and I’ve administered it at multiple companies. Unless every one of your initial hires is a SWE turned SRE, you’re in for a bad time (and you don’t need it).
I’d personally start with Linux services on some VMs, but Docker Compose is also valid. There are plenty of wrappers around Compose to add features if you’d like.
I’d personally start with Linux services on some VMs, but Docker Compose is also valid. There are plenty of wrappers around Compose to add features if you’d like.
Simplest version of Kubernetes is zero Kubernetes. You can instead run your service using a process manager like PM2 or similar. I think even using Docker is overkill for a lot of small teams.
Related: Crawshaw's "one process programming":
https://crawshaw.io/blog/one-process-programming-notes
https://crawshaw.io/blog/one-process-programming-notes
Tach, mentioned in the article, seems to be in maintained. But I dug around and found this comment:
https://github.com/gauge-sh/tach/issues/791#issuecomment-338...
> since nobody else has done it yet, i've forked tach and updated it to work on python 3.14 - https://github.com/DetachHead/dtach
https://github.com/gauge-sh/tach/issues/791#issuecomment-338...
> since nobody else has done it yet, i've forked tach and updated it to work on python 3.14 - https://github.com/DetachHead/dtach
Most of the startups actually plan to survive for more than 2 months. And it makes total sense to think about scalability, reliability, and performance while it's still possible to change your whole stack every other week. Not forgetting about other things such as securing your cash flow, growing your talent pool, protecting your IP, etc. Finding a good balance between multiple focii is exactly the job for a founder. Of course, it's a hard job, that's why we don't see many successful startups to begin with.
Unfortunately software engineering suffers from the opposite problem of many other engineering disciplines.
Any idiot could build a bridge by just overbuilding everything, an engineer helps you build the minimum viable bridge.
In software, it’s the opposite. Idiots can easily roll out products and services with crude and basic code. You only need true engineers for the high volume high performance stuff. And if that’s not what you’re doing – you don’t need engineers. The logical conclusion is then to fire them.
Any idiot could build a bridge by just overbuilding everything, an engineer helps you build the minimum viable bridge.
In software, it’s the opposite. Idiots can easily roll out products and services with crude and basic code. You only need true engineers for the high volume high performance stuff. And if that’s not what you’re doing – you don’t need engineers. The logical conclusion is then to fire them.
Some related blog posts of mine with some thematic overlap:
• https://benwheatley.github.io/blog/2025/02/26-14.04.07.html
• https://benwheatley.github.io/blog/2024/04/07-21.31.19.html
• https://benwheatley.github.io/blog/2025/02/26-14.04.07.html
• https://benwheatley.github.io/blog/2024/04/07-21.31.19.html
Because when the cloud initially came out 2000-2010, linux had a few things to improve including networking, virtualization.
The public cloud provided a way to avoid all of this headache.
Once it got figured out, the cloud wasn't the only way to scale anymore.
Except, more people than not might not know that today about the cloud.
Linux has gotten orders of magnitude more efficient and effective, so has the public cloud, and by extension, so has scaling (and self hosting)..
The public cloud provided a way to avoid all of this headache.
Once it got figured out, the cloud wasn't the only way to scale anymore.
Except, more people than not might not know that today about the cloud.
Linux has gotten orders of magnitude more efficient and effective, so has the public cloud, and by extension, so has scaling (and self hosting)..
Frontend folks weren’t happy with how simple things were, so after seeing microservices, they invented microfrontends, and balance in complexity restored.
as an aside to the last bit about monorepos, from a dev point of view monorepos are pretty awesome. you never need to deal with complicated merge commits and you always have a consistent state of the world, enforced by ci. i would imagine maintaining the tooling and infrastructure to support said monorepo is pretty hard and painful, but as a user I love working in one.
The problem I never see addressed: scaling is not proportional to popularity. There is some sunk cost or "baseload" scale required to run a service.
Think google maps purchasing and processing satellite imagery; if you want to compete, you also need to regularly update petabytes of data. That's table stakes, even if you have one user.
Think google maps purchasing and processing satellite imagery; if you want to compete, you also need to regularly update petabytes of data. That's table stakes, even if you have one user.
CV padding is a real thing. Too many engineers love joining moonshot programs at large companies. The reason - being able to play with cool toys with the least amount of accountability.
Google has Google problems. So unless you are operating at that scale, blindly adopting their tech won’t solve your problems. But it might bring you a raise.
Google has Google problems. So unless you are operating at that scale, blindly adopting their tech won’t solve your problems. But it might bring you a raise.
Because Microsoft, Google and Amazon, to name a few, sell the story of the cloud to decision makers who can sign a subscription contract. Developers go with the flow, not daring to question the setup. Meanwhile, they host their own servers on a r-pi and ship sideprojects. Devs are not at fault here. It’s the management.
Having been on both sides, nope, it's 100% the devs pushing for this. I've had innumerable discussions trying to convince developers to simplify things, but their reaction was basically "how can we go against best practice?! Unthinkable!". It was a huge uphill battle.
Meanwhile, management didn't have an opinion, it was up to the teams to architect things how they liked, and they liked sprawling microservice everything, because they wanted to own 100% of their code, any other concern be damned.
Meanwhile, management didn't have an opinion, it was up to the teams to architect things how they liked, and they liked sprawling microservice everything, because they wanted to own 100% of their code, any other concern be damned.
Everything is scalable, because it became very easy to write scalable software. I guess that's the reason.
> The first problem every startup solves is scalability. The first problem every startup should solve is “how do we have enough money to not go bust in two months”
Why is the second question the devs' responsibility? Shouldn't it be the founders'?
Why is the second question the devs' responsibility? Shouldn't it be the founders'?
Because how the devs decide to build the system influences whether you will have enough money to not go bust in two months.
Good point, but unless the only dev is also a founder, I'd think they aren't the ones with the responsibility to come up with a business model.
They definitely have the responsibility to write software so it most efficiently serves that business model.
They definitely have the responsibility to write software so it most efficiently serves that business model.
I recall hearing, more than a decade ago IIRC, that facebook.com ran as a monolith for much longer than you would have expected. Perhaps someone with direct experience can comment?
Stack Overflow also famously ran on very little hardware.
Great article!
It seems like the whole industry has overfitted to the System designs interviews of FAANG, thus focused on extreme scaling need that few companies actually have.
The architecture the author is describing is called SOA and to me is much more optimal. There's variations of SOA that can occur in a monolith or as separate services but at the end of the day it stresses separation at interfaces that most people like microservices for. Microservices are really only architecturally necessary if certain parts of your application have outlier performance characteristics where it wouldn't make sense to scale the whole thing for any number of reasons (eg: database connection pooling or some other metric).
As for why microservices got so popular I think the answer lies in writing. The more a pattern is written on the more likely it is to be repeated and modeled.
Finally, the author also goes in after cloud usage in general. We're a decade and a half away from the first of these convergences. I was a systems engineer in those days and I remember how terrible they were. Software engineers requested a virtual machine, a pool of network resources, firewall rules, etc and eventually they got what they needed. The primary motivator of the cloud wasn't scale per se, to me it was that now a competent developer has an API to request those things on. They're operating on machine time instead of human time.
Some people extrapolate the same argument above and replace cloud with virtual machine and containers, "Why do I need containers when I can simply operate this load balancer, some VMs on an autoscaling group, and a managed database?!" Again, we quickly forgot that for many software engineer immersing themselves in image pipelines, operating systems maintenance, and networking details bogs them down in releasing the thing they really care about - the thing that logically gets them paid. Containers, again, traded that complexity to another team that solves it once for many people and lets software engineers live a relatively less complex life from their perspective.
There's an old adage I used to hear in the Marines that goes something like, "If you're not in the infantry then you're serving it. If you aren't doing that then you should question exactly what it is that you do here." The same can be said for software and those outside of product - we end up living to serve those that are building more closely on the product itself.
That's how I contextualize this history/evolution anyway.
As for why microservices got so popular I think the answer lies in writing. The more a pattern is written on the more likely it is to be repeated and modeled.
Finally, the author also goes in after cloud usage in general. We're a decade and a half away from the first of these convergences. I was a systems engineer in those days and I remember how terrible they were. Software engineers requested a virtual machine, a pool of network resources, firewall rules, etc and eventually they got what they needed. The primary motivator of the cloud wasn't scale per se, to me it was that now a competent developer has an API to request those things on. They're operating on machine time instead of human time.
Some people extrapolate the same argument above and replace cloud with virtual machine and containers, "Why do I need containers when I can simply operate this load balancer, some VMs on an autoscaling group, and a managed database?!" Again, we quickly forgot that for many software engineer immersing themselves in image pipelines, operating systems maintenance, and networking details bogs them down in releasing the thing they really care about - the thing that logically gets them paid. Containers, again, traded that complexity to another team that solves it once for many people and lets software engineers live a relatively less complex life from their perspective.
There's an old adage I used to hear in the Marines that goes something like, "If you're not in the infantry then you're serving it. If you aren't doing that then you should question exactly what it is that you do here." The same can be said for software and those outside of product - we end up living to serve those that are building more closely on the product itself.
That's how I contextualize this history/evolution anyway.
Scalability is solved problem as of 2025, this is why. Do we need it? Probably not, in most cases anyways. Most of the time people do it because they like complex systems.
[deleted]
scale vertically before horizontally...
- scaling vertically is cheaper to develop
- scaling horizontally gets you further.
What is correct for your situation depends on your human, financial and time resources.
- scaling vertically is cheaper to develop
- scaling horizontally gets you further.
What is correct for your situation depends on your human, financial and time resources.
Vertically scaling hetzner/do machines with some basic networking gets you farther than most startups survive.
[deleted]
It isn't hard to make something a bit scalable, but it is very hard to make it scalable _later_.
>The first problem every startup solves is scalability.
I don't think this is true at all. The first problem they solve is typically finding product market fit, which startups will do by sacrificing scalability and quality for speed of execution.
I don't think this is true at all. The first problem they solve is typically finding product market fit, which startups will do by sacrificing scalability and quality for speed of execution.
There's also the 'myth' of the modular monolith.
if copyright didn't exist, i'm not so sure that our current data systems would be considered "scalable" - scalability is a relative term.
That is helluva verbose way to quote Knuth ...
BEAM for the win
The topic of this blog post is the same thing that DHH has been railing on for the last few years. Run your own infrastructure, keep things simple.
[deleted]
TL;DR: premature optimization is the root of all evil
I read this, and have the opposite experience. Your monolith will fester as developers step on each others toes. You aren’t solving for scalability, you’re solving for sovereignty. Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions and your lack of understanding what a pod is or how to get your logs from your cloud.
No, this whole article reads like someone who is crying that they no longer have their AS/200. Bye. The reason people use AWS and all those 3rd party is so they don’t have to reinvent the wheel which this author seems hell bent on.
Why are we using TCP when a Unix file is fine… why are we using databases when a directory and files is fine? Why are we scaling when we aren’t Google when my single machine can serve a webpage? Why am I getting paid to be an engineer while eschewing all the things that we have advanced over the last two decades?
Yeah, these are not the right questions. The real question should be: “Now that we have scale what are we gonna do with it?”
No, this whole article reads like someone who is crying that they no longer have their AS/200. Bye. The reason people use AWS and all those 3rd party is so they don’t have to reinvent the wheel which this author seems hell bent on.
Why are we using TCP when a Unix file is fine… why are we using databases when a directory and files is fine? Why are we scaling when we aren’t Google when my single machine can serve a webpage? Why am I getting paid to be an engineer while eschewing all the things that we have advanced over the last two decades?
Yeah, these are not the right questions. The real question should be: “Now that we have scale what are we gonna do with it?”
> Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions
IME at many different SaaS companies, the only one that had serious reliability was the one that had “archaic grey beard architecture restrictions.” Devs want to use New Shiny X? Put a formal request before the architectural review committee; they’ll read it, then explain how what the team wants already exists in a different form.
I don’t know why so many developers - notably, not system design experts, nor having any background in infrastructure - think that they know better than the gray beards. They’ve seen some shit.
> and your lack of understanding what a pod is or how to get your logs from your cloud.
No one said the gray beards don’t know this. At the aforementioned company, we ran hybrid on-prem and AWS, and our product was hybrid K8s and traditional Linux services.
Re: cloud logs, every time I’ve needed logs, it has consistently been faster for me to ssh onto the instance (assuming it wasn’t ephemeral) and use ripgrep. If I don’t know where the logs were emitted from, I’ll find that first, then ssh. The only LaaS I’ve used that was worth a damn was Sumologic, but I have no idea how they are now, as that was years ago.
IME at many different SaaS companies, the only one that had serious reliability was the one that had “archaic grey beard architecture restrictions.” Devs want to use New Shiny X? Put a formal request before the architectural review committee; they’ll read it, then explain how what the team wants already exists in a different form.
I don’t know why so many developers - notably, not system design experts, nor having any background in infrastructure - think that they know better than the gray beards. They’ve seen some shit.
> and your lack of understanding what a pod is or how to get your logs from your cloud.
No one said the gray beards don’t know this. At the aforementioned company, we ran hybrid on-prem and AWS, and our product was hybrid K8s and traditional Linux services.
Re: cloud logs, every time I’ve needed logs, it has consistently been faster for me to ssh onto the instance (assuming it wasn’t ephemeral) and use ripgrep. If I don’t know where the logs were emitted from, I’ll find that first, then ssh. The only LaaS I’ve used that was worth a damn was Sumologic, but I have no idea how they are now, as that was years ago.
Splunk was (and is) the gold standard for centralized logging. The problem with it now is mainly that it's crazy expensive, though the operational engineering burden in order to run it well is non-zero and has to be accounted for. But being able to basically grep across all logs on the whole fleet, and then easily being able to visualize those results, made me never want to go back to having to ssh somewhere and run grep manually. I could write a script to ssh to all the app servers, grab the past 15 minutes of requests, extract their IPs, and plot them on a map to see which countries are hot, but that would be annoying enough that I'd really have to want to do that.
Meanwhile if you have Splunk, you specify the logfile name and how to extract the IP and then append "| iplocation clientip | geostats count by Country" to see which countries requests are coming from, for example. Or append "| stats count by http_version" and then click pie chart and get a visualization that breaks down how much traffic is still on HTTP 1.1, who's on 1.2, whos is on 2, and who's moved to QUIC/3.
Meanwhile if you have Splunk, you specify the logfile name and how to extract the IP and then append "| iplocation clientip | geostats count by Country" to see which countries requests are coming from, for example. Or append "| stats count by http_version" and then click pie chart and get a visualization that breaks down how much traffic is still on HTTP 1.1, who's on 1.2, whos is on 2, and who's moved to QUIC/3.
>step on each others toes
Which leads us to a huge problem I’ve seen over the past few decades.
Too many developers for the task at hand. It’s easier for large companies to hire 100 developers with a lower bar that may or may not be a great fit than it is to hire 5 experts.
Then you have a 100 developers that you need to keep busy and not all of them can be busy 100% of the time because most people aren’t good at making their own impactful work. Then instead of trying to actually find naturally separate projects for some of them to do, you attempt to artificially break up your existing project in a way that 100 developers can work on together (and enforce those boundaries at through a network).
This artificial separation fixes some issues (merge conflicts, some deployment issues), but it causes others (everything is a distributed system now, multi stage and multi system deployments required for the smallest changes, massive infrastructure, added network latency everywhere).
That’s not to say that some problems aren’t really so big that you need a huge number of devs, but the vast majority aren’t.
> they don’t have to reinvent the wheel
Everything is a trade off, but we shouldn’t discount the cost of using generic solutions in place of bespoke ones.
Generic solutions are never going to be as good of a fit as something designed to do exactly what you need. Sometimes the tradeoff is worth it. Sometimes it’s isn’t. Like when you need to horizontally scale just to handle the overhead. Or when you have to maintain a fork of a complex system that does way more than you need.
It’s the same problem as hiring 100 generic devs instead of 5 experts. Sometimes worth it. Sometimes not.
There’s another issue here too. If not enough people are reinventing the wheel we get stuck in local optima.
The worst part is that not enough people spend enough time even thinking about these issues to make informed decisions regarding the tradeoffs they are making.
Which leads us to a huge problem I’ve seen over the past few decades.
Too many developers for the task at hand. It’s easier for large companies to hire 100 developers with a lower bar that may or may not be a great fit than it is to hire 5 experts.
Then you have a 100 developers that you need to keep busy and not all of them can be busy 100% of the time because most people aren’t good at making their own impactful work. Then instead of trying to actually find naturally separate projects for some of them to do, you attempt to artificially break up your existing project in a way that 100 developers can work on together (and enforce those boundaries at through a network).
This artificial separation fixes some issues (merge conflicts, some deployment issues), but it causes others (everything is a distributed system now, multi stage and multi system deployments required for the smallest changes, massive infrastructure, added network latency everywhere).
That’s not to say that some problems aren’t really so big that you need a huge number of devs, but the vast majority aren’t.
> they don’t have to reinvent the wheel
Everything is a trade off, but we shouldn’t discount the cost of using generic solutions in place of bespoke ones.
Generic solutions are never going to be as good of a fit as something designed to do exactly what you need. Sometimes the tradeoff is worth it. Sometimes it’s isn’t. Like when you need to horizontally scale just to handle the overhead. Or when you have to maintain a fork of a complex system that does way more than you need.
It’s the same problem as hiring 100 generic devs instead of 5 experts. Sometimes worth it. Sometimes not.
There’s another issue here too. If not enough people are reinventing the wheel we get stuck in local optima.
The worst part is that not enough people spend enough time even thinking about these issues to make informed decisions regarding the tradeoffs they are making.
Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube.
The general setup for the largest players in that space was haproxy in front of nginx in front of several PHP servers in front of a MySQL database that had one primary r/w with one read only replica. Storage (at that time) was usually done with glusterfs. This was scalable enough at the time for hundreds of thousands of concurrent users, though the video quality was quite a bit lower than what people expect today.
Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.