Figma spends $300k on AWS daily(datacenterdynamics.com)
datacenterdynamics.com
Figma spends $300k on AWS daily
https://www.datacenterdynamics.com/en/news/design-platform-figma-spends-300000-on-aws-daily/
54 comments
I see lot of comments questioning why can't they move to another provider. When you start using any of the cloud providers there so many services you start using and then you start building integrations, email notifications, api gateways, and what not. This makes it very difficult to move all the established processes.
This is why I don't use AWS or any "cloud" provider. That is not 100% true. I have 110 GB or so of personal data backed up on Glacier. The bill is 6 cents a month, and they obviously don't charge my card.
It may take longer to start, but owning all your servers is cheaper in the long run.
At the very least, don't lock yourself into one provider. Spread out any services you use.
More than one company has screwed itself over permanently by tying itself to a single service. At the end of the day, it is the extremely poor decision to follow the trend of locking oneself into a single company that dooms it.
You would think people would have learned the lesson to avoid lock-in by now. It is absolutely Darwinian at this point.
It may take longer to start, but owning all your servers is cheaper in the long run.
At the very least, don't lock yourself into one provider. Spread out any services you use.
More than one company has screwed itself over permanently by tying itself to a single service. At the end of the day, it is the extremely poor decision to follow the trend of locking oneself into a single company that dooms it.
You would think people would have learned the lesson to avoid lock-in by now. It is absolutely Darwinian at this point.
It’s wild how someone can invent an idea, force it on everyone something no one asked for then turn around and sell it to us, layering on FOMO to keep us hooked.
Emotional manipulation at its finest.
No wonder nothing feels special anymore. Everything’s automated, abstracted, owned by someone else. If you make things by hand, you have to prove their value meanwhile, platforms like AWS are printing money.
They want us to own nothing. And if “heaven” is real, why aren’t they there already?
Emotional manipulation at its finest.
No wonder nothing feels special anymore. Everything’s automated, abstracted, owned by someone else. If you make things by hand, you have to prove their value meanwhile, platforms like AWS are printing money.
They want us to own nothing. And if “heaven” is real, why aren’t they there already?
Sir, this is Wendy's
It's insane spends like that which are the main reason that reducing cloud costs is the main money maker for my consulting business at the moment.
It's extremely rare to come across people using cloud services where you can't cut their costs by at least 30%, often more than half, sometimes as much as 90%+, and usually it results in reduced devops spends as well.
The interesting thing is that the hardest part of the sales process is to convince people that cloud isn't cheap, because there's a near religious belief that cloud providers must be cost effective in some companies.
It's extremely rare to come across people using cloud services where you can't cut their costs by at least 30%, often more than half, sometimes as much as 90%+, and usually it results in reduced devops spends as well.
The interesting thing is that the hardest part of the sales process is to convince people that cloud isn't cheap, because there's a near religious belief that cloud providers must be cost effective in some companies.
I've seen so many people being baited by whole "you will not only not need to buy expensive servers, but also not have expensive ops guys to manage it!" schtick.
Meanwhile your devs put more man-hours than your actual hardware ops teams would ever do into wrangling cloud quirks and spend way more than getting a bunch of servers (even if you include costs of financing them in that) would ever cost.
Meanwhile your devs put more man-hours than your actual hardware ops teams would ever do into wrangling cloud quirks and spend way more than getting a bunch of servers (even if you include costs of financing them in that) would ever cost.
We're discussing Figma's spend here which is a bit of a cherry pick. At their size the savings for on-prem can be substantial. For the vast majority of companies without such heavy bandwidth needs, the additional cost needs to be measured in terms of additional intenance and staffing.
Savings can be substantial at much smaller sizes and generally devops maintenance/staffing costs goes down if anything.
This is literally my bread and butter, and we make more money off customers who insist on staying on cloud providers, because they consistently need more help.
This is literally my bread and butter, and we make more money off customers who insist on staying on cloud providers, because they consistently need more help.
I'm curious what size and use case you're serving describing here. Up to a certain point, devops effort and cost is negligible when offloading to the cloud.
"Negligible" varies greatly from organisation to organisation. People tend to come to my consultancy with needs that'll cost them anything from a few hours a month and up of access to a devops resource. That tends to involve far smaller setups for cloud users than for managed hosting.
How large the setup is before that point varies a lot by organisational maturity and what exactly they do, but that can typically reflect hosting costs ranging from a couple of thousand a month and up.
But the biggest cost differential is the hosting itself. It's not unusual for a managed setup to cost half or less what the equivalent cloud environment does, for similar levels of resiliency, including the fully loaded cost of devops resources.
How large the setup is before that point varies a lot by organisational maturity and what exactly they do, but that can typically reflect hosting costs ranging from a couple of thousand a month and up.
But the biggest cost differential is the hosting itself. It's not unusual for a managed setup to cost half or less what the equivalent cloud environment does, for similar levels of resiliency, including the fully loaded cost of devops resources.
At the startup I work at, we have an AWS instance that only runs our GitLab server. The damn thing runs at 10-15% CPU nearly constantly (because GitLab's founding assumption is that everyone wants to scale to ten million global users) so our spend racks up to $70/month every months.
And yes, that's absolute peanuts for any business. But in my view, spending almost $1k annually for a GitLab server for a team of ten is ridiculous.
We could accomplish exactly the same thing on-prem with hardware we already own. It would take a couple of engineer-hours per year. As long as it's under 40 hours of maintenance per year, we come out day ahead. And over the last two years, I've had to spend a total of maybe 10 hours with hands on this box.
I just don't get the thinking that leads businesses to put unnecessary crap in the cloud. Just save your money, on-prem is cheaper for pretty much everyone not doing a saas or less than a few hundred employees
And yes, that's absolute peanuts for any business. But in my view, spending almost $1k annually for a GitLab server for a team of ten is ridiculous.
We could accomplish exactly the same thing on-prem with hardware we already own. It would take a couple of engineer-hours per year. As long as it's under 40 hours of maintenance per year, we come out day ahead. And over the last two years, I've had to spend a total of maybe 10 hours with hands on this box.
I just don't get the thinking that leads businesses to put unnecessary crap in the cloud. Just save your money, on-prem is cheaper for pretty much everyone not doing a saas or less than a few hundred employees
Your estimate is off by an order of magnitude unless you value developer time at $21 an hour. I pay union electricians 6x more than that to work for me.
> $70 a month times 12 is $840
> As long as it’s under 40 hours of maintenance (??)
> $70 a month times 12 is $840
> As long as it’s under 40 hours of maintenance (??)
I don't know where the person in question is, but those numbers seems low if you assume zero hours of maintenance for the cloud setup, sure.
The thing is, you're not likely to spend amortised more than minutes a year maintaining the on-prem hardware (or you could rent managed servers and spend nothing on it), and as they pointed out they already spend hours maintaing the EC2 instance.
I wouldn't typically recommend on prem hardware for just a single server - if you already have a rack and a proper setup, sure - otherwise just rent a cheap server somewhere, and ensure you have backups.
The thing is, you're not likely to spend amortised more than minutes a year maintaining the on-prem hardware (or you could rent managed servers and spend nothing on it), and as they pointed out they already spend hours maintaing the EC2 instance.
I wouldn't typically recommend on prem hardware for just a single server - if you already have a rack and a proper setup, sure - otherwise just rent a cheap server somewhere, and ensure you have backups.
I often starts that way, yes. And then you end up accreting more and more.
I usually tell people that if they don't want something on-prem, then at least pick a cheaper provider.
But as long as it's a VM and you don't build in reliance on other AWS services, at least you can move off it as you scale. The real problem comes when you start depending on a single cloud.
Not only do you lose the ability to do an easy move, but you also lose almost all negotiating leverage as your bill increases.
I usually tell people that if they don't want something on-prem, then at least pick a cheaper provider.
But as long as it's a VM and you don't build in reliance on other AWS services, at least you can move off it as you scale. The real problem comes when you start depending on a single cloud.
Not only do you lose the ability to do an easy move, but you also lose almost all negotiating leverage as your bill increases.
My mental model of startups is that they're free of the bureaucracy that would cause you to whine in comments versus Just Fucking Do It. So, what's the impediment to using your time to test your "10 hours" theory?
Also, my experience with GitLab isn't that the thing is hurp-durping because of scaling to ten million, it's that ruby gonna ruby
Also, my experience with GitLab isn't that the thing is hurp-durping because of scaling to ten million, it's that ruby gonna ruby
Is this production only? Based on my personal experience, many orgs could save a decent chunk just be cleaning up or spinning down dev environments.
We'd typically look at everything, and you're absolutely right. Dev environments are often also poorly managed and you can find lots of resources nobody knows who are using.
How do you recommend dev envs be treated when the company wants it to mimic prod but you still want to save money compared to prod?
Usually this isn't a big issue if people are on managed hosting, because costs tends to be so much lower, and you can often overprovision more when you know it's a dev environment, while turning down number of instances etc.
The most important thing tends to be to make it possible to run services in isolation so devs can test individual services on their own machines. Then you'll end up with much more time-limited need to spin up larger environments for integration tests etc.
The most important part, though, tends to be to ensure you have a system to keep track of what is running and who is currently using it.
The most important thing tends to be to make it possible to run services in isolation so devs can test individual services on their own machines. Then you'll end up with much more time-limited need to spin up larger environments for integration tests etc.
The most important part, though, tends to be to ensure you have a system to keep track of what is running and who is currently using it.
This 100%. SRE at $dayjob periodically has to ask us to clean up dev and uat kubernetes deployments
What percentage of clients do you think don't even need cloud?
For the vast majority of them, rented servers at a managed hosting provider would beat cloud in price hands down, but for some the difference might be small enough once they've actually optimised their cloud setup that it's a judgement call. I usually recommend clients plan for optimizing their cloud setup first, both to get a fair comparison and also because a lot of the assessments you need to do of your cloud environment to do that will also make it easier to figure out what you'd pay for a managed hosting setup as well (e.g. understanding base load vs. spikes, durability requirements for different subsets of storage etc.)
For some of them, a colo facility would be cheaper, but that's highly dependent on where you want to host it (e.g. I'm in London - putting things in a colo in London is really hard to make cost effective vs. renting servers somewhere with lower land costs; data centre operators are real-estate plays)
However, you can usually make managed hosting/colo even cheaper by sprinkling some cloud in. E.g. a "trick" that can work amazingly well is to set up the bare minimum to let you spin up what you need to handle traffic spikes in a cloud environment, and then set up monitoring for your load balancer so that you start scaling into the cloud environment once load hits a certain level, but use only the managed hosting below that level.
That way, you almost never end up actually spinning up cloud instances, but you gain the ability to run the managed hosting environment far closer to the wire than you could otherwise safely do, and drive your cost per request down accordingly.
For some of them, a colo facility would be cheaper, but that's highly dependent on where you want to host it (e.g. I'm in London - putting things in a colo in London is really hard to make cost effective vs. renting servers somewhere with lower land costs; data centre operators are real-estate plays)
However, you can usually make managed hosting/colo even cheaper by sprinkling some cloud in. E.g. a "trick" that can work amazingly well is to set up the bare minimum to let you spin up what you need to handle traffic spikes in a cloud environment, and then set up monitoring for your load balancer so that you start scaling into the cloud environment once load hits a certain level, but use only the managed hosting below that level.
That way, you almost never end up actually spinning up cloud instances, but you gain the ability to run the managed hosting environment far closer to the wire than you could otherwise safely do, and drive your cost per request down accordingly.
This is basically technical debt of some sort - they may be minting so much money it doesn't matter atm, but eventually, someone will get some wins cutting costs. Probably just the bandwidth is a lot of it.
Am I the only one that finds that not that crazy?
Any chance we can get a high-level breakdown of the AWS services they are paying for?
No. The figure is from their S-1 filing and further detail is not provided.
I don't understand how it takes 3 years to get off the cloud. I'm not a cloud developer, though. The most I've done is run code on free hosts or compute instances. Presumably there's something to the microservices and lambdas and distributed compute that makes this hard. I'm thinking if this was a monolith (like AWS themselves admit is cheaper), they could just run it locally? What a giant waste of money. I'm very glad to start seeing xAAS start to die out. At the end of the day it's just looking like more middle-men instead of how I've always assumed it was intended to be: economies of scale.
However, and missing from this article + discussion so far, is their revenue. If they pay $4/day and make $2 in revenue, that's bad. They pay $300k/day but make ~ $2250k/day in revenue. I don't know what the ratio is supposed to be, but at first blush that doesn't actually seem too bad. I'll let the more qualified take over, I'm struggling to find out how big a % of their total expenses this is.
However, and missing from this article + discussion so far, is their revenue. If they pay $4/day and make $2 in revenue, that's bad. They pay $300k/day but make ~ $2250k/day in revenue. I don't know what the ratio is supposed to be, but at first blush that doesn't actually seem too bad. I'll let the more qualified take over, I'm struggling to find out how big a % of their total expenses this is.
A mistake I see commonly whenever someone says to "just move off of the cloud" is that they see the cloud as just a VM provider. If it was, then yeah, moving to another provider wouldn't be such a big deal.
In reality, the cloud creeps into your systems in all sorts of ways. Your permissions use cloud identities, your firewalls are based on security group referencing, your cross-region connectivity relies on cloud networking products, you're managing secrets and secret rotation using cloud secrets management, your observability is based on cloud metrics, your partners have whitelisted static ip ranges that belong to the cloud provider, your database upgrades are automated by the cloud provider, your VM images are built specifically for your cloud provider, your auditing is based on the cloud provider's logs, half the items in your security compliance audit reference things that are solved by your cloud provider, your applications are running on a container scheduler managed by your cloud provider, your serverless systems are strongly coupled distributed monoliths dependent on events on cloud specific event buses, your disaster recovery plans depend on your cloud provider's backup or region failover capabilities, etc. Not to mention that when you have several hundred systems, you're not going to be moving them all at the same time. They still need to be able to communicate during the transition period (extra fun when your service-to-service authentication is dependent on your cloud) without any downtime.
It's not just a matter of dropping a server binary onto a VM from a different provider. If I think about how long it would take my org to move fully off of _a_ cloud (just to a different cloud with somewhat similar capabilities), 3 years doesn't sound unrealistic.
In reality, the cloud creeps into your systems in all sorts of ways. Your permissions use cloud identities, your firewalls are based on security group referencing, your cross-region connectivity relies on cloud networking products, you're managing secrets and secret rotation using cloud secrets management, your observability is based on cloud metrics, your partners have whitelisted static ip ranges that belong to the cloud provider, your database upgrades are automated by the cloud provider, your VM images are built specifically for your cloud provider, your auditing is based on the cloud provider's logs, half the items in your security compliance audit reference things that are solved by your cloud provider, your applications are running on a container scheduler managed by your cloud provider, your serverless systems are strongly coupled distributed monoliths dependent on events on cloud specific event buses, your disaster recovery plans depend on your cloud provider's backup or region failover capabilities, etc. Not to mention that when you have several hundred systems, you're not going to be moving them all at the same time. They still need to be able to communicate during the transition period (extra fun when your service-to-service authentication is dependent on your cloud) without any downtime.
It's not just a matter of dropping a server binary onto a VM from a different provider. If I think about how long it would take my org to move fully off of _a_ cloud (just to a different cloud with somewhat similar capabilities), 3 years doesn't sound unrealistic.
> A mistake I see commonly whenever someone says to "just move off of the cloud" is that they see the cloud as just a VM provider. If it was, then yeah, moving to another provider wouldn't be such a big deal.
I think it can still be a big deal depending on what's the overall system architecture, where are all data stores, how many services you run, and what constraints you're dealing with.
For example, when you are between two cloud providers, more often than not you will have to replace internal calls with external calls, at least within the migration stage. That as important impacts on reliability and performance. In some scenarios, this performance impact is not acceptable and might require architecting services.
I think it can still be a big deal depending on what's the overall system architecture, where are all data stores, how many services you run, and what constraints you're dealing with.
For example, when you are between two cloud providers, more often than not you will have to replace internal calls with external calls, at least within the migration stage. That as important impacts on reliability and performance. In some scenarios, this performance impact is not acceptable and might require architecting services.
This is why I have a hard rule of just doing everything in the VM for stuff I build. I'm able to move between cloud providers and even self host often with near zero effort because of this.
But you're also not the CTO of Figma, they probably have use for at least 150k$ a day of cloud :)
Which is exactly why you don't make the mistake of relying on all that.
If you can't run it locally, don't use it unless you have absolutely no choice.
If you can't run it locally, don't use it unless you have absolutely no choice.
Cloud is a drug for companies… once get on and become addictive addicted, getting off is almost impossible.
If you use cloud brand queue system, you not only need to run some servers to run the queue, you also need to rewrite all parts of the system that relied on cloud brand queue system into something else.
Repeat it by a bunch of cloud services your app use and you're in for major rewrite.
Cloud offers many, many "good enough" solutions for common problems in development and it is oh so easy to use another cloud api for few cents more
And that's not even touching a bunch of auxiliary services you now need to provide like monitoring, backups, metrics etc.
Usually best strategy is to start move from most expensive parts but if that part is entangled in cloud services it could take forever
Repeat it by a bunch of cloud services your app use and you're in for major rewrite.
Cloud offers many, many "good enough" solutions for common problems in development and it is oh so easy to use another cloud api for few cents more
And that's not even touching a bunch of auxiliary services you now need to provide like monitoring, backups, metrics etc.
Usually best strategy is to start move from most expensive parts but if that part is entangled in cloud services it could take forever
People dig themselves in hard, with reliances on all kinds of proprietary services, and complex relationships.
My experience in helping people do cloud migrations are that companies also often quickly lose oversight over which cloud services they are even still running. Sometimes systems that should've been shut down years ago are still hanging around, or parts of them anyway, like S3 buckets etc. Most companies that use cloud systems underprovision their devops because they think they don't need much for a cloud system (in fact, they typically need more to do it well).
$300k/day for their revenue is very much crazy high.
My experience in helping people do cloud migrations are that companies also often quickly lose oversight over which cloud services they are even still running. Sometimes systems that should've been shut down years ago are still hanging around, or parts of them anyway, like S3 buckets etc. Most companies that use cloud systems underprovision their devops because they think they don't need much for a cloud system (in fact, they typically need more to do it well).
$300k/day for their revenue is very much crazy high.
During good times, many companies simply don't care about which services or how much. I've worked at several startups where they got a large funding round and the word was "we don't care about cloud costs, just get it done fast / make it scale." Unfortunately, one bad mistake (like storing all data in a proprietary service like DynamoDB) can make this difficult to unwind when things get bad...
> I've always assumed it was intended to be: economies of scale
IMO, the value proposition these days is rather to avoid maintenance. I.e. help with up with all the latest patches on your infrastructure.
IMO, the value proposition these days is rather to avoid maintenance. I.e. help with up with all the latest patches on your infrastructure.
To me it's the same thing. You can pay somebody to care about that, but they might be underutilized for the majority of time so it's not worth it. If you have a service, instead of your security expert being used idk 1/x of full time, they can be y/x where y is the number of contracts. For me and my time we are just way too small to have somebody full-time dedicated. So that's how I think about it
It is a reasonable point. But i think it is not exactly that. Having your organisation focus on maintenance is a certain type of opportunity cost. It is pretty often one of your most knowledgeable engineer that does this. And it also interrupts the flow of many of your other engineers.
All of these services - be it in or out of the cloud - are trivially available as service contract with external providers.
I'd love to see some real figures on that. My gut feeling is most companies spend as much on AWS experts as they previously did on people running in house facilities but I really don't know.
Probably more. Folks that do IAC / Terraform are more expensive than old school sysadmins that would write shell scripts.
I have never had a client that got away with less maintenance because they used cloud.
In fact, those of my clients who insist of relying on cloud, tend to spend far more with me for similar complexity systems. I love taking their money, but I'd frankly rather help them save it, because longer term it's better.
In fact, those of my clients who insist of relying on cloud, tend to spend far more with me for similar complexity systems. I love taking their money, but I'd frankly rather help them save it, because longer term it's better.
The services are the "easy" part; moving data out of a cloud provider is slow and expensive. For a _really_ big dataset it can take months, sometimes years, just to complete the data transfer.
> I don't understand how it takes 3 years to get off the cloud.
Because in fact cloud is not just someone else's computer.
Because in fact cloud is not just someone else's computer.
When you code your entire app to proprietary APIs like Lambda or DynamoDB, it becomes complex to migrate.
This is absolutely bonkers, $110mln/year?? How’re they spending this much??
Utterly bananas.
Seriously the CTO should be fired.
Seriously the CTO should be fired.
On what jeeeeeeezus