Move Back to Datacenters(rakkhi.substack.com)
rakkhi.substack.com
Move Back to Datacenters
https://rakkhi.substack.com/p/why-you-should-move-back-to-datacenters
96 comments
In general when people mention "cost" as an issue for cloud, they're narrowly comparing what a CPU core or GB of RAM or disk costs on paper between cloud and datacenter/onprem.
This completely misses the business case for cloud, and is why these sorts of analyses don't convince that many business folk, and why cloud use continues to grow fast.
The business case is that someone else is doing the infrastructure management for you, at commodity prices, but you still have complete, direct control over "your" resources and ability to easily automate everything.
This tends to save much more money than any % of your CPU, memory, and disk budget could possibly achieve.
It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies!
For many companies, the "just host it in a datacenter" argument is no less silly than these scenarios.
This completely misses the business case for cloud, and is why these sorts of analyses don't convince that many business folk, and why cloud use continues to grow fast.
The business case is that someone else is doing the infrastructure management for you, at commodity prices, but you still have complete, direct control over "your" resources and ability to easily automate everything.
This tends to save much more money than any % of your CPU, memory, and disk budget could possibly achieve.
It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies!
For many companies, the "just host it in a datacenter" argument is no less silly than these scenarios.
> The business case is that someone else is doing the infrastructure management for you.
Many business people believe that moving to cloud can reduce the headcount needed for managing the infra, but that is usually not what's happening. You will still need more or less the same amount of people to patch the OS and configure the networking, but with a slightly different skill set -- instead of Cisco IOS commands, they now need to deal with AWS transit gateway.
Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you. There are also third-party companies that offers this service. Redundant power and air conditioning are the responsibility of the colocation provider and those are covered by the SLA. Those data centres do have generators on-site.
That said, configuring you top-of-rack switch is usually not covered by the co-location contract. This need to be done by a network engineer, usually the same person that would manage VPCs.
Many business people believe that moving to cloud can reduce the headcount needed for managing the infra, but that is usually not what's happening. You will still need more or less the same amount of people to patch the OS and configure the networking, but with a slightly different skill set -- instead of Cisco IOS commands, they now need to deal with AWS transit gateway.
Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you. There are also third-party companies that offers this service. Redundant power and air conditioning are the responsibility of the colocation provider and those are covered by the SLA. Those data centres do have generators on-site.
That said, configuring you top-of-rack switch is usually not covered by the co-location contract. This need to be done by a network engineer, usually the same person that would manage VPCs.
If you're treating cloud as a sort of 1:1 replacement of hardware infrastructure, then what you're saying makes sense.
But we don't ever "patch the OS." Our cloud provider does that for us. Upgrades and patching are automatic within the maintenance window. We deploy containers, the OS is just a platform layer that the cloud provider manages.
As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.
At larger companies, it can be harder to make this comparison, since you'll typically have a mixture of cloud and onprem. But if you look at teams with products deployed in the cloud, they'll often have much more frequent deployment, with better automation, and much less dependence on the always overcommitted infrastructure team(s).
> Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you.
That's not very compatible with infrastructure as code. You're essentially saying "keep doing things the old inefficient way, it's not really that bad."
There are all sorts of failure scenarios that a managed cloud system can cover from completely automatically within minutes, that would require frantic calls to remote hands in the old world. And even when recovery isn't automatic, it may still require nothing more than some pointing and clicking to recover.
> instead of Cisco IOS commands, they now need to deal with AWS transit gateway.
> This need to be done by a network engineer, usually the same person that would manage VPCs.
These aren't comparable at all. Unless a company has some sort of very special networking requirements, they can set up VPCs with some pointing and clicking in the cloud provider's web UI. This is a far cry from the level of knowledge required to manage networks at the hardware level.
But we don't ever "patch the OS." Our cloud provider does that for us. Upgrades and patching are automatic within the maintenance window. We deploy containers, the OS is just a platform layer that the cloud provider manages.
As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.
At larger companies, it can be harder to make this comparison, since you'll typically have a mixture of cloud and onprem. But if you look at teams with products deployed in the cloud, they'll often have much more frequent deployment, with better automation, and much less dependence on the always overcommitted infrastructure team(s).
> Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you.
That's not very compatible with infrastructure as code. You're essentially saying "keep doing things the old inefficient way, it's not really that bad."
There are all sorts of failure scenarios that a managed cloud system can cover from completely automatically within minutes, that would require frantic calls to remote hands in the old world. And even when recovery isn't automatic, it may still require nothing more than some pointing and clicking to recover.
> instead of Cisco IOS commands, they now need to deal with AWS transit gateway.
> This need to be done by a network engineer, usually the same person that would manage VPCs.
These aren't comparable at all. Unless a company has some sort of very special networking requirements, they can set up VPCs with some pointing and clicking in the cloud provider's web UI. This is a far cry from the level of knowledge required to manage networks at the hardware level.
> But we don't ever "patch the OS." Our cloud provider does that for us.
This is absolutely not true. Unless your cloud services are limited to function-as-a-service and serverless/containerized applications,you are indeed responsible for "patching the OS".
This kind of thing is explicitly covered even by AWS' intro to AWS courses, namely in its shared responsibility model.
> As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.
Sorry, your claim is outright unbelievable. I've never seen a company who owned any web application that only had "one or two guys, possibly even part-time, dealing with their cloud requirements." Unless the whole company is only "one or two guys, possibly even part-time", your claim simply is extremely far-fetched and blatantly unbelievable.
This is absolutely not true. Unless your cloud services are limited to function-as-a-service and serverless/containerized applications,you are indeed responsible for "patching the OS".
This kind of thing is explicitly covered even by AWS' intro to AWS courses, namely in its shared responsibility model.
> As for needing "more or less the same amount of people", that simply isn't true IME. I've seen startups with whole teams devoted to running systems in a datacenter, whereas cloud-based startups will often have one or two guys, possibly even part-time, dealing with their cloud requirements.
Sorry, your claim is outright unbelievable. I've never seen a company who owned any web application that only had "one or two guys, possibly even part-time, dealing with their cloud requirements." Unless the whole company is only "one or two guys, possibly even part-time", your claim simply is extremely far-fetched and blatantly unbelievable.
> Unless your cloud services are limited to function-as-a-service and serverless/containerized applications,
Yes, I'm talking about containerized applications, deploying to fully managed environments like EKS, Fargate, GKE, Cloud Run. If you're not using containers then yes, it will be more difficult to achieve what I'm describing.
I've been the one part-time guy for three different SaaS startups with funding in the $20-30m range, and globally distributed dev teams. I'm a software architect and dev primarily, so setting up the cloud platform is just a side activity.
For one of those companies, I was literally just a part-time contractor. Once it's set up properly, it should be easy for regular admins to operate, and devs can just follow the templates for configuring a service. It's not rocket science.
All the stuff you're imaging is so complex is exactly the stuff that the cloud lets you delegate to the provider. But you have to be willing to do it, you can't stick to the old way you've always done things and expect the cloud to make anything easier for you.
Yes, I'm talking about containerized applications, deploying to fully managed environments like EKS, Fargate, GKE, Cloud Run. If you're not using containers then yes, it will be more difficult to achieve what I'm describing.
I've been the one part-time guy for three different SaaS startups with funding in the $20-30m range, and globally distributed dev teams. I'm a software architect and dev primarily, so setting up the cloud platform is just a side activity.
For one of those companies, I was literally just a part-time contractor. Once it's set up properly, it should be easy for regular admins to operate, and devs can just follow the templates for configuring a service. It's not rocket science.
All the stuff you're imaging is so complex is exactly the stuff that the cloud lets you delegate to the provider. But you have to be willing to do it, you can't stick to the old way you've always done things and expect the cloud to make anything easier for you.
> For one of those companies, I was literally just a part-time contractor. Once it's set up properly, it should be easy for regular admins to operate, and devs can just follow the templates for configuring a service. It's not rocket science.
It's also a gross misrepresentation of the initial statement. Having a part time employee setting up a few containers to run in A container orchestration system is absolutely not what "setting up the company's cloud requirements" means. It's a blatant attempt to oversell a couple of clicks worth of work as covering a company's whole cloud requirements.
It's also a gross misrepresentation of the initial statement. Having a part time employee setting up a few containers to run in A container orchestration system is absolutely not what "setting up the company's cloud requirements" means. It's a blatant attempt to oversell a couple of clicks worth of work as covering a company's whole cloud requirements.
And now you need to patch the containers, too!
I've never heard of patching containers.
Our build pipelines rebuild container images for every deployment. All you need to upgrade some dependency is to set a version number in the Dockerfile.
There's no "too" because that's the only place we deal with that kind of version dependency. (Not counting dependencies internal to a service, like libraries.)
Our build pipelines rebuild container images for every deployment. All you need to upgrade some dependency is to set a version number in the Dockerfile.
There's no "too" because that's the only place we deal with that kind of version dependency. (Not counting dependencies internal to a service, like libraries.)
I had to do that with on-prem anyway.
In addition to the servers.
In addition to patching the routers.
In addition to the servers.
In addition to patching the routers.
I see, so you are using PaaS and I was talking about IaaS offerings. The term cloud is so vaguely defined that it can mean so many things.
If the PaaS works for your scale and budget then it’s perfectly fine. I just want to mention that it is probably no longer the case for scaled-ups.
If the PaaS works for your scale and budget then it’s perfectly fine. I just want to mention that it is probably no longer the case for scaled-ups.
You're right, I should have clarified that IaaS is exactly the stuff you want a cloud provider to take care of for you, and that's where many of the benefits come from.
But IaaS has been around for nearly 20 years now, and containers for at least 10 years, so I tend to assume people have got that message by now, especially on HN.
Re scale, you can run pretty high scale on managed Kubernetes clusters. E.g. GKE supports up to 15,000 nodes per cluster, hosting over 3.8 million concurrently running pods. Multi-cluster ingress is also supported if you need to run multiple clusters.
But IaaS has been around for nearly 20 years now, and containers for at least 10 years, so I tend to assume people have got that message by now, especially on HN.
Re scale, you can run pretty high scale on managed Kubernetes clusters. E.g. GKE supports up to 15,000 nodes per cluster, hosting over 3.8 million concurrently running pods. Multi-cluster ingress is also supported if you need to run multiple clusters.
> It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies!
I think this is an unfair analogy.
It would be more akin to saying instead of hiring Ubers and taxi's and spending several tens of thousands a quarter, buy a couple of cars for the company. But instead of needing a single person to onboard people onto the company's taxi account and paying the invoice each month, you instead need someone to buy the cars, ensure they're taxed and certified, create an internal hiring system, etc.
For some companies it does make financial sense to have your own fleet of cars, in others it makes more sense to get Ubers.
It's exactly the same as cloud vs not-cloud. If you're not taking advantage of the cloud (e.g. serverless technologies) why pay that premium when a small number of dedicated machines and a simpler architecture/stack will do the job better and cheaper? They're only as difficult to maintain as complex cloud environments.
I think this is an unfair analogy.
It would be more akin to saying instead of hiring Ubers and taxi's and spending several tens of thousands a quarter, buy a couple of cars for the company. But instead of needing a single person to onboard people onto the company's taxi account and paying the invoice each month, you instead need someone to buy the cars, ensure they're taxed and certified, create an internal hiring system, etc.
For some companies it does make financial sense to have your own fleet of cars, in others it makes more sense to get Ubers.
It's exactly the same as cloud vs not-cloud. If you're not taking advantage of the cloud (e.g. serverless technologies) why pay that premium when a small number of dedicated machines and a simpler architecture/stack will do the job better and cheaper? They're only as difficult to maintain as complex cloud environments.
Do the math and think again.
Infra engineers needed to run your "managed" cloud stack cost much more than generic sysadmins for commodity open-source on-prem solutions.
Infra engineers needed to run your "managed" cloud stack cost much more than generic sysadmins for commodity open-source on-prem solutions.
One problem is those cheap, generic sysadmins usually can't deliver a fully-automated end-to-end SDLC with IaC and automated deployment out of source control. And asking devs to do that without depending on managed services is slow and expensive.
The real business case for the cloud is it replaces capex with opex.
Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus.
In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice.
At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.
Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus.
In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice.
At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.
The REAL business case is cargo culting.
[deleted]
Genuinely feels like there’s a constant trickle of arguments from people looking to do all they can to shift the tides such that their skill set is more in demand. Or maybe it’s just an ideological obsession. I don’t know. I feel the same way whenever the “use C for everything” people catch a whiff of performance mattering and very strongly suggest that the only way to dig ourselves out of this mess is to move back to the One True Language.
In my experience The Cloud™ requires a much larger skillset, ensuring that your ops department will never stop growing.
It also depend on what requirements your business has and who you have available. The skillets are different and they overlap in places, but one way or the other if your business is growing your operations requirements will grow.
I am hugely in favour of AWS for web startups I am involved with because 1) I know AWS very well, 2) they provide hundreds of thousands of euros/dollars of credit to startups (I think all the big cloud vendors do at this stage), 3) it's useful to have architectural flexibility, and 4) they can take on a bunch of the security and availability burden that can otherwise be very labour intensive in a small company.
Also you can pry S3 and RDS Aurora out of my cold dead hands!
I am hugely in favour of AWS for web startups I am involved with because 1) I know AWS very well, 2) they provide hundreds of thousands of euros/dollars of credit to startups (I think all the big cloud vendors do at this stage), 3) it's useful to have architectural flexibility, and 4) they can take on a bunch of the security and availability burden that can otherwise be very labour intensive in a small company.
Also you can pry S3 and RDS Aurora out of my cold dead hands!
I have an impression that is "avoided" in many companies by demanding from the newly hired programmers to have "cloud experience". My problem is that that usually ends up being "programmers have taken courses about the way cloud vendor Y does the job" which then actually means "the programmers had drunk the propaganda of the cloud vendor Y" which is not a directly reusable knowledge for anything than "here's how we adjust our work to be charged by the cloud vendor Y". Which I think is guaranteed to have a company which can't even imagine doing it any other way.
I believe both arguments held some merits. It all depend on the scale.
If you are a startup and/or have a limited/specialized business scope* Cloud can be the best choice.
For organizations of 4000+ people that manage 100+ applications (from different vendors) with a 90+ people IT department this is maybe not the silver bullet. At this scale you need to keep complexity and security manageable and a more traditional redondant datacenter architecture might be less of a headache than the cloud.
*PS: What I mean by "limited/specialized business scope" is for instance Twitter/Meta. Huge scale but heavily specialized around their main products and develop most of theirs application in house.
If you are a startup and/or have a limited/specialized business scope* Cloud can be the best choice.
For organizations of 4000+ people that manage 100+ applications (from different vendors) with a 90+ people IT department this is maybe not the silver bullet. At this scale you need to keep complexity and security manageable and a more traditional redondant datacenter architecture might be less of a headache than the cloud.
*PS: What I mean by "limited/specialized business scope" is for instance Twitter/Meta. Huge scale but heavily specialized around their main products and develop most of theirs application in house.
Yeah, at that scale your devs end up bogged down in cloud concerns. We can't keep anyone at big companies very long because of all the arduous infrastructure bullshit that they have to deal with. Lots of it ends up being done outside of Jira. Total mess.
[deleted]
The other thing they keep missing.
Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired.
There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents?
It is easy to say that people can move to onprem, but orgs can barely hire the talents they need already. Sure let put even more pressure on a limited supply.
Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired.
There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents?
It is easy to say that people can move to onprem, but orgs can barely hire the talents they need already. Sure let put even more pressure on a limited supply.
AWS and Azure is a much more niche skill set.
E.g., there's probably three orders of magnitude more engineers familiar with Postgres out there in the world than with DynamoDB.
E.g., there's probably three orders of magnitude more engineers familiar with Postgres out there in the world than with DynamoDB.
To the point of running it smoothly at scale in prod with backup?
Nope. Hence you learn the RDS api and use it.
Nope. Hence you learn the RDS api and use it.
[deleted]
not to mention that the obvious/incremental move is not to completely exclusive DC, but to simply cheaper providers with managed VMs, then dedicated (but still managed) HW, and there's a long tail of that before breaking ground for a new DC makes any sense. somewhere along there's Oxide's "cloud in a rack" thing.
I agree on the security part, and what about the cost argument? I think I've seen a few companies switching over to data centers due to the costs they can save money on moving from the clouds, and I also think mature orgs prefer cap ex and depreciation to op ex (although I am not far from clueless on this point)?
[deleted]
It handwaves all the things EC2 and all services built on top do.
The amount of engineering needed just to migrate droplets live is enormous - it’s an art (according to a friend in hypervisors), and that ignores everything else regarding hypervisors and security.
Scalability is also a thing you get “for free” with serverless computing.
The amount of engineering needed just to migrate droplets live is enormous - it’s an art (according to a friend in hypervisors), and that ignores everything else regarding hypervisors and security.
Scalability is also a thing you get “for free” with serverless computing.
> The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure
Even if you feel that is a serious obstacle (which I personally don't) you can buy a managed solution (such as an Oxide rack) and use that on prem.
Even if you feel that is a serious obstacle (which I personally don't) you can buy a managed solution (such as an Oxide rack) and use that on prem.
Thanks all for your engagement and comments. I have updated the article to address many of these points.
Why is the choice always portrayed as if it's either cloud, or on-prem / colocation ?
Those are two extremes.
At work I use dedicated physical machines from Hetzner.
If I need an extra one they deliver it in a few minutes, and thanks to Ansible it's provisioned within a few minutes more.
Hetzner keeps an eye on the hardware and replaces disks, PSU's and the like if needed.
I wouldn't often advocate to do colocation or on-prem since that indeed comes with a whole set of headaches, but renting dedicated physical servers like we do offers a lot of flexibility, with very little overhead, at a price/performance ratio that makes AWS look like pure extortion.
I wouldn't often advocate to do colocation or on-prem since that indeed comes with a whole set of headaches, but renting dedicated physical servers like we do offers a lot of flexibility, with very little overhead, at a price/performance ratio that makes AWS look like pure extortion.
Yep, that's the compromise I came to as well at my solo bootstrapped SaaS. However, there are still some benefits of colocation that I would love to have: 1. ability to use M1 Mac minis as servers for specialized workloads, while having an identical developer setup for testing. This would really enable me to finetune, test, and squeeze out performance to the max and have the dream of a "single box" deployment serving hundreds of thousands of customers.
> at a price/performance ratio that makes AWS look like pure extortion.
I think if you try to host in US to have lower latency to many customers, and with some less cheap and more reliable server grade network/hardware, it is likely cost will be higher than for hetzner.
I think if you try to host in US to have lower latency to many customers, and with some less cheap and more reliable server grade network/hardware, it is likely cost will be higher than for hetzner.
I think your example falls under the cloud use case, it's a form of IaaS. The reason it doesn't scale is at some point it becomes cheaper to own the hardware or you need better scaling/redundancy and switch to managed cloud VMs/containers/functions
Move stuff back on-prem. Great. But aside from all the usual practical matters (power, backups, spare parts, service contracts, etc...), software is rapidly becoming a roadblock to move back to on-prem. If everything you have is running on Kubernetes or other open source software, great. If not, then there's an increasing amount of roadblocks being put up:
* Some software is only available as SaaS anymore
* Jacking up prices to ridiculous levels for on-prem licenses (to favor their SaaS offering of course)
* Intentionally knee capping on-prem software feature wise
* Stifle development of the on-prem product
* Force you to use some parts of their cloud services to make other things work
* Dark patterns and endless nags in software to push people to use some part of their cloud services
* Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex
* Slower response from the software vendor in case of security issues
* Exporting data from the cloud and importing it on-prem is impossible
And I can make this list go on and on and on... My point being, for small time firms that don't have the resources and solely rely on commercial software, moving back might not even be an option anymore.
* Some software is only available as SaaS anymore
* Jacking up prices to ridiculous levels for on-prem licenses (to favor their SaaS offering of course)
* Intentionally knee capping on-prem software feature wise
* Stifle development of the on-prem product
* Force you to use some parts of their cloud services to make other things work
* Dark patterns and endless nags in software to push people to use some part of their cloud services
* Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex
* Slower response from the software vendor in case of security issues
* Exporting data from the cloud and importing it on-prem is impossible
And I can make this list go on and on and on... My point being, for small time firms that don't have the resources and solely rely on commercial software, moving back might not even be an option anymore.
> Intentionally knee capping on-prem software feature wise
Or unavoidably. We have a SaaS product that's also available onprem - because we have some big enterprise customers who won't accept anything else - but there's a bunch of stuff we take advantage of in the cloud that we can't easily replicate in arbitrary onprem environments. So the onprem software ends up "kneecapped" just out of sheer pragmatism. There's not a single missing feature onprem that isn't because it would be too much effort to implement and support it.
> Stifle development of the on-prem product
Similar issue here. We have finite engineering resources, and onprem is a fraction of revenue.
> Force you to use some parts of their cloud services to make other things work
See first point.
> Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex
See second point.
In short, many of the things you're saying are indeed disadvantages of onprem products today, but it's often not the result of some sort of "intentional kneecapping" or "dark patterns". In my direct experience, it has never been the result of that.
Or unavoidably. We have a SaaS product that's also available onprem - because we have some big enterprise customers who won't accept anything else - but there's a bunch of stuff we take advantage of in the cloud that we can't easily replicate in arbitrary onprem environments. So the onprem software ends up "kneecapped" just out of sheer pragmatism. There's not a single missing feature onprem that isn't because it would be too much effort to implement and support it.
> Stifle development of the on-prem product
Similar issue here. We have finite engineering resources, and onprem is a fraction of revenue.
> Force you to use some parts of their cloud services to make other things work
See first point.
> Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex
See second point.
In short, many of the things you're saying are indeed disadvantages of onprem products today, but it's often not the result of some sort of "intentional kneecapping" or "dark patterns". In my direct experience, it has never been the result of that.
Would you mind giving an example of something that is easy in cloud but hard on prem (all I can think of is scaling up (down?) which seems kinda obviously hard)
Scaling is definitely one, but another big category is just depending on other cloud services, whether it's message queues, managed databases, object storage, ML training pipelines, software-defined networking, whatever.
In any of those cases, we either have a choice to implement and support some onprem alternative, require the customer to provide the necessary service locally (which is not particularly effective for all sorts of reasons), or simply disable features onprem.
In the cloud, we get significant benefits from using managed services. Onprem, we not only lose those benefits, we have to pay an additional cost in terms of making our code support both environments and building out the deployment and support of the services in question.
It's really exactly the same issues that make cloud attractive to businesses in the first place.
In any of those cases, we either have a choice to implement and support some onprem alternative, require the customer to provide the necessary service locally (which is not particularly effective for all sorts of reasons), or simply disable features onprem.
In the cloud, we get significant benefits from using managed services. Onprem, we not only lose those benefits, we have to pay an additional cost in terms of making our code support both environments and building out the deployment and support of the services in question.
It's really exactly the same issues that make cloud attractive to businesses in the first place.
I noticed this issue a few years ago looking for COTS or OS distributed schedule/process managers to replace a home grown solution to reduce some support risk.
Turns out in the two decades I’ve been in my industry everyone else has gone to ‘the cloud’, and the tooling has gone with it. Every product or project I looked at was skewed entirely toward physical host agnostic and/or single service reliability agnostic management.
So not only is in an issue with going back on prem, there’s entirely unserviced customers that cannot adopt the tooling given the direction it’s been developed.
Turns out in the two decades I’ve been in my industry everyone else has gone to ‘the cloud’, and the tooling has gone with it. Every product or project I looked at was skewed entirely toward physical host agnostic and/or single service reliability agnostic management.
So not only is in an issue with going back on prem, there’s entirely unserviced customers that cannot adopt the tooling given the direction it’s been developed.
> Some software is only available as SaaS anymore
This reminds me of the idea of "Service as a Software Substitute" (SaaSS): https://www.gnu.org/philosophy/who-does-that-server-really-s...
> What does it mean to say that a given computing activity is your own? It means that no one else is inherently involved in it. To clarify the meaning of “inherently involved,” we present a thought experiment. Suppose that any software tasks you might need for the job are implemented in free software and you have copies, and you have whatever data you might need, as well as computers of whatever speed, functionality and capacity might be required. Could you do this particular computing activity entirely within those computers, not communicating with anyone else's computers?
> If you could, then the activity is entirely your own. For your freedom's sake, you deserve to control it. If you do it by running free software, you do control it. However, doing it via someone else's service would give that someone else control over your computing activity. We call that scenario SaaSS, and we say it is unjust.
It sounds a bit preachy like some of the FSF/GNU stuff sometimes does, but at the same time I can't shake the feeling that they're somewhat right and that those arguments have also fallen on deaf ears most of the time.
For my own personal stuff, I just self-host everything I need in containers and mostly use free software, so I can (and have) easily move hosting providers or host things myself in my homelab, have full backups and so on. But at the same time, it definitely takes a bit of work to figure everything out and not everyone wants that, especially in environments where you can point a finger and go "the vendor is having some issues, we're off the hook".
Plus, I don't think there are many alternatives to certain software out there, like MongoDB (maybe JSON in RDBMS if you really need to, but the API would be different then), or S3 (MinIO has a license that's too strong for some, Zenko feels abandoned, Garage is the same as MinIO, maybe SeaweedFS but that doesn't have full/mostly full API compatibility from looking into it).
Honestly, it sometimes unironically feels like building boring software that relies on MariaDB/PostgreSQL and maybe Redis/RabbitMQ is the right answer. Not quite sure about blob storage, though.
This reminds me of the idea of "Service as a Software Substitute" (SaaSS): https://www.gnu.org/philosophy/who-does-that-server-really-s...
> What does it mean to say that a given computing activity is your own? It means that no one else is inherently involved in it. To clarify the meaning of “inherently involved,” we present a thought experiment. Suppose that any software tasks you might need for the job are implemented in free software and you have copies, and you have whatever data you might need, as well as computers of whatever speed, functionality and capacity might be required. Could you do this particular computing activity entirely within those computers, not communicating with anyone else's computers?
> If you could, then the activity is entirely your own. For your freedom's sake, you deserve to control it. If you do it by running free software, you do control it. However, doing it via someone else's service would give that someone else control over your computing activity. We call that scenario SaaSS, and we say it is unjust.
It sounds a bit preachy like some of the FSF/GNU stuff sometimes does, but at the same time I can't shake the feeling that they're somewhat right and that those arguments have also fallen on deaf ears most of the time.
For my own personal stuff, I just self-host everything I need in containers and mostly use free software, so I can (and have) easily move hosting providers or host things myself in my homelab, have full backups and so on. But at the same time, it definitely takes a bit of work to figure everything out and not everyone wants that, especially in environments where you can point a finger and go "the vendor is having some issues, we're off the hook".
Plus, I don't think there are many alternatives to certain software out there, like MongoDB (maybe JSON in RDBMS if you really need to, but the API would be different then), or S3 (MinIO has a license that's too strong for some, Zenko feels abandoned, Garage is the same as MinIO, maybe SeaweedFS but that doesn't have full/mostly full API compatibility from looking into it).
Honestly, it sometimes unironically feels like building boring software that relies on MariaDB/PostgreSQL and maybe Redis/RabbitMQ is the right answer. Not quite sure about blob storage, though.
It's not just services like databases or other APIs applications hook into, it's the applications themselves, like accounting software or Microsoft Exchange and Office 365, etc...
> It's not just services like databases or other APIs applications hook into, it's the applications themselves, like accounting software or Microsoft Exchange and Office 365, etc...
That's a fair point!
I mean, there are some options out there like Odoo, or stuff like Nextcloud/Collabora, OpenProject and you can run your own mail server etc., but those will increasingly feel like bad ideas for someone who needs something that "just works" and also has deadlines to manage.
It's a bit like "Nobody got fired for choosing IBM", except nowadays with Microsoft/Google/Amazon cloud offerings, the Jira's of the world and so on.
That's a fair point!
I mean, there are some options out there like Odoo, or stuff like Nextcloud/Collabora, OpenProject and you can run your own mail server etc., but those will increasingly feel like bad ideas for someone who needs something that "just works" and also has deadlines to manage.
It's a bit like "Nobody got fired for choosing IBM", except nowadays with Microsoft/Google/Amazon cloud offerings, the Jira's of the world and so on.
Actually, it often seems like "the cloud" is more along the lines of DRM. It enables software providers to guarantee payment, it often guarantees for vendors the ability to totally cut service whenever they want to. To absolutely guarantee nobody can touch their software except through them, so customer's can't automate anything without going through (and paying) the authors.
Secure computing, but in the way that's always been done: secure from the customer. Not secure for the customer.
Secure computing, but in the way that's always been done: secure from the customer. Not secure for the customer.
With IAM you can restrict all dynamodb endpoint access to a VPCE/private link. But the insecurity of the public facing endpoints is vastly overstated even without using VPCE.
The rest is a bunch of FUD - I spent years going through these points with some of the worlds best security teams to secure some of the most systemically important workloads. These arguments are fairly tired.
I’ll tackle another one - speculative attacks. First, you certainly can get bare metal exclusive access to hosts. But instances move around the broader infrastructure of an AZ, even if you’re using something like placement groups which only assure a local affinity. The chance a bad actor can colocate in the same physical device as your workload and successfully attack through side channels is vanishingly low in larger regions. To target anyone specific you would need to do such an enormous fishing expedition that it’s impractical. Further cloud providers aren’t insensate to such attacks and accounts that are doing that sort of topological mapping are easily detected. A better solution is to simply cycle your instances periodically to migrate your workloads around. For very sensitive workloads where the extraordinary unlikelihood isn’t sufficient, just get a bare metal instance.
I don’t dissuade anyone from running data centers. But I’ve yet to find anyone running back.
The rest is a bunch of FUD - I spent years going through these points with some of the worlds best security teams to secure some of the most systemically important workloads. These arguments are fairly tired.
I’ll tackle another one - speculative attacks. First, you certainly can get bare metal exclusive access to hosts. But instances move around the broader infrastructure of an AZ, even if you’re using something like placement groups which only assure a local affinity. The chance a bad actor can colocate in the same physical device as your workload and successfully attack through side channels is vanishingly low in larger regions. To target anyone specific you would need to do such an enormous fishing expedition that it’s impractical. Further cloud providers aren’t insensate to such attacks and accounts that are doing that sort of topological mapping are easily detected. A better solution is to simply cycle your instances periodically to migrate your workloads around. For very sensitive workloads where the extraordinary unlikelihood isn’t sufficient, just get a bare metal instance.
I don’t dissuade anyone from running data centers. But I’ve yet to find anyone running back.
> worlds best security teams
> the most systemically important workloads
Big claims require big evidence. How did you end up with these superlatives?
Big claims require big evidence. How did you end up with these superlatives?
A long career working hard and building a reputation for delivering hard stuff well and building the right network that I used to good effect.
I'd say the attack surface on rolling your own tech in a datacenter is substantially higher.
Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it?
Not all internet exposure is equal. Moving out of cloud often makes sense, but security isn't the right motivation for it.
Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it?
Not all internet exposure is equal. Moving out of cloud often makes sense, but security isn't the right motivation for it.
> Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it?
Is that AWS security engineer looking specifically at how your systems interact with DynamoDB? And how you have specifically configured your DynamoDB instances?
And someone rolling their own infrastructure isn’t building out their own custom software to do it. They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software.
Now, it may be true that in recent times with the amount of money made in the cloud a lot of these vendors are using carrots and sticks to push you to their cloud offerings (often by making their on Prem offerings worse), but let’s not pretend that on-prem software is two cups tied together with a string or that moving to the cloud somehow alleviates the need to have in house expertise to ensure your cloud software is running correctly and securely.
Is that AWS security engineer looking specifically at how your systems interact with DynamoDB? And how you have specifically configured your DynamoDB instances?
And someone rolling their own infrastructure isn’t building out their own custom software to do it. They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software.
Now, it may be true that in recent times with the amount of money made in the cloud a lot of these vendors are using carrots and sticks to push you to their cloud offerings (often by making their on Prem offerings worse), but let’s not pretend that on-prem software is two cups tied together with a string or that moving to the cloud somehow alleviates the need to have in house expertise to ensure your cloud software is running correctly and securely.
Again, not all configuration is equal. In hosted services a lot of the decisions have been made for you, that reduces customizability and flexibility but also footguns.
>They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software.
Of course. Even with well designed software the risk is much higher for the end user though. Deploying a cloud load balancer is pretty idiot proof. If I do the same via local nginx...configuration files, local firewall, file permission, maybe my distro repackaged it, did I set up the user right, am I updating it regularly, what about my underlying operating system security etc. And on top of that it's a once off configuration...it isn't battle tested by thousands running the same load balancer.
It's not that it can't be done, but you do pay the price in complexity and engineering hours.
>And how you have specifically configured your DynamoDB instances?
Well yeah - I'd expect them to have thought through the possible permutations in reasonable detail, which is feasible given the limited customizability. If you're self deploying the possible combinations are endless.
It's a bit like testing on android vs iphone. You can reasonably test against the 30 or whatever common iphones, but you're never testing against the 1000s of funky android hardware combinations out there.
>They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software.
Of course. Even with well designed software the risk is much higher for the end user though. Deploying a cloud load balancer is pretty idiot proof. If I do the same via local nginx...configuration files, local firewall, file permission, maybe my distro repackaged it, did I set up the user right, am I updating it regularly, what about my underlying operating system security etc. And on top of that it's a once off configuration...it isn't battle tested by thousands running the same load balancer.
It's not that it can't be done, but you do pay the price in complexity and engineering hours.
>And how you have specifically configured your DynamoDB instances?
Well yeah - I'd expect them to have thought through the possible permutations in reasonable detail, which is feasible given the limited customizability. If you're self deploying the possible combinations are endless.
It's a bit like testing on android vs iphone. You can reasonably test against the 30 or whatever common iphones, but you're never testing against the 1000s of funky android hardware combinations out there.
Is there some plugin that removes GIFs from blog posts? IMHO an article looses a lot of credibility if I also have to look at some unrelated GIF in an otherwise very well written article.
Ublock origin rule:
.gif^$domain=*.substack.com,image
.gif^$domain=*.medium.com,image
> I used to “sell” computer leases about 20 years ago saying hey don’t buy a computer, rent it and upgrade it in a year. Turns out the fine print was terrible.
This prompts the question of "So, what are you selling this year?"
I'm sympathetic to on-prem and datacenters, but maybe all the reaction GIFs are distracting the CIO/CTO from the new fine print?
This prompts the question of "So, what are you selling this year?"
I'm sympathetic to on-prem and datacenters, but maybe all the reaction GIFs are distracting the CIO/CTO from the new fine print?
No. We rolled our own stack for 10 years, until 2018, well after AWS and Azure were around.
We switched to Azure in 2018 and never looked back.
Sure you trade security (do you really, though?) in exchange for:
- not needing to head to the DC because a power supply failed and a rando who was in the cage never plugged in the redundant one
- not having to be way over-capacity in scalability, or suddenly under-capacity and emergency ordering some more 1Us
- not sifting through eBay to buy a spare hard drive that one of your boxes from 2011 needs but dell no longer makes
and the list goes on ad infinitum.
This problem has been solved. It's time to move on. Any time a company spends tinkering with their stack is less time spent delivering something of value.
We switched to Azure in 2018 and never looked back.
Sure you trade security (do you really, though?) in exchange for:
- not needing to head to the DC because a power supply failed and a rando who was in the cage never plugged in the redundant one
- not having to be way over-capacity in scalability, or suddenly under-capacity and emergency ordering some more 1Us
- not sifting through eBay to buy a spare hard drive that one of your boxes from 2011 needs but dell no longer makes
and the list goes on ad infinitum.
This problem has been solved. It's time to move on. Any time a company spends tinkering with their stack is less time spent delivering something of value.
Then there are options that are probably better for you. I've been doing colocation for many years, but I've never seen the inside of the datacenter.
Remote hands set up the hardware for me and replace failed hardware. The service is so rarely needed that it's basically free.
In recent years I've started to rent the hardware I want as I want more frequent changes. I tell my host what hardware I want and they'll acquire it and install it.
If you're running very old hardware that you can't even buy replacement drives for then you are doing something very wrong.
Need to scale fast? Then I don't have to wait for servers to arrive, I use their cloud offering that's in the same network.
I agree that it's a solved problem, but the solution to anything somewhat permanent is not the cloud. It's not really any harder to run your own hardware.
Remote hands set up the hardware for me and replace failed hardware. The service is so rarely needed that it's basically free.
In recent years I've started to rent the hardware I want as I want more frequent changes. I tell my host what hardware I want and they'll acquire it and install it.
If you're running very old hardware that you can't even buy replacement drives for then you are doing something very wrong.
Need to scale fast? Then I don't have to wait for servers to arrive, I use their cloud offering that's in the same network.
I agree that it's a solved problem, but the solution to anything somewhat permanent is not the cloud. It's not really any harder to run your own hardware.
All of those things still require management, which is the issue at hand here.
Also if you're renting your hardware... for a subscription ... that sounds an awful lot like the cloud with extra steps.
Also if you're renting your hardware... for a subscription ... that sounds an awful lot like the cloud with extra steps.
So many negative comments. Makes you wonder who works for aws…
Everyone here talks up the cloud.
They’ve all drunk the kool aid and are cloud true believers.
They’ve all drunk the kool aid and are cloud true believers.
Also, a lot of people here work for startups or have their own startups.
The cloud makes a lot of sense for startups for a variety of reasons. Low capex, easier to get started, you’re willing to pay a premium to not have to deal with certain issues, at least in the beginning, and your costs are likely not very high anyways because your usage is relatively low.
The cloud makes a lot of sense for startups for a variety of reasons. Low capex, easier to get started, you’re willing to pay a premium to not have to deal with certain issues, at least in the beginning, and your costs are likely not very high anyways because your usage is relatively low.
Reasonable professionals can disagree – strongly – without being part of a conspiracy.
That's why they call it Big Data
The analogy of this article would be: Stop using airbnbs when traveling, just build a house.
I think people overestimate the expenses related to maintaining their own infrastructure. I work for a large org that does both. We have a lot of workloads running in the public cloud, and many more on premises (for various reasons, including compliance).
Having worked on these for years, building solutions, diagnosing problems and so on I'd risk saying we have reached the point where the cloud is slowly getting more complex than traditional setups. In theory, it should be simpler, right? A Transit Gateway is a virtual device so easier to configure than a Cisco router, right?
The problem is, as the cloud providers offering gets richer, with time it has to be more complex in response to various customers' requirements, so basically you need to learn both: traditional Linux operations and networking, and a whole new class of interactions between dozens of services with hundreds of API calls. It quickly becomes overwhelming. IaC obviously helps but anybody who has maintained a large repository of CloudFormation code will tell you its not a panacea.
And then a million small quirks that basically come from you not owning the infra. E.g. Amazon is pushing the Control Tower as a model for new setups especially for larger orgs. Guess what, this thing has almost no API. Last time I checked it allows you to maybe check a state of a control. This means you need to set it up manually! In 2023! After 5 years of development! And if you add an OU, you need to register it (you guessed it - also manually!). Say you want to create a dozen of VPC endpoints - they won't have names because there is no API, you need to name them manually. And so on and so forth. Why? because you are living in someone else's place, so you you are totally dependent on them.
Having worked on these for years, building solutions, diagnosing problems and so on I'd risk saying we have reached the point where the cloud is slowly getting more complex than traditional setups. In theory, it should be simpler, right? A Transit Gateway is a virtual device so easier to configure than a Cisco router, right?
The problem is, as the cloud providers offering gets richer, with time it has to be more complex in response to various customers' requirements, so basically you need to learn both: traditional Linux operations and networking, and a whole new class of interactions between dozens of services with hundreds of API calls. It quickly becomes overwhelming. IaC obviously helps but anybody who has maintained a large repository of CloudFormation code will tell you its not a panacea.
And then a million small quirks that basically come from you not owning the infra. E.g. Amazon is pushing the Control Tower as a model for new setups especially for larger orgs. Guess what, this thing has almost no API. Last time I checked it allows you to maybe check a state of a control. This means you need to set it up manually! In 2023! After 5 years of development! And if you add an OU, you need to register it (you guessed it - also manually!). Say you want to create a dozen of VPC endpoints - they won't have names because there is no API, you need to name them manually. And so on and so forth. Why? because you are living in someone else's place, so you you are totally dependent on them.
This. A dozen time this.
I am lost in trying to start off a sensible / official / approved AWS setup but I can use some years old bash and salt s riots to setup my raspberry pis sitting there in front of me.
It's not just me (maybe mostly me) but there is a new layer of complexity - I can learn and understand the FOSS but trying to learn AWS means piercing the layers of marketing that are creeping in - one of the huge advantages of a FOSS readme is it comes from the head of the person who designed the system i want to use. not a product manager who probably misses the technical point anyway
So yeah. I am wary.
I am lost in trying to start off a sensible / official / approved AWS setup but I can use some years old bash and salt s riots to setup my raspberry pis sitting there in front of me.
It's not just me (maybe mostly me) but there is a new layer of complexity - I can learn and understand the FOSS but trying to learn AWS means piercing the layers of marketing that are creeping in - one of the huge advantages of a FOSS readme is it comes from the head of the person who designed the system i want to use. not a product manager who probably misses the technical point anyway
So yeah. I am wary.
> anybody who has maintained a large repository of CloudFormation code will tell you its not a panacea.
CloudFormation is one of the worst IaCs out there, so it's not really a good example.
One really nice model is managed Kubernetes with IaC/GitOps via tools like ArgoCD or Flux CD. This eliminates the need for things like CloudFormation, Terraform, Ansible, Puppet, OS version and patch management, etc. - most of the stuff that those tools are used to manage is replaced by the cloud provider's management of the cluster nodes. This lets you focus on the application layer, which is where the business value is. IaC in this context is pretty seamless. It also makes you much less dependent on the cloud provider's specific APIs - Kubernetes becomes your interface to all that.
CloudFormation is one of the worst IaCs out there, so it's not really a good example.
One really nice model is managed Kubernetes with IaC/GitOps via tools like ArgoCD or Flux CD. This eliminates the need for things like CloudFormation, Terraform, Ansible, Puppet, OS version and patch management, etc. - most of the stuff that those tools are used to manage is replaced by the cloud provider's management of the cluster nodes. This lets you focus on the application layer, which is where the business value is. IaC in this context is pretty seamless. It also makes you much less dependent on the cloud provider's specific APIs - Kubernetes becomes your interface to all that.
I agree it's a reasonable solution as far as workloads are concerned, but at the same time you still need the infra around it. Most of the time you need to store data outside of Kubernetes, so you'll need to deal with things like S3, RDS, DynamoDB. You will probably want ECR etc. All these things need policies and configuration. And, depending on the size or your org, also security/governance/compliance services, and you end up with a complex setup even if your main focus is k8s.
People 100 % overestimate the complexity and cost of running infrastructure on their own.
No, the analogy would be: stop staying in the same airbnb for a year, just build a house.
Or:
You have a lot of business travel wherein your employees currently stay in hotels whenever they visit major cities? Don't use hotels, they are expensive! Build and operate your own company owned full-service apartments in those cities.
You have a lot of business travel wherein your employees currently stay in hotels whenever they visit major cities? Don't use hotels, they are expensive! Build and operate your own company owned full-service apartments in those cities.
The traveling analogies make no sense.
For most companies their software is their business. Take away the software and the company stops functioning.
Their employees aren’t just visiting their software every few weeks or so. They’re using it every day from morning to evening.
There is no travel involved. Your software is your home.
So really a better analogy would be renting vs buying your home. The rental comes with the furniture and for several things that might go wrong the landlord will come and fix it, whereas when you own the house, you have to buy the furniture and arrange it, and if anything goes wrong you are responsible for fixing it.
Of course, the analogy breaks down in many ways because when you’re on Prem you are buying established software with a support license so if something does fail the support license should provide as much, if not more service than you would get on AWS.
The entire analogy is completely pointless because it adds a layer of indirection for no reason other than for some reason tech people absolutely love making analogies to physical objects.
For most companies their software is their business. Take away the software and the company stops functioning.
Their employees aren’t just visiting their software every few weeks or so. They’re using it every day from morning to evening.
There is no travel involved. Your software is your home.
So really a better analogy would be renting vs buying your home. The rental comes with the furniture and for several things that might go wrong the landlord will come and fix it, whereas when you own the house, you have to buy the furniture and arrange it, and if anything goes wrong you are responsible for fixing it.
Of course, the analogy breaks down in many ways because when you’re on Prem you are buying established software with a support license so if something does fail the support license should provide as much, if not more service than you would get on AWS.
The entire analogy is completely pointless because it adds a layer of indirection for no reason other than for some reason tech people absolutely love making analogies to physical objects.
> in traditional datacenters, with Infrastructure and Support teams separate from Development (anti DevOps), there are/were strong human checks and balances. If your Devs wanted to make an API Internet accessible and connect it to what they thought was a “sanitized” database, they probably had to raise a change, submit some firewall rules, maybe talk to a DBA to get credentials.
This is just romanticizing things. In every deployment - whether in it's "in cloud" or anywhere else - there is always the quick on off change that someone makes. Probabably for a valid reason, e.g. solving a production problem quickly. Chances are high that will go unnoticed, until another problem manifests (and that is hopefully not a security issue!).
I would argue that in cloud setups the chances for that to happen are actually slightly lower, because teams are incentivized to use immutable and declarative infrastructure. And that there might be an audit log in place which tells what changes have been made - although that still requires people to look at it, which again happens only in the case problems show up.
This is just romanticizing things. In every deployment - whether in it's "in cloud" or anywhere else - there is always the quick on off change that someone makes. Probabably for a valid reason, e.g. solving a production problem quickly. Chances are high that will go unnoticed, until another problem manifests (and that is hopefully not a security issue!).
I would argue that in cloud setups the chances for that to happen are actually slightly lower, because teams are incentivized to use immutable and declarative infrastructure. And that there might be an audit log in place which tells what changes have been made - although that still requires people to look at it, which again happens only in the case problems show up.
I thought for a second I was hearing a.suggestion that by running your own hardware you can turn off all mitigations. Which should save you a ton of money; for the same core you may be getting more than double the performance. And since you're not running any unknown foreign workloads, no one would have access to be running timing attacks on you.
The article pretty quickly moved off that point. And it seems to at points be saying that making it hard to setup infrastructure is a feature, will help the bottom line. Which has some truth but at enormous emotional cost to your teams.
I do think this shift needs to happen and I appreciate such lengthy sets of concerns being brought out. But I'm pretty lukewarm on this analysis.
The article pretty quickly moved off that point. And it seems to at points be saying that making it hard to setup infrastructure is a feature, will help the bottom line. Which has some truth but at enormous emotional cost to your teams.
I do think this shift needs to happen and I appreciate such lengthy sets of concerns being brought out. But I'm pretty lukewarm on this analysis.
In my experience small firms will struggle to hire excellent system engineers that can manage a bare metal setup, it's easier however to hire okay-ish SWE's that can design applications appropriate for cloud.
> Serverless functions – still running on someone’s server… but if you have a function that you need to evoke infrequently and cold start times don’t matter that much, yeah good for cloud.
While I agree in principle, these functions don't exist in thin air. You will often want to store the results somewhere. And then protect them. So with time you are almost replicating a lot of infrastructure. And if you are a big org you will probably want to enable SecurityHub and other security/governance services like AWS Config and things get expensive again.
While I agree in principle, these functions don't exist in thin air. You will often want to store the results somewhere. And then protect them. So with time you are almost replicating a lot of infrastructure. And if you are a big org you will probably want to enable SecurityHub and other security/governance services like AWS Config and things get expensive again.
[deleted]
Important point they miss is that unless your monthly spend with AWS is couple of mil a month you are nothing to them.
Best of luck running your own data center properly. It's a massive pain in the hoop.
Most university IT departments I know manage just fine to do this.
Allthough I get the idea of paying others to run your servers (I do so myself), I am somewhat puzzled that people so readily want to move on from the option of selfhosting. Knowing how to do things yourself has its own value. This reminds me a bit of developers who can't write anything useful without using a framework. There is life outside of frameworks and life outside of the cloud.
These are the technological foundations of being able to do anything we are doing. Granted, it can be useful to outsource these parts, but that doesn't come without it's very own short and long term costs, it strengthens some market asymmetries that you maybe don't wanna strengthen, it let's your organization forget to do things that are the basis of your business and so on.
I don't say everybody has to run their own infra, just that the work of doing so can pay off in different ways — and whether these trade offs are worth it depends entirely on the nature of your endevour.
Allthough I get the idea of paying others to run your servers (I do so myself), I am somewhat puzzled that people so readily want to move on from the option of selfhosting. Knowing how to do things yourself has its own value. This reminds me a bit of developers who can't write anything useful without using a framework. There is life outside of frameworks and life outside of the cloud.
These are the technological foundations of being able to do anything we are doing. Granted, it can be useful to outsource these parts, but that doesn't come without it's very own short and long term costs, it strengthens some market asymmetries that you maybe don't wanna strengthen, it let's your organization forget to do things that are the basis of your business and so on.
I don't say everybody has to run their own infra, just that the work of doing so can pay off in different ways — and whether these trade offs are worth it depends entirely on the nature of your endevour.
It depends on what level of "run your own infra" were talking about here. Like, purchasing hardware, racking and connecting, coming up with a nice way to PXE build your fleet and a way to manage the OS and config (chef/ansible/puppet/nix?), before implementing a complete monitoring stack that works on network equipment, raid controllers etc, and then bootstrapping your own Kubernetes cluster or hashistack (or pretending that you don't need complexity, but over time cobbling together your own unique undocumented mess). It's a huge amount of work that startups usually can't afford. The time and effort to do it properly is monstrous. You need to be a company of a certain size to devote the required engineering and support staff to it.
Try telling teams they’ll have to go back to old procurement timelines and procedures.
I couldn't read it all because the number of GIFs started to make me nervous :(
Oh, "just" run Kubernetes. Gotcha.
Naturally the first vulnerability they point out was never exploitable for cross-VM attacks on any major cloud provider at any point…
Thats a strong statement we only know about breaches that have
a) being identified
b) publicly disclosed
It requires concurrent execution on the same core of multiple VMs, which none of the major cloud providers do (at least on machines new enough to have Zen cores).
Sure they do for "serverless workloads"
I can’t find specific documentation that indicates that AWS deploys firecracker with core isolation (though I would be pretty surprised if they didn’t given their documented caution around shared-core VMs). Google uses gVisor with Linux Core Scheduling to prevent this for GKE sandbox, but I can’t find any documentation that they do the same for Cloud Functions.
In any case ”don’t use serverless functions” is a very far goalpost from “move back to the data center”.
In any case ”don’t use serverless functions” is a very far goalpost from “move back to the data center”.
That is not the only possible vulnerability.
The security points, which the main points hinges on, seem hyperfocused and in most cases misguided.
Redshift is not internet facing. If your Redshift is internet facing, you've messed up somewhere.
The CPU attack example given is for AMD ZEN, which the most common AWS workload CPUs aren't. Further, the benefit of using a cloud provider is that they put mitigations in place for most exploits, whereas running your own stack means it's now on you; running your own stack does not excuse you from having to put mitigations in place.
In the examples again, the speed to market's problems are a reflection of your organisation, not the cloud.
Stepping back a little, I'm thinking (as terrible as it is) that this is a case of blaming the tools, but never yourself; a lot of the problems the author is seeing seem very specific to their observations but are a poor use and poor understanding of AWS in general.
Overall not a great article, with a headline designed for people who already dislike AWS/GCP/Azure.