Infrastructure as code is not the answer (2022)(lukeshaughnessy.medium.com)
lukeshaughnessy.medium.com
Infrastructure as code is not the answer (2022)
https://lukeshaughnessy.medium.com/infrastructure-as-code-is-not-the-answer-cfaf4882dcba
72 comments
This article is basically one long "be careful, that hammer can hit your thumb."
While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative.
Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.
While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative.
Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.
And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs.
You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production.
Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.
This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense.
This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race).
To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.
If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.
It should have never been called infra-as-code, but infra-as-data.
See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/
To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra.
If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline.
It should have never been called infra-as-code, but infra-as-data.
See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/
This does not change my statement at all though? You fundamentally can't really predict the impact of some changes in a given environment until it's deployed. Just because you can obtain the current state of the environment and reconcile some stuff doesn't change this.
That's why you should call what you store in git the _desired_ state, not anything else. A git repository is not a live database. It's a collection of static text files that change less often than your live system. There will be bugs and misconfiguration, and sometimes the desired state is just technically not reachable, and that's fine. What the actual state is doesn't matter. Leave that to the controller. State drifting is a problem your gitops engine should detect, and should be fixed by the owner of controller code.
Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
This is correct. You need some kind of running check on the environment and when possible code that handle exceptional cases.
Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".
Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".
Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
> If you do 'DROP TABLE' in production, nothing can save you.
Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'
Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'
In particular, IaC replaced “thousands of lines of YAML and regex” or “magic shell script” solutions.
IaC is much better than what it replaced, even if there are still many sharp corners.
IaC is much better than what it replaced, even if there are still many sharp corners.
That’s not infrastructure as code means. It means engineers wouldn’t have ability to run terraform on laptop at all. They have to make a PR, get it reviewed and the code lands in some prod level branch where CI would run terraform for them.
And if things got messed up, revert the commit and we are back (mostly - May need backup restores).
It means your infrastructure is a commit that you can see history and forward/reverse.
Git represents your infra. If it’s not in the repo, it’s not there.
And if things got messed up, revert the commit and we are back (mostly - May need backup restores).
It means your infrastructure is a commit that you can see history and forward/reverse.
Git represents your infra. If it’s not in the repo, it’s not there.
The "as code" means the quality and stability of the implementation varies wildly. If you use a declarative style with proper SDLC in place, then it's fantastic.
On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.
On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.
It doesnt absolve the whole article but a delete command for your entire infrastructure is obviously a major footgun that doesnt exist with alternatives (to use one of your own lenses).
It doesn't exist with Terraform either if you are following least privilege and using a cloud provider.
Or if you just use the terraform cli properly and actually glance at the report it gives you before the “hey guy are you really, really sure about this?” Safety prompt.
You not paying attention is not terraforms fault.
You not paying attention is not terraforms fault.
"It takes an admin to take down a server, it takes a configuration management system to take down a site".
Eh, Azure has a big delete button for any Subscription. Seen it happen and take down a production system.
Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
To be fair when you click the button:
* it throws a.big warning text
* it lists out all the resources you're deleting
* you have to type out the name of the subscription
* click another button
* and then another yes I'm sure button with more warning text
At the end of the day they have to provide a button.
* it throws a.big warning text
* it lists out all the resources you're deleting
* you have to type out the name of the subscription
* click another button
* and then another yes I'm sure button with more warning text
At the end of the day they have to provide a button.
[deleted]
I'll say straight away, IaC for things like AWS in my company at least is invaluable.. the thought of even trying to deploy services with user policies etc, let alone the mess people would make of that is a disaster.
Terraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.
Terraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.
All of the downsides are 10x worse when doing infra manually.
Just do IaC well to avoid the pitfalls, rather than polishing the turd of manual infra.
Just do IaC well to avoid the pitfalls, rather than polishing the turd of manual infra.
not to mention the potential security implications of doing it manually. not being consistent across the whole deployment. Sounds like a nightmare.
With the same logic you can say that contraceptives are not the answer because they aren't 100% effective.
Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.
Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.
All of the things mentioned in the article seem more a result of bad IaC, instead of general IaC...
> Packages go out of date. Features are deprecated and stop working. Code syntax changes. Images are no longer available.
This is true for all code and we have tried and tested methods/principles for dealing with that.
> Someone’s user account got built in, and they don’t work here any more.
No comment needed.
> He was in the Prod directory — and starting nuking production.
Say it with me: Elevate Permissions
> Packages go out of date. Features are deprecated and stop working. Code syntax changes. Images are no longer available.
This is true for all code and we have tried and tested methods/principles for dealing with that.
> Someone’s user account got built in, and they don’t work here any more.
No comment needed.
> He was in the Prod directory — and starting nuking production.
Say it with me: Elevate Permissions
> Say it with me: Elevate Permissions
I always tell my team that we should be able to build and deploy from our laptops. That we should make sure we don't bake our build/deployment into third party systems, so we're still able to do things when they have outages and can move to a different one if they become unreasonably priced etc.
But that we SHOULDN'T be doing those builds/deploys from our laptops UNLESS it's an emergency. Almost no dev needs a key for the whole production infrastructure sitting on their laptop, they don't need a production DB superuser. They should know how to get access to things when needed, but embrace the safety of not being able to accidently kill something important.
I love having the power to destroy a whole companies offering at my fingertips, but the terror of running a command and realising you're doing it by accident is just not worth it.
I always tell my team that we should be able to build and deploy from our laptops. That we should make sure we don't bake our build/deployment into third party systems, so we're still able to do things when they have outages and can move to a different one if they become unreasonably priced etc.
But that we SHOULDN'T be doing those builds/deploys from our laptops UNLESS it's an emergency. Almost no dev needs a key for the whole production infrastructure sitting on their laptop, they don't need a production DB superuser. They should know how to get access to things when needed, but embrace the safety of not being able to accidently kill something important.
I love having the power to destroy a whole companies offering at my fingertips, but the terror of running a command and realising you're doing it by accident is just not worth it.
Letting teams self-serve XaaS is how you get 50 dramatically mis-sized and ill-configured DB instances in prod. This team thought they’d have 1000 QPS, and have an instance that’s 50x as large as necessary. This team doesn’t know what IOPS are but figured they’d better have a lot of them, so they’re paying thousands per month for an io2 drive when the entire dataset fits comfortably into memory. This team has no idea (fair – there are a million knobs) how to configure Postgres, and the defaults suck, so they’re getting 1/4 the performance that they’re paying for.
IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.
IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.
A lot of people work in tech these days. A lot of people write blogs.
Anyway... back to scale, consistency, and speed - IAC.
Anyway... back to scale, consistency, and speed - IAC.
Infrastructure as Code is fine, its that the current implementations are too whacky and too manpower-intensive to make it worth it.
Good automation would be where you need less manpower afterwards..
Good automation would be where you need less manpower afterwards..
IAC is one technology I will defend to the hilt. IAC is difficult because the underlying problem it is trying to solve is hard. Different people over time are making updates to a mission-critical, stateful, system that need to be coordinated with each other.
IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change.
If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.
IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change.
If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.
To avoid data loss in prod, you best be using
> trying to get some old module to work in a new cloud environment where a bunch of assumptions about how things are done no longer apply
Square hole type energy. Just because you can coerce the code to run doesn't mean it's a good idea.
New environment, new assumptions, new code. Copy-paste what's relevant, but in a new project.
> getting totally confused because what they see in the code doesn’t match reality
Terraform compares state to detect drift? When reality doesn't match code, you clobber reality or you update the code.
> It Prevents Drift!
> This is true if and only if, you only ever, ever, ever, create code to change things in your environment.
It at least makes drift obvious, even if it requires attention for it to be addressed.
> code is, in my opinion, really not good at being documentation
Unlike comments, IaC "can't" be inaccurate after applying.
> But in my experience, the effort to keep your code up-to-date is a sink for engineering time.
If the code is out of date, it's because you made a click change and didn't update the code. When this happens, terraform tells you exactly what changed in simple scenarios, and in more complex scenarios you can use import blocks to rein things in.
> Yes it’s a clickbait headline. You clicked on it, didn’t you?
Got me good. The article is a good launchboard for discussion.
Overall, the IaC scene has plenty of room for growth. Terraform import blocks and code generation is helping reduce the problem of "I clicked in the portal now I gotta update the code" because you can sync both ways. Code updates infra, infra can be used to generate code.
lifecycle {
prevent_destroy = true
}
> trying to get some old module to work in a new cloud environment where a bunch of assumptions about how things are done no longer apply
Square hole type energy. Just because you can coerce the code to run doesn't mean it's a good idea.
New environment, new assumptions, new code. Copy-paste what's relevant, but in a new project.
> getting totally confused because what they see in the code doesn’t match reality
Terraform compares state to detect drift? When reality doesn't match code, you clobber reality or you update the code.
> It Prevents Drift!
> This is true if and only if, you only ever, ever, ever, create code to change things in your environment.
It at least makes drift obvious, even if it requires attention for it to be addressed.
> code is, in my opinion, really not good at being documentation
Unlike comments, IaC "can't" be inaccurate after applying.
> But in my experience, the effort to keep your code up-to-date is a sink for engineering time.
If the code is out of date, it's because you made a click change and didn't update the code. When this happens, terraform tells you exactly what changed in simple scenarios, and in more complex scenarios you can use import blocks to rein things in.
> Yes it’s a clickbait headline. You clicked on it, didn’t you?
Got me good. The article is a good launchboard for discussion.
Overall, the IaC scene has plenty of room for growth. Terraform import blocks and code generation is helping reduce the problem of "I clicked in the portal now I gotta update the code" because you can sync both ways. Code updates infra, infra can be used to generate code.
I read the addendum. I think it was a good, understandable rant.
When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself.
If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code.
I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease.
But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.
When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself.
If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code.
I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease.
But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.
Terraform isn’t Ansible, and vice-versa. I’ve seen both being shoehorned to do the job of the other, and it isn’t pretty.
Terraform should spin up infra, period. The image it uses may well be custom-built with something like Ansible + Packer, and that’s fine – great, even. Terraform should not be used to manage configuration of the image or its applications.
Similarly, just because you can use Ansible to do anything (hello, builtin.shell) doesn’t mean you should. You should ideally make idempotent plays, of course, and if it gets to the point that you’re spending as much time post-provisioning as the TF deploy, consider baking a new image.
Terraform should spin up infra, period. The image it uses may well be custom-built with something like Ansible + Packer, and that’s fine – great, even. Terraform should not be used to manage configuration of the image or its applications.
Similarly, just because you can use Ansible to do anything (hello, builtin.shell) doesn’t mean you should. You should ideally make idempotent plays, of course, and if it gets to the point that you’re spending as much time post-provisioning as the TF deploy, consider baking a new image.
Sure, there are many tools in this area, and they can be used in many ways.
We used Ansible for infra management, we ran it against a VMWare cluster, not just patching up virtual servers managed by something else. To a larger extent than we did we could have designed the system through VMWare GUI, including a fair amount of automation but opted not to, in part because higher-ups thought it sounded nice to have YAML in git and IAC and all that jazz.
As far as I understood Ansible at the time idempotency in provisioning was easiest to achieve through a let it crash style, VMWare errored on naming being in use already and broke Ansible, and things like that.
We used Ansible for infra management, we ran it against a VMWare cluster, not just patching up virtual servers managed by something else. To a larger extent than we did we could have designed the system through VMWare GUI, including a fair amount of automation but opted not to, in part because higher-ups thought it sounded nice to have YAML in git and IAC and all that jazz.
As far as I understood Ansible at the time idempotency in provisioning was easiest to achieve through a let it crash style, VMWare errored on naming being in use already and broke Ansible, and things like that.
Not sure about the vice versa part. Ansible is pretty much a superset of Terraform. I've used in multiple places to manage vm:s and other cloud services and generally been happy with the setup.
Besides the point, but Medium.com is not the answer if you want to publish text on the internet.
As for the article he's not proposing anything better.
Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.
As for the article he's not proposing anything better.
Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.
Decent GUIs typically have sanity checks and validation built in. Yaml otoh has an extremely relaxed structure, which to me sounds dangerously easy to mess up. Terraform isn’t my domain, but do senior yaml engineers these days have proper guardrails against typos, wrong indentation, type errors etc?
I disagree with a lot of this, but here are the bit i really disagree with:
> The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there.
This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS.
Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood.
> It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read.
Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.
> The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there.
This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS.
Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood.
> It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read.
Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.
IaC is undeniably a good pattern for dealing with complex systems. The bigger problem is that it's not a solution for delineating responsibilities between software engineers and infrastructure engineers. All the specifics about an environment that need to make their way from software->infra (and in some cases the other way) still need to be worked out, and that's often a bigger problem than getting code to a box that always has access to a database (oversimplification).
I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications.
1. https://noop.dev/blog/build-apps-not-infrastructure/
I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications.
1. https://noop.dev/blog/build-apps-not-infrastructure/
I've done IaC is various forms over a decade and I believe the term does more harm than good.
It's not "Infrastructure", and it's not really "Code" either. "Infrastructure Configuration as Text" perhaps would be more descriptive. Or just stick with the samewhat older term "Configuration Management".
These days, however, it's not really in vogue. Nobody thinks they need it anymore. They have Kubernetes and containers now. When you ask enough questions about how those are managed, it usually turns out some state is being managed somewhere outside of version control. I believe the pendulum will swing back in time, but in the meantime there will be K8s on my resume, not IaC.
It's not "Infrastructure", and it's not really "Code" either. "Infrastructure Configuration as Text" perhaps would be more descriptive. Or just stick with the samewhat older term "Configuration Management".
These days, however, it's not really in vogue. Nobody thinks they need it anymore. They have Kubernetes and containers now. When you ask enough questions about how those are managed, it usually turns out some state is being managed somewhere outside of version control. I believe the pendulum will swing back in time, but in the meantime there will be K8s on my resume, not IaC.
A significant benefit of IaC is the work-savings it yields for recovery from a total-site-loss event, or the migration to multi-geo/multi-site.
Sure, very few IaC environments are going to make that sort of thing seamless--bootstrapping new infra from nothing in a hurry is always going to turn up every skeleton in every closet and require a lot of manual work. But having even 80% IaC automation yields huge time savings in those rare-but-impactful "start from zero" cases.
Having even just the easy/copy-paste parts of the infra encoded in automation can be the difference between hours and days for disaster recovery, or days and weeks for creating a new geo/staging/whatever environment.
Sure, very few IaC environments are going to make that sort of thing seamless--bootstrapping new infra from nothing in a hurry is always going to turn up every skeleton in every closet and require a lot of manual work. But having even 80% IaC automation yields huge time savings in those rare-but-impactful "start from zero" cases.
Having even just the easy/copy-paste parts of the infra encoded in automation can be the difference between hours and days for disaster recovery, or days and weeks for creating a new geo/staging/whatever environment.
Back in the olden days, I had a junior dev ask me if he could learn ops, I said sure, I was just a hacker myself. He wrote a script to make sure the hard drives in all the servers were good. A month or so later we rebooted a server and it wouldn't come back up. We finally rebooted another to see if we could learn something about the dead server. But it also wouldn't boot. Then to our horror we learned that the disk check script was writing and reading to the boot sector to make sure it was good. All of our servers were affected. Good times.
Setting up solutions in AWS is a most painful experience.
Even AWS consultants do not find it easy and trivial.
However, IaS being a complicated hornets' nest is more direct evidence of vast complexity in architecture.
The best way to address it is to decrease the complexity of the system. It will have great rewards across the board.
Most systems composed these days, have far more moving parts than what is needed.
Simplify and shrink.
Some projects really do need it, no doubt about that, but I predict that somewhere around 80% at least coud make do with far less, and the project will benefit from it to to bottom.
However, IaS being a complicated hornets' nest is more direct evidence of vast complexity in architecture.
The best way to address it is to decrease the complexity of the system. It will have great rewards across the board.
Most systems composed these days, have far more moving parts than what is needed.
Simplify and shrink.
Some projects really do need it, no doubt about that, but I predict that somewhere around 80% at least coud make do with far less, and the project will benefit from it to to bottom.
Imho, I think we've gone a bit overboard with Terraform at my company. Sure, it'll automate a bunch of stuff, but only a handful of people will really "get" it.
I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case.
I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.
I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case.
I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.
I often see people making what could be very simple terraform setups into incompehensible monolithic monsters as well.
I tend to preach separating things out and keeping things small. Why do you need a huge TF project to manage 15 services and have weird conditional parts when you can have 15 smaller projects that can deviate from a boilerplate template where it's sensible? I get that running "terraform apply" in one place makes the ops team's life easier, but the mess it creates in the actual TF isn't worthwhile IMO. And god forbid you use TF for your infrastructure and deploy using something that works more appropriately.
I tend to preach separating things out and keeping things small. Why do you need a huge TF project to manage 15 services and have weird conditional parts when you can have 15 smaller projects that can deviate from a boilerplate template where it's sensible? I get that running "terraform apply" in one place makes the ops team's life easier, but the mess it creates in the actual TF isn't worthwhile IMO. And god forbid you use TF for your infrastructure and deploy using something that works more appropriately.
The opposite can be true as well. I’ve seen TF environments that were designed with so many layers of abstractions that unraveling a single piece of infra was an incredibly long endeavor.
Solving Imaginary Problems At Scale is a thing. Monolithic TF – or at least one per env – will take you extremely far, and is easy to immediately understand. When it comes time to break out of it, don’t go overboard.
Solving Imaginary Problems At Scale is a thing. Monolithic TF – or at least one per env – will take you extremely far, and is easy to immediately understand. When it comes time to break out of it, don’t go overboard.
The network effects benefits of a large population using a common domain specific language and abstraction are easily overlooked when one on the inside of that echo chamber dealing with the warts of that context frame. I’m also not overly fond of the current state of IaC but we have a good enough foundation upon which I have a rough line of sight on a future where we develop better representations going forward, and that alone is enough for me to avoid wholesale tossing it out or halting/slowing its use.
Almost all the listed downsides of IaC are also downsides of _any_ code. I was starting to address the points one-by-one but that’s the gist of the entire article.
The alternative is we all start using Alice to click and drag pieces of logic together to build software. This is the equivalent of building infra through a web UI.
I don’t see the author’s point. Sure IaC isn’t a magical solution with zero problems but was anybody claim that?
The alternative is we all start using Alice to click and drag pieces of logic together to build software. This is the equivalent of building infra through a web UI.
I don’t see the author’s point. Sure IaC isn’t a magical solution with zero problems but was anybody claim that?
reproducible - IAC is a quantum leap over manual on this one.
self-documenting, visible - with iac you have the config in writing and in a repo vs. what the admin wrote in a paper notebook
prevents mistakes - maybe. but at least the mistakes are documented and visible.
prevents toil, lowers costs - iac wins as soon as you factor in time saved on set ups, deployments, etc... versus manual... not even close
self-documenting, visible - with iac you have the config in writing and in a repo vs. what the admin wrote in a paper notebook
prevents mistakes - maybe. but at least the mistakes are documented and visible.
prevents toil, lowers costs - iac wins as soon as you factor in time saved on set ups, deployments, etc... versus manual... not even close
tbh this is the only sound comment in this whole thread.
IaC is literally a million times better than the alternative.
sounds like we have a bunch of bitter devs in the comments who haven't had to maintain a swath of deployments and servers over the years as their main task and duty.
IaC is literally a million times better than the alternative.
sounds like we have a bunch of bitter devs in the comments who haven't had to maintain a swath of deployments and servers over the years as their main task and duty.
Maybe it is that IaC is usually imposed on developers, injecting a forced learning curve. In this respect, IaC is a lot like JavaScript, and I suspect that, much like JavaScript, people are just mad about being forced to learn it and haven’t realized all the “good parts” yet.
I'm going to agree - because there's also the complexity of keeping all the state in sync too.
Give me Fly.io or something similar any day. I'll pay the premium gratefully.
Give me Fly.io or something similar any day. I'll pay the premium gratefully.
Regardless of what the author himself is saying, this is a clear example of someone that did not actually get passed intermediate level in knowledge and expertise.
Infrastructure as code can be useful for complex systems.
These days, the cloud providers have pretty good abstractions that IMO diminish the need for IaC for many use cases.
These days, the cloud providers have pretty good abstractions that IMO diminish the need for IaC for many use cases.
We also went from - click around in the gui -> automate everything -> click around in the gui. Zero regrets.
We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk.
Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too.
To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost.
YMMV of course.
We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk.
Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too.
To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost.
YMMV of course.
When you have 10 boxes to manage doing it manually is within the realm of 1 person doing it and maybe even correctly! When you have 500... You will want consistency and repeatability. To this day I am still 'fixing' things someone did in the 10 machine area. 'oh yeah that machine is set up this way and the others are that way'. But I can not get rid of those 10 boxes because they became key infrastructure.
we use elastic beanstalk which is functionally not that different than elastic container service/Fargate. It’s basically Heroku when all is said and done.
Everything sits behind application load balancer and API gateway, all basic stuff
We have about 100 boxes, but basically only 3 “environments” for our distributed monolith. Every box in a distinct environment is provisioned and configured in exactly the same way.
need to change some config? It’s at most a few extra clicks to do it in multiple environments. Then EB handles the rolling deployments and health check failover stuff. It’s very hands off, with another small benefit of not needing to bother with fudging around with terraform state, wasting compute on building containers and paying for registry storage.
Everything sits behind application load balancer and API gateway, all basic stuff
We have about 100 boxes, but basically only 3 “environments” for our distributed monolith. Every box in a distinct environment is provisioned and configured in exactly the same way.
need to change some config? It’s at most a few extra clicks to do it in multiple environments. Then EB handles the rolling deployments and health check failover stuff. It’s very hands off, with another small benefit of not needing to bother with fudging around with terraform state, wasting compute on building containers and paying for registry storage.
I don't know much about Elastic Beanstalk but bare EC2 instances connected to the Internet should fill anyone with dread. Securing the O/S and the network is not trivial and AWS will gladly let you do the wrong thing out of the box.
IaC doesn't magically solve any of those problems but if you make a mistake once you can fix it once instead of fixing it N times.
IaC doesn't magically solve any of those problems but if you make a mistake once you can fix it once instead of fixing it N times.
We stopped hosting our website and send real post by horse carriage and, when weather is good, post pigeons. Zero regrets here too!
It’s a good thing we have tons of server cattle to feed all that straw to
At least you get less SEO spam emails than I do!
And those savings. A bunch of pigeons instead of SREs! It’s a steal!
IaC, like DevOps, is a mindset.
Dumb article, written by someone who probably has never had to setup dev>acceptance>prd environments.
GL clicking your way through that buddy.
GL clicking your way through that buddy.
Honestly a lot of the arguments in that post seem in favor of IaC, not against it.
Truly cursed take. Loved the “uh… ha ha, got’em!” walk-back in the addendum. Here’s an another anecdote for you, if that’s what we’re basing our arguments on:
My prior team had dozens of services to deploy and they were all done in the console. It took hours to roll out deployments, people constantly fucked something up (wrong version, missed dependency, config variable, etc), you had spend sometimes up to an hour to get a picture of how a service was currently stood up, was almost completely opaque and thus extremely expensive to get new hires up to speed, obviated the ability to perform post deploy automation for smoke tests and such, required giving leads broad access to environments that needed to be locked down, and the best part was you had to do it all over again for each environment. All of this went completely away when I rolled it into CloudFormation, hooked it into CI/CD pipeline, and let a maintainer kick it off with the click of a button. I guess I must have been doing something wrong /s
My prior team had dozens of services to deploy and they were all done in the console. It took hours to roll out deployments, people constantly fucked something up (wrong version, missed dependency, config variable, etc), you had spend sometimes up to an hour to get a picture of how a service was currently stood up, was almost completely opaque and thus extremely expensive to get new hires up to speed, obviated the ability to perform post deploy automation for smoke tests and such, required giving leads broad access to environments that needed to be locked down, and the best part was you had to do it all over again for each environment. All of this went completely away when I rolled it into CloudFormation, hooked it into CI/CD pipeline, and let a maintainer kick it off with the click of a button. I guess I must have been doing something wrong /s
> I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late.
That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.