The Snowflakes as Code Antipattern(infrastructure-as-code.com)
infrastructure-as-code.com
The Snowflakes as Code Antipattern
https://infrastructure-as-code.com/book/2021/11/19/snowflakes-as-code.html
25 comments
I got tired reading that. How about letting me just push php files via ftp to some outdated Debian machine. All these layers of complexity made me swear never to touch "devops" again.
The way I read the article, the “code” being talked about is not necessarily application code but the IaC used to provision the actual infrastructure.
That is, the target audience is the sysadmin who would provision the PHP/FTP server and discussing how to sanely manage that as the fleet grows with slightly separate but mostly similar differences between instances.
Maybe you as a web developer don’t want this complexity but a lot of it is inherent and someone has to deal with it at the end of the day.
That is, the target audience is the sysadmin who would provision the PHP/FTP server and discussing how to sanely manage that as the fleet grows with slightly separate but mostly similar differences between instances.
Maybe you as a web developer don’t want this complexity but a lot of it is inherent and someone has to deal with it at the end of the day.
i read it as semantic satiation. all hes done is change the terms, but you still have snowflakes because some system somewhere is cofiguring your snosflakes.
If the snowflake is defined as
class SnowflakeConfig(NormalConfig):
# Normally this is 6, but for staging it has to be 2
foo = 2
deployments:
us-east-1.prod: NormalConfig
us-east-1.staging = SnowflakeConfig
I mean modern devops is basically circling back around to that idea but with “php files” replaced with “git repo.”
To deploy code to production at $dayjob all that’s necessary is to commit to master and press the play button once you’re satisfied the code works in dev/qa/staging.
The pipeline is relatively complex because production ready infrastructure has some irreducible complexity but devs don’t have to care about it most of the time. In some cases we adopt a team of devs to work on infra related performance initiatives but that’s rare.
To deploy code to production at $dayjob all that’s necessary is to commit to master and press the play button once you’re satisfied the code works in dev/qa/staging.
The pipeline is relatively complex because production ready infrastructure has some irreducible complexity but devs don’t have to care about it most of the time. In some cases we adopt a team of devs to work on infra related performance initiatives but that’s rare.
> I mean modern devops is basically circling back around to that idea but with “php files” replaced with “git repo.”
I don't agree. DevOps is and has always been a guy FTP'ing files to a server. Except that guy was smart and wrote a script that runs as a git hook to automatically run that for him.
That script was so successful that the same figurative guy also integrated tests and checks and rollbacks into his script.
After spending some time with the script, that guy noticed that the whole software development cycle would be far better if the software they were developing and deploying had some design features to eliminate potential problems and simplify said scripts.
But in the end it's always a guy pushing files to a server. But doing it the smart way.
I don't agree. DevOps is and has always been a guy FTP'ing files to a server. Except that guy was smart and wrote a script that runs as a git hook to automatically run that for him.
That script was so successful that the same figurative guy also integrated tests and checks and rollbacks into his script.
After spending some time with the script, that guy noticed that the whole software development cycle would be far better if the software they were developing and deploying had some design features to eliminate potential problems and simplify said scripts.
But in the end it's always a guy pushing files to a server. But doing it the smart way.
It would be fine if you used programmatic scp instead of FTP I guess.
> How about letting me just push php files via ftp to some outdated Debian machine.
Cool, let's go with that.
Which file are you going to push to which server?
I mean, you need to run integration tests to ensure everything works, but you'd be out of a job in no time if you ran those pointing to a production data store. Thus you better make sure you push that php file configured to point to a gamma stage data store.
But after you ran those tests, you better promote those changes to production. Are you going to ftp that as well? You better not sneak a typo there.
And bad luck struck and you broke prod when you ftp'd your php file to prod. How much time went on until you noticed it? And what's your plan to roll back that change? You better be damn sure about what php file you want to push this time around.
With a CICD pipeline you do not have to do anything and the odds that you break a deployment because of a typo are terribly low.
> All these layers of complexity made me swear never to touch "devops" again.
Why not consider the problems that continuous delivery solve instead of put up a blanket ban on a methodology?
Because with DevOps, you do not have to ftp any random file anywhere. You just push a commit to a mainline branch and your devops magic takes care of running unit tests + integration tests + end-to-end tests + canary tests to any stage, and automatically roll back any deployment that fails some test, without any interaction at all and before you know anything went wrong.
Cool, let's go with that.
Which file are you going to push to which server?
I mean, you need to run integration tests to ensure everything works, but you'd be out of a job in no time if you ran those pointing to a production data store. Thus you better make sure you push that php file configured to point to a gamma stage data store.
But after you ran those tests, you better promote those changes to production. Are you going to ftp that as well? You better not sneak a typo there.
And bad luck struck and you broke prod when you ftp'd your php file to prod. How much time went on until you noticed it? And what's your plan to roll back that change? You better be damn sure about what php file you want to push this time around.
With a CICD pipeline you do not have to do anything and the odds that you break a deployment because of a typo are terribly low.
> All these layers of complexity made me swear never to touch "devops" again.
Why not consider the problems that continuous delivery solve instead of put up a blanket ban on a methodology?
Because with DevOps, you do not have to ftp any random file anywhere. You just push a commit to a mainline branch and your devops magic takes care of running unit tests + integration tests + end-to-end tests + canary tests to any stage, and automatically roll back any deployment that fails some test, without any interaction at all and before you know anything went wrong.
Maybe I'm spoiled, but this seems like a solution in search of a problem. Is it truly common to have separate environments described by independent copy-pastes of the conceptually-same code, rather than one configurable CDK construct or Terraform equivalent? If I heard "When someone makes a change to the code for one instance, they copy or merge the change to other instances. The process for doing this is usually manual, and involves effort and care to ensure that deliberate differences between instances are maintained, while avoiding unintended differences." rather than using a deployment pipeline, I'd run screaming.
You can still end up here with a stellar deployment pipeline.
Let’s say you have servers on AWS, Scaleway and OVH, running separate instances of the same code, in staging/test/prod. They don’t provide the exact same abstractions and accesses for VPCs and VMs. Certainly not for PaaS workloads like containers.
Let’s say you have servers on AWS, Scaleway and OVH, running separate instances of the same code, in staging/test/prod. They don’t provide the exact same abstractions and accesses for VPCs and VMs. Certainly not for PaaS workloads like containers.
Thanks for explaining! I appreciate the insight. Seem that I _have_ indeed, been spoiled (or, at least, coddled), because I've never worked on a multi-cloud environment, so have never had to consider such a case.
Terraform leads to this pattern. Unless you can make your environments exact mirror images of each other, this is what you end up with. Almost no one does exact mirror images for cost reasons. Every environment would have to be at production scale.
The "this" in your second paragraph has an unclear referent. Are you saying that copy-pasted code definitions per-environment are necessary unless those environments are exact mirror-images? If so, that is untrue - CDK (and, I assume, any other IaC framework, with the appropriate terminology change) allows Constructs to be instantiated with different variables. For instance, you could have a Construct for your environment that accepts a `numServers` variable, and instantiate that Construct multiple times with different variables for beta, gamma, etc. Only one instance of the definitional code, but multiple instantiations.
Wait… different branches for different environments? Wouldn’t you run into the same problems?
Not really, but it depends on what the repo represents. Not all envs can be identical despite the logical preference - non-wildcard SSL certs for example. I think the author here is referring to an infrastructure repo with branch-per-env as a common pattern for devops / IaaC / terraform code bases, versus env-per-folder or similar in the trunk if you absolutely have to have differences between envs. It mentions deploying build artifacts from a parallel CI/CD at the end of the article for running the software running on the infra.
Yes.
I personally prefer having IaC in the same repo as the code. Any changes to the code OR the IaC uses the git flow as per normal and requires a PR. When the PR is merged to develop its pushed to QA environments automatically - therefore updating infrastructure with the updated IaC.
We update deployment configuration separately (because why would we include anything environment-specific in the repo?), this never presents any major issues although sometimes I guess you'd need to update the config just at the same time as merging the code PR which is slightly inconvenient and is somewhat OCD triggering, but its a productive setup so I don't mind.
I personally prefer having IaC in the same repo as the code. Any changes to the code OR the IaC uses the git flow as per normal and requires a PR. When the PR is merged to develop its pushed to QA environments automatically - therefore updating infrastructure with the updated IaC.
We update deployment configuration separately (because why would we include anything environment-specific in the repo?), this never presents any major issues although sometimes I guess you'd need to update the config just at the same time as merging the code PR which is slightly inconvenient and is somewhat OCD triggering, but its a productive setup so I don't mind.
My orgs problem is that the IaC is owned by different people than the apps code running on the infra the IaC controls.
Much fun ensues in trying to have any sense of a predictable timeline when you want to rollout any app feature requiring IaC changes.
Feels worse than on-prem datacenter work cuz at least there the team was bigger, and it was slow, but mature with a ticketing system, SLAs and no key man dependencies.
With someone-else-owns-myIaC, I am dealing with 2-3 DevOps dudes who kind of sort of use Jira sometimes, and lack staff in some of the region timezones our business operates in.
Much fun ensues in trying to have any sense of a predictable timeline when you want to rollout any app feature requiring IaC changes.
Feels worse than on-prem datacenter work cuz at least there the team was bigger, and it was slow, but mature with a ticketing system, SLAs and no key man dependencies.
With someone-else-owns-myIaC, I am dealing with 2-3 DevOps dudes who kind of sort of use Jira sometimes, and lack staff in some of the region timezones our business operates in.
He has a real point, but it doesn't really give a good alternative other than "read my book".
The crux of the problem when writing Terraform code is that it doesn't have if statements. Without them you end up doing hacky things to try to recreate them. Environments are not all going to be exact replicas of each other for 100 different reasons, and so trying to use the exact same code for each just doesn't work.
I don't consider dynamic/for_each an elegant solution with the current syntax.
The crux of the problem when writing Terraform code is that it doesn't have if statements. Without them you end up doing hacky things to try to recreate them. Environments are not all going to be exact replicas of each other for 100 different reasons, and so trying to use the exact same code for each just doesn't work.
I don't consider dynamic/for_each an elegant solution with the current syntax.
Haven't used Terraform - ARM templates have a workable (though not elegant) expression system where you can use parameters to resolve stuff - so you indeed have one IaC definition and use your CI/CD to set parameter values and therefore setup whatever is needed for different environments. So your CI/CD knows about the envs and not the IaC code. As it should be I think.
> Other teams use a folder structure to maintain separate projects for each environment. They copy and edit code between projects to make changes across environments.
Yeah, it was a common pattern especially in older TF repositories from before workspaces had better support.
It takes quite a bit of discipline (and balance; sometimes the effort to make it “right” is not well-spent) to not end up in a mess when your infrastructure starts to scale across regions and providers.
It takes quite a bit of discipline (and balance; sometimes the effort to make it “right” is not well-spent) to not end up in a mess when your infrastructure starts to scale across regions and providers.
I would say it is still a common pattern. I don't see how workspaces really improves anything. The fundamental problem with Terraform is it lacks if statements. There are solutions that involve dynamic blocks and for_each, but I don't think they completely solve the problem. I would say they don't have great readability.
I would love to hear a reasonable way to not snowflake that doesn't dive down the dynamic hole.
Overall I think this is all an argument for Pulumi over Terraform.
I would love to hear a reasonable way to not snowflake that doesn't dive down the dynamic hole.
Overall I think this is all an argument for Pulumi over Terraform.
To get around the separate directories/copies someone established a pattern of doing a for each environment at the top of each tf file and then all variables are stored as a map of each env, which I find interesting.
The pattern exists in one repo with the other being all the duplicates, they’ve been in “it just works” for long enough that no one has changed them. Workspaces look pretty slick, I’ll have to check them out
The pattern exists in one repo with the other being all the duplicates, they’ve been in “it just works” for long enough that no one has changed them. Workspaces look pretty slick, I’ll have to check them out
>The two main ways people implement snowflakes as code are folders and branches.
I've seen plenty of yaml and/or json used for this, though maybe he's saying that's the sort of thing that isn't a snowflake? But it's just basically a folder structure in "code".
I've seen plenty of yaml and/or json used for this, though maybe he's saying that's the sort of thing that isn't a snowflake? But it's just basically a folder structure in "code".