How to keep your Terraform code DRY by using Terramate(blog.mineiros.io)
blog.mineiros.io
How to keep your Terraform code DRY by using Terramate
https://blog.mineiros.io/how-to-keep-your-terraform-code-dry-by-using-terramate-be5807fef8f6
16 comments
I've been using Hiera (Puppet yaml data store) to source data for terraform. As a long time puppet user, when I started writing terraform code I really missed hiera. Turns out someone ported it to Go, and someone else turned that into a terraform provider. It's a game changer. No need for terragrunt or terramate. Hiera can return anything from a single value to a very complex data block that can be used to feed a module or a whole stack. Hiera needs some information about where you're deploying to, and it gets this in the in the provider configuration in the form of key/vals: I use environment, region & stack. I decided to use a workspace naming convention that encodes these (eg: prod_use1.network). For each workspace, terraform can automatically figure out exactly what lookups it needs to do, and hiera returns native terraform data relevant to that workspace only. I have 100% dry code, and no need to create a separate directory for each environment/region state. I run my terraform init/apply in the top-level directory no matter where I'm deploying. The result is that once I've written my modules and stacks, I can build out a whole data center by creating few workspaces and running "terraform init && terraform apply" in each one. Zero extra files or directories. Dry as a bone
Cursory Googling returns two Terraform hiera providers, borh last updated in 2020. Are you using either of these, or something else?
https://github.com/ribbybibby/terraform-provider-hiera https://github.com/lyraproj/hiera_terraform
https://github.com/ribbybibby/terraform-provider-hiera https://github.com/lyraproj/hiera_terraform
Neither! It's this one:
https://github.com/chriskuchin/terraform-provider-hiera5
On the terraform registry here: https://registry.terraform.io/providers/chriskuchin/hiera5
On the terraform registry here: https://registry.terraform.io/providers/chriskuchin/hiera5
Over the weekend I plan to cut out the heart of my setup, make it generic enough to share on GitHub and do an extremely short video walkthrough of how I use it. (Except I know how my plans don't always get turned into actions)
I used to be obsessed with DRY. After having written tens of thousands of terraform lines in the last few years across a multitude of platforms and use cases, I say everything in moderation.
Going DRY is a decision you pay for in complexity, it's the typical programming problem of genericity. Sometimes it absolutely makes sense and you're a mad man if you skip it, but in isolated cases making things DRY just for the sake of it is just as nuts as not doing it when required.
It's a fine line to walk and it requires hands-on experience to know when to employ DRY and when to repeat yourself.
Going DRY is a decision you pay for in complexity, it's the typical programming problem of genericity. Sometimes it absolutely makes sense and you're a mad man if you skip it, but in isolated cases making things DRY just for the sake of it is just as nuts as not doing it when required.
It's a fine line to walk and it requires hands-on experience to know when to employ DRY and when to repeat yourself.
I don't think it is a question of moderation. It's a question of "does this chunk of code make sense as an abstraction?"
If I change the common code, do I want all users of it to realize those changes implicitly?
Terraform provides good enough tools to allow for deviations (even significant ones) between dev, staging, and prod.
IMO one of the most valuable things Terraform can do for you is stand up _the exact same stuff_ in multiple environments. If you're applying different code, your confidence in that shrinks significantly.
If I change the common code, do I want all users of it to realize those changes implicitly?
Terraform provides good enough tools to allow for deviations (even significant ones) between dev, staging, and prod.
IMO one of the most valuable things Terraform can do for you is stand up _the exact same stuff_ in multiple environments. If you're applying different code, your confidence in that shrinks significantly.
I feel like teams should be using tfe workspaces and env vars to instantiate environments rather than having a folder per-environment. It's pretty easy to replicate this workflow and results in substantially less env drift and code duplication when we do it.
I tend to use a similar approach, though with Terraform var files instead of env vars. It was also possible to have this work with TF Cloud (and presumably Enterprise?) with a single workspace env var specifying the var file. It might have been improved recently to be a first class feature.
I found this to be my favourite workflow for managing workspace variables in a way that was also flexible to being run outside of TFC/TFE for when that makes sense.
I found this to be my favourite workflow for managing workspace variables in a way that was also flexible to being run outside of TFC/TFE for when that makes sense.
How do you only deploy certain resources to certain environment this way? We don't always have matching dev/qa/prod environments.
You can conditionally deploy resources by setting the count on most resources. So you can have an environment variable with something like has_cache. Then in your terraform use a ternary or something to set the count to 1 if has_cache is true and 0 if it’s false. I have done this without much fuss.
Alternatively, consider splitting your environment up into multiple stacks which can run independently and deploy each to a workspace per env. If one env lacks the app, don't have a workspace for it.
It sounds like a lot, but it ends up being fewer files to manage and edit and your workflow can be captured in github/terraform rather than in some arbitrary file structure. It also helps prevent mistakes like updating dev and having prod rerun by accident which I've seen happen.
It sounds like a lot, but it ends up being fewer files to manage and edit and your workflow can be captured in github/terraform rather than in some arbitrary file structure. It also helps prevent mistakes like updating dev and having prod rerun by accident which I've seen happen.
branch or tag from main if they have different release cadences. control the creation or non-creation ENTIRELY thru env vars with either boolean or count. If you have some thing that doesn't exist in all envs it should be obvious and maybe painful. Having to define it in the vars file makes it clear what differences each env has at a glance.
> If you have some thing that doesn't exist in all envs it should be obvious and maybe painful.
Exactly this. In $previous_client we had `snowflake_*.tf` files. Anything that couldn't be specialised using Terraform variable files should go on those. One advantage was that we started finding things that should be commoditised to get rid of the snowflakes. Everything else was very obviously out of shape and seen as generally undesirable - although many times necessary.
> control the creation or non-creation ENTIRELY thru env vars with either boolean or count.
I would also highly discourage this. Counts for different scales across environments, sure. Counts or bools to decide whether or not to deploy something depending on the environment are discouraged and should become a snowflake.
Exactly this. In $previous_client we had `snowflake_*.tf` files. Anything that couldn't be specialised using Terraform variable files should go on those. One advantage was that we started finding things that should be commoditised to get rid of the snowflakes. Everything else was very obviously out of shape and seen as generally undesirable - although many times necessary.
> control the creation or non-creation ENTIRELY thru env vars with either boolean or count.
I would also highly discourage this. Counts for different scales across environments, sure. Counts or bools to decide whether or not to deploy something depending on the environment are discouraged and should become a snowflake.
I want to emphasize that I don't normally use counts, it's just one tool in the toolbox. Normally I try to isolate env specific apps to their own stack and I simply don't have a workspace for the env they're not deployed in. Assuming a traditional 3 tier env setup (which I don't do either -- I only have 2) it might look like:
Workspaces:
- PROD-us-east-1-All-the-VPC-stuff
- Env
- Cloud region (not always -- and the cross region ones are usually called out)
- Loosely de-coupled app border
Workspaces provide really easy "manholes" for servicing your IaC quickly when there's a mistake. They don't encourage you to manually change resources, but rather work in a smaller area and reduce blast radius. The issue I see with a lot of teams is when they're not sure how to handle cross-stack references or what they should output. I think that's an area for training and leveling up in TF.
Workspaces:
- PROD-us-east-1-All-the-VPC-stuff
- NAT count = 3
- NAT count = 3
- var instance count = 2
- var instance count = 2
- NAT count = 3
- var instance count = 1
- NAT count = 0 (maybe since we don't have the network tool we're transit vpcing or something else in dev)
- Env
- Cloud region (not always -- and the cross region ones are usually called out)
- Loosely de-coupled app border
Workspaces provide really easy "manholes" for servicing your IaC quickly when there's a mistake. They don't encourage you to manually change resources, but rather work in a smaller area and reduce blast radius. The issue I see with a lot of teams is when they're not sure how to handle cross-stack references or what they should output. I think that's an area for training and leveling up in TF.
Yep, mostly agree here.
> The issue I see with a lot of teams is when they're not sure how to handle cross-stack references or what they should output.
Very much agree. What I try to do as much as possible these days is to not cross-reference. I find terraform remote state resources a code smell. Data sources are a better way to go. No code coupling and no terraform versioning dependencies between stacks.
> The issue I see with a lot of teams is when they're not sure how to handle cross-stack references or what they should output.
Very much agree. What I try to do as much as possible these days is to not cross-reference. I find terraform remote state resources a code smell. Data sources are a better way to go. No code coupling and no terraform versioning dependencies between stacks.
I usually do data sources to get access to the attribute I might need and output arns or unique ids (that will allow people to use the data source reliably) but not guess every output they might need.
I think outputs are fine and are only a smell if you have many "peer" stacks that rely on each-others outputs. For stacks that have dramatically different cadences (IE - your vpc stack vs app stack) I don't see any issue with outputting something like VPC arn. We use a naming scheme to locate "peer" references that might be smells and try to resolve them.
As part of our DR plan, we regularly run all tf stacks from nothing to ensure they stand up without issue and that the instructions for stack order are actionable. This helped us sort out early cyclic cross-references and develop a tree/DAG style multi-stack approach. When workspaces start to settle down, we typically adopt an init-style naming system for the workspace, eg: 000-prod-vpc or 010-dev-my-web-app which allows us to ensure that we recognize which workspaces occur before and after a workspace. These names are kinda hard to predict while you're writing them, so we defer that process until it's stable since changing names is more process than it's worth. Typically we advise: xyz numbers where z increments if it's a peer but after y, y increments when it depends on a parent, x increments when there are human involved tasks. Most of our workspaces are named with the prefixes 000, 010, 011 or 020 with very rare exceptions being 012 or 021, 030 etc. This helps us identify bottlenecks or late-recovery items in our stacks as well as dependent stacks.
We're talking about deprecating this naming system with the pulumi automation api to manage deploy order when we move to pulumi, but I think the way we have it forward loads a lot of operational helpers and I'm not sure if the team will push back on pulumi or not.
I think outputs are fine and are only a smell if you have many "peer" stacks that rely on each-others outputs. For stacks that have dramatically different cadences (IE - your vpc stack vs app stack) I don't see any issue with outputting something like VPC arn. We use a naming scheme to locate "peer" references that might be smells and try to resolve them.
As part of our DR plan, we regularly run all tf stacks from nothing to ensure they stand up without issue and that the instructions for stack order are actionable. This helped us sort out early cyclic cross-references and develop a tree/DAG style multi-stack approach. When workspaces start to settle down, we typically adopt an init-style naming system for the workspace, eg: 000-prod-vpc or 010-dev-my-web-app which allows us to ensure that we recognize which workspaces occur before and after a workspace. These names are kinda hard to predict while you're writing them, so we defer that process until it's stable since changing names is more process than it's worth. Typically we advise: xyz numbers where z increments if it's a peer but after y, y increments when it depends on a parent, x increments when there are human involved tasks. Most of our workspaces are named with the prefixes 000, 010, 011 or 020 with very rare exceptions being 012 or 021, 030 etc. This helps us identify bottlenecks or late-recovery items in our stacks as well as dependent stacks.
We're talking about deprecating this naming system with the pulumi automation api to manage deploy order when we move to pulumi, but I think the way we have it forward loads a lot of operational helpers and I'm not sure if the team will push back on pulumi or not.