Unknown Values: The Secret to Terraform Plan(log.martinatkins.me)
log.martinatkins.me
Unknown Values: The Secret to Terraform Plan
https://log.martinatkins.me/2021/06/14/terraform-plan-unknown-values/
21 comments
Well... It's just my own personal opinion, but writing Pulumi code (be it Go or Python) feels like I'm writing a program that emits some Terraform-like code (conceptually, not the actual shape) under the hood and runs an interpreter afterwards. I've used that mode of operation because I needed to continuously update infrastructure to match database updates - there's another and probably more mainstream option is that Pulumi runtime runs a language runtime for me which feels even weirder as my program is not really a program but just an oddly-linked library.
I haven't really looked under the hood but I strongly suspect I'm not that far away from truth. At some point I've seriously considered a question why I'm not just templating HCL directly rather than dealing with all those Pulumi oddities.
Either way it feels very different and pretty much alien.
I haven't really looked under the hood but I strongly suspect I'm not that far away from truth. At some point I've seriously considered a question why I'm not just templating HCL directly rather than dealing with all those Pulumi oddities.
Either way it feels very different and pretty much alien.
Isn't that what Terraform does now too, with their JSON intermediate language and CDK?
I use pulumi to write infra code and I like it better than TF because I can do things "in the middle" like bring input from external files and write out values to files and it's far more natural to me than contorting myself to TF's DSL. I have an example up where I do some SES work[0].
Terraform's 0.12 release[0] was the shining light realization for me that Pulumi's was absolutely the better paradigm. When you start adding expressions, type systems, and ad-hoc iteration primitives to a DSL, you probably should be using a fully powered programming language.
As far as what is kind of offputting to you, I argue that of the two of them Pulumi is the only one with a shot of not doing it the write-it-out-to-text way. Building a representation of desired state is necessary to achieve the goals of both tools -- Pulumi gives you exposure to the objects that make up the tool in your programming language of choice, before they get serialized.
Another key innovation of Pulumi is that you can build those objects and abstractions much easier (IMO) than with TF (i.e. writing a provider), and you can build them with the tooling you're familiar with -- if you're using a supported pulumi language.
[0]: https://vadosware.io/post/setting-up-ses-with-pulumi/
[1]: https://www.hashicorp.com/blog/announcing-terraform-0-12
[0]: https://www.hashicorp.com/blog/announcing-terraform-0-12
Terraform's 0.12 release[0] was the shining light realization for me that Pulumi's was absolutely the better paradigm. When you start adding expressions, type systems, and ad-hoc iteration primitives to a DSL, you probably should be using a fully powered programming language.
As far as what is kind of offputting to you, I argue that of the two of them Pulumi is the only one with a shot of not doing it the write-it-out-to-text way. Building a representation of desired state is necessary to achieve the goals of both tools -- Pulumi gives you exposure to the objects that make up the tool in your programming language of choice, before they get serialized.
Another key innovation of Pulumi is that you can build those objects and abstractions much easier (IMO) than with TF (i.e. writing a provider), and you can build them with the tooling you're familiar with -- if you're using a supported pulumi language.
[0]: https://vadosware.io/post/setting-up-ses-with-pulumi/
[1]: https://www.hashicorp.com/blog/announcing-terraform-0-12
[0]: https://www.hashicorp.com/blog/announcing-terraform-0-12
A Pulumi program executes effectively as a coroutine with the Pulumi engine, which calls providers and resolves future values in the program which allows more resources to proceed. The fundamental difference here is that in Terraform the graph must be known up front to produce a plan, whereas in Pulumi the graph is constructed on a more ad-hoc basis. Both approaches have merit in different situations.
Yep, definitely disappointed that Pulumi was not mentioned.
Does anyone have any experiences with Pulumi to share?
I have used it in greater scale at the previous client. Working with multiple providers was much easier than with the Terraform setup the current client runs. The integrated secret handling feature is also something I miss, because it allowed for a very clean setup of GitHub configuration using Pulumi.
I also prefer Pulumi‘s custom component abstraction to terraform modules, you don’t have to declare inputs and outputs manually, and don’t require complete teardown and recreation for every change.
Also, you don’t even have to write custom components, you can simply write a function that create the resources you need, in that case you may have to handle unknown values a bit more manually, which wasn’t so bad, as it’s essentially Promise-based code.
Personally, I see the landscape like this: If you like CloudFormation templates, then Terraform is a better version of that. Especially as it’s workflows are very similar. If you like, or want, CDK then Pulumi will make you very happy, as it’s a much better version of that. Pulumi in particular makes some alternate patterns possible, e.g. using the integrated secret management, that give you more flexibility with how you approach your project.
As a side note, Pulumi is actively working with the community, accepting PRs, etc. and new provider APIs generally are available when they release. This may influence your decision, or not.
Personally, I see the landscape like this: If you like CloudFormation templates, then Terraform is a better version of that. Especially as it’s workflows are very similar. If you like, or want, CDK then Pulumi will make you very happy, as it’s a much better version of that. Pulumi in particular makes some alternate patterns possible, e.g. using the integrated secret management, that give you more flexibility with how you approach your project.
As a side note, Pulumi is actively working with the community, accepting PRs, etc. and new provider APIs generally are available when they release. This may influence your decision, or not.
I've just started using it for personal stuff and I'm impressed. Its a better CDK in my opinion, as its cloud agnostic.
I don't mean this in a snide way, but https://hn.algolia.com/?q=pulumi makes for good reading about that question as it comes up a lot
It's called Jsonnet and it definitely exists.
I want so badly to use pulumi's tools, but relying on them as a 3rd party service is just not acceptable to me.
Just found out they do support alternative state backends like Terraform does.
Correct. For my team we leverage S3 for state storage and with a recent upstream change, there is no longer a need to explicitly set a lock on the state as the lock is now set for you by default.
EDIT: Should mention that enabling bucket versioning here is also a little cherry on top (to me).
EDIT: Should mention that enabling bucket versioning here is also a little cherry on top (to me).
Can you tell me more about the state lock or link to some relevant details?
The section about promises is weird. Why would you do that? It's clearly a far better match for the kind of symbolic computation stuff lisp is famous for and functional languages in general tend to be good at, and that compilers often have to deal with (which being why those languages are so good at them). Heck, the article even mentions that `plan` is basically a compiler.
it's classic NIH. Our problem is special (spoiler alert: it's not) and we will invent a great solution that solves our special problem.
I suspect you could implement this as a Haskell DSL.
The discussion part where you are not allowed (from Terraform lang) to create behavior dependent on the actual values of "Unknown" values (to prevent an ambiguous / non-deterministic plan), reminds of Applicatives vs Monads a bit.
(Sketchy potentially non-100% true illustration ahead)
For example, if you write a command-line parsing library using a Monad, then the set of command-line options is not an upfront fixed set easy to list with --help, but can change depending on previous command line options (here the command line parsing can peek into the upfront "Unknown" values, the options used and their values, to generate even more options).
While if you write it using Applicative, then peeking into the actual values is not possible, you can't create branching behaviour.
The discussion part where you are not allowed (from Terraform lang) to create behavior dependent on the actual values of "Unknown" values (to prevent an ambiguous / non-deterministic plan), reminds of Applicatives vs Monads a bit.
(Sketchy potentially non-100% true illustration ahead)
For example, if you write a command-line parsing library using a Monad, then the set of command-line options is not an upfront fixed set easy to list with --help, but can change depending on previous command line options (here the command line parsing can peek into the upfront "Unknown" values, the options used and their values, to generate even more options).
While if you write it using Applicative, then peeking into the actual values is not possible, you can't create branching behaviour.
...but maybe you don't need to go as far as Haskell. Any language with generics could work, like Java. (Or well, void pointers or anonymous interfaces..). Just make sure the user-written plan code doesn't get any meaningful info about the unknown values, only able to pass it around.
Yes, Pulumi has the concept of `Input` and `Output`. You need to unwrap an `Output` to inspect the computed value, but that unwrap only exists because Pulumi allows you to.
EDIT: and note that Pulumi has language targets in C#, Typescript, golang, and Python
EDIT: and note that Pulumi has language targets in C#, Typescript, golang, and Python
the author is conflating expressing the infrastructure (the definition) with materializing the infrastructure. Materializing the infrastructure is an implementation detail.
there is absolutely no reason why you shouldn't be able to express your definition in python, or ruby or json.
anyone remember Vagrant? Where you could write your config is a Ruby DSL and could drop into Ruby whenever you wanted to do something "special"?
there is absolutely no reason why you shouldn't be able to express your definition in python, or ruby or json.
anyone remember Vagrant? Where you could write your config is a Ruby DSL and could drop into Ruby whenever you wanted to do something "special"?
That sure does sound like the author believes "pulumi preview" shouldn't ever work