A Year of Helping Customers Build Production-Ready Kubernetes Infrastructure(pulumi.com)
pulumi.com
A Year of Helping Customers Build Production-Ready Kubernetes Infrastructure
https://www.pulumi.com/blog/crosswalk-kubernetes/
35 comments
I’ve been very impressed with what you guys are doing. We’re extremely excited about moving from our collection of YAML, bash, and Helm templates to Pulumi. The big question mark we had was what local development would look like. Tilt and Skaffold were the two big candidates we looked at for sane local development with Kubernetes. Looks like Pulumi’s new `watch` could give us what we need?
Are there any plans to have something like rsync support to enable things like hot-reloading apps? Both Tilt and Skaffold have examples showing this capability.
Are there any plans to have something like rsync support to enable things like hot-reloading apps? Both Tilt and Skaffold have examples showing this capability.
We have a Docker[1] library which allows you to build and push Docker a container—you can plug that directly into the `image` field and it should work.
I'm _guessing_ that eventually `watch` will eventually make this more automated, but we don't have any specific plans for that, yet.
[1]: https://github.com/pulumi/pulumi-docker
I'm _guessing_ that eventually `watch` will eventually make this more automated, but we don't have any specific plans for that, yet.
[1]: https://github.com/pulumi/pulumi-docker
I’ll look into this. What I think Tilt and Skaffold are doing is just setting up a persistent volume to a local directory and letting all the reloading stuff be handled by the end-user. As long as the persistent volume links the local files to the cluster it should be good to do.
Yeah, Tilt has a couple of different ways of providing similar functionality. In one mode, they simply rebuild the container and ship update Kubernetes on each file modification. In another mode, they boot a dev container in the pod and then transparently sync the local FS content to the cluster, which means that "time to live" is the amount of time it takes to compile your app in the cluster. `watch` with the Docker library is a direct analog for the first, but we have no analog for the second.
Gotcha, so things like hot-reloading wouldn’t work right now. Still, looks like y’all are going in the right direction. Thanks for all the details! One more question, is there a way to interop Pulumi with Skaffold or Tilt?
How many of your customers actually NEEDED kubernetes ?
shrug If you have a bunch of services that you need to reliably run in containers you are going to need an orchestration of some sort. You can attempt to reinvent the wheel by doing your own or you can use an existing solution.
If you are not in the mood for wheel reinventing, your choice becomes somewhat limited. You have Kubernetes, Docker Swarm, Mesos, Nomad, a few platform specific solutions like Elastic Beanstalk and maybe some more obscure candidates.
If you don't like vendor lock-in, don't like living on the cutting edge and don't want to write too much code, your choices are down to Kubernetes, Swarm and Mesos.
If you are not in the mood for wheel reinventing, your choice becomes somewhat limited. You have Kubernetes, Docker Swarm, Mesos, Nomad, a few platform specific solutions like Elastic Beanstalk and maybe some more obscure candidates.
If you don't like vendor lock-in, don't like living on the cutting edge and don't want to write too much code, your choices are down to Kubernetes, Swarm and Mesos.
[Pulumi kube co-lead here]
I agree with most of this, but would go further. My experience (here and at Heptio) is that no one _chooses_ Kubernetes. They choose Docker, and then they (i.e., the infra team) effectively have to choose ECS or Kubernetes. In contrast, what most app developers want is a better `docker run`.
I don't know how controversial that is, so I should say this is my conclusion from ~50 intensive customer qualification conversations in my 1.5 years at Pulumi and 1 year at Heptio. In each of these conversations I ask why the customer is moving to Kubernetes. In mid- or large-sized companies (i.e., 500+ employees) the answer is completely unwavering: Docker makes it easy to ship software without worrying too much about the host, and once you have that, having a way to schedule containers to run _somewhere_ is very useful. And that's effectively how the decision is made IME.
I agree with most of this, but would go further. My experience (here and at Heptio) is that no one _chooses_ Kubernetes. They choose Docker, and then they (i.e., the infra team) effectively have to choose ECS or Kubernetes. In contrast, what most app developers want is a better `docker run`.
I don't know how controversial that is, so I should say this is my conclusion from ~50 intensive customer qualification conversations in my 1.5 years at Pulumi and 1 year at Heptio. In each of these conversations I ask why the customer is moving to Kubernetes. In mid- or large-sized companies (i.e., 500+ employees) the answer is completely unwavering: Docker makes it easy to ship software without worrying too much about the host, and once you have that, having a way to schedule containers to run _somewhere_ is very useful. And that's effectively how the decision is made IME.
I am very much hunting for a cli, automatable method to do:
- app installs
- orchestration of same across multiple boxes in a cluster
- cluster monitoring: detect when a task or box goes down running the service(s) elsewhere in the cluster
It seems like kubernetes is incomplete with respect to these needs. Kubernetes depends on docker but neither does orchestration. In the context of this HN post what do I need. I am doing golang on 100pct Linux. Other than toolng I need for app installs the boxes themselves are provisioned by others. Here I am focused on app installs only.
Any help much, much appreciated.
- app installs
- orchestration of same across multiple boxes in a cluster
- cluster monitoring: detect when a task or box goes down running the service(s) elsewhere in the cluster
It seems like kubernetes is incomplete with respect to these needs. Kubernetes depends on docker but neither does orchestration. In the context of this HN post what do I need. I am doing golang on 100pct Linux. Other than toolng I need for app installs the boxes themselves are provisioned by others. Here I am focused on app installs only.
Any help much, much appreciated.
Kubernetes is pretty feature complete, but its focus is on containers, not naked apps. If you are willing to pack your apps into docker, kubernetes will solve the orchestration and monitoring problems. If you are not, you might want to look for something else. In particular, take a look at Nomad (https://www.nomadproject.io/). I am not affiliated with the project in any way and at my workplace we went the kube route, but Nomad does provide an ability to orchestrate arbitrary processes across multiple hosts.
Take a look at Kubernetes, Helm, and the Cluster API.
Can’t tell if grandparent post wants to manage apps on a cluster (see below) or apps that are a cluster (see https://cluster-api.sigs.k8s.io/)
If it’s letting a dev manage apps on a cluster someone else let them put apps on, while staying in dev mode, maybe devs could appreciate the cluster provisioning SREs enabling app devs with so-called GitOps or AutoDevOps approaches, prescriptivist ways of leveraging Docker, K8s, and Helm, such as:
https://www.weave.works/blog/what-is-gitops-really
https://docs.gitlab.com/ee/topics/autodevops/index.html
If it’s letting a dev manage apps on a cluster someone else let them put apps on, while staying in dev mode, maybe devs could appreciate the cluster provisioning SREs enabling app devs with so-called GitOps or AutoDevOps approaches, prescriptivist ways of leveraging Docker, K8s, and Helm, such as:
https://www.weave.works/blog/what-is-gitops-really
https://docs.gitlab.com/ee/topics/autodevops/index.html
That was my first tought.
Microsoft, Amazon, Google, IBM, Alibaba and maybe the rest of the small cloud providers.
Microsoft, Amazon, Google, IBM, Alibaba and maybe the rest of the small cloud providers.
Ha! I co-lead these efforts, and what I will say is that we try not to "lead the witness" here. :) So customers choose Kubernetes, and then we help them, but we don't help them to choose Kubernetes, so to speak.
We've had customers and users of Pulumi at many different sizes and stages looking at or migrating to Kubernetes - and I'd say there are a few consistent reasons we hear for making the choice to adopt Kubernetes.
First, Kubernetes offers a rich, standards-based API for compute. For teams who want to deliver containerized applications, using Kubernetes allows targeting a standards API instead of a cloud-provider specific API. This up-levels the same benefit folks get with cloud IaaS solutions where they can write standard Linux applications. Now the standardized and "portable" part of their applications extends to some key parts of the compute infrastructure as well.
Second, we see a ton of Cloud + Kubernetes - teams who are still taking advantage of a lot of managed services and operational benefits from AWS/Azure/GCP/etc., but want to combine that with the container orchestration and rich API of Kubernetes. This means leaving stateful workloads in managed services like Aurora or Cloud SQL, delegating auth to managed IAM services, etc. Pulumi happens to be a really nice fit for doing these together.
And lastly, there's just a really active ecosystem that's building a lot of value around Kubernetes. Some of those pieces aren't yet fully mature, but directionally many teams are betting on this being where the next wave of mature cloud platform services will become available.
First, Kubernetes offers a rich, standards-based API for compute. For teams who want to deliver containerized applications, using Kubernetes allows targeting a standards API instead of a cloud-provider specific API. This up-levels the same benefit folks get with cloud IaaS solutions where they can write standard Linux applications. Now the standardized and "portable" part of their applications extends to some key parts of the compute infrastructure as well.
Second, we see a ton of Cloud + Kubernetes - teams who are still taking advantage of a lot of managed services and operational benefits from AWS/Azure/GCP/etc., but want to combine that with the container orchestration and rich API of Kubernetes. This means leaving stateful workloads in managed services like Aurora or Cloud SQL, delegating auth to managed IAM services, etc. Pulumi happens to be a really nice fit for doing these together.
And lastly, there's just a really active ecosystem that's building a lot of value around Kubernetes. Some of those pieces aren't yet fully mature, but directionally many teams are betting on this being where the next wave of mature cloud platform services will become available.
We are going w/ Fargate for now, but the similarity in syntax may help us make the leap to k8s when we're ready.
Any tool that helps build infrastructure as code is great news. I hear good things about pulumi.
In my case I have been moving our current infrastructure from AWS to GCP. After much evaluation and a small stint with Helm, we decided to go 100% terraform.
There is more to running a product than just deploying to kubernetes and terraform providers have given us the flexibility to chain all these resources together. For example, I can create a GCP service account (google cloud provider), save its credentials as a kubernetes secret (kubernetes provider), register a name in route 53 (aws provider) and deploy our product in kubernetes (kubernetes provider).
Having it all in one tool that has state is really powerful and more importantly follow the same patterns.
Obviously, I am a big fan of terraform
In my case I have been moving our current infrastructure from AWS to GCP. After much evaluation and a small stint with Helm, we decided to go 100% terraform.
There is more to running a product than just deploying to kubernetes and terraform providers have given us the flexibility to chain all these resources together. For example, I can create a GCP service account (google cloud provider), save its credentials as a kubernetes secret (kubernetes provider), register a name in route 53 (aws provider) and deploy our product in kubernetes (kubernetes provider).
Having it all in one tool that has state is really powerful and more importantly follow the same patterns.
Obviously, I am a big fan of terraform
Everything you describe is also straightforward with Pulumi using `pulumi-gcp` [1], `pulumi-aws` [2] and friends - Kubernetes is one of many libraries.
On top of this, you can build actual abstractions thanks to the configuration being written using a general purpose programming language - I don't think there is a first-party set of abstractions for GKE yet but there is for EKS [3].
Many of the most compelling examples deploy entire infrastructure stacks from "empty account" to multi-cloud Kubernetes.
Disclaimer: I have worked extensively on Terraform (both core and providers) at and after HC, and have also contributed to Pulumi since it was open sourced. Edited for formatting.
[1]: https://github.com/pulumi/pulumi-gcp
[2]: https://github.com/pulumi/pulumi-aws
[3]: https://github.com/pulumi/pulumi-eks
On top of this, you can build actual abstractions thanks to the configuration being written using a general purpose programming language - I don't think there is a first-party set of abstractions for GKE yet but there is for EKS [3].
Many of the most compelling examples deploy entire infrastructure stacks from "empty account" to multi-cloud Kubernetes.
Disclaimer: I have worked extensively on Terraform (both core and providers) at and after HC, and have also contributed to Pulumi since it was open sourced. Edited for formatting.
[1]: https://github.com/pulumi/pulumi-gcp
[2]: https://github.com/pulumi/pulumi-aws
[3]: https://github.com/pulumi/pulumi-eks
Great write up, the experience of managing k8s end to end with pulumi is so much better compared to terrafom.
One thing I'm murky on with Pulumi is around how to execute proper blue/green or canary deployments with a GitOps approach. Any insights?
For AWS users AWS CDK is a far better take on infrastructure as code - strongly typed TypeScript resources and higher level constructs that hide the underlying details of load balancers, IAM, etc. Pulumi was not as useful the last time I tried it - most resources were string key-value pairs with no typing or code completion. You're still referencing documentation to do anything and defining tons of low level resources to accomplish basic tasks. Might as well be writing raw Terraform or CloudFormation configs at that point, as they have more documentation available.
Hmm... just to set the record straight, Pulumi definitely does have TypeScript type definitions for all resource types, across all of our providers, and it has had them since our initial launch.
Happy to help you debug why you're not getting type hints, drop me a line at [email protected] or on the community slack: https://slack.pulumi.com/
Happy to help you debug why you're not getting type hints, drop me a line at [email protected] or on the community slack: https://slack.pulumi.com/
Something was wrong w/ your setup I think. All of their code has Typescript annotations, and their autocomplete w/ intellisense is remarkably useful.
Here are two simple database definitions:
Database in Pulumi:
Database in Pulumi:
const db = new gcp.sql.DatabaseInstance('app-db', {
databaseVersion: 'POSTGRES_11',
region: 'us-central1',
settings: {
tier: 'db-f1-micro',
diskType: 'PD_HDD',
},
});
const db = new rds.DatabaseInstance(this, 'AppDb', {
engine: rds.DatabaseInstanceEngine.POSTGRES,
engineVersion: '11.5',
instanceClass: ec2.InstanceType.of(
ec2.InstanceClass.STANDARD5,
ec2.InstanceSize.XLARGE
),
storageType: rds.StorageType.GP2
})
`DatabaseInstance` not only has TypeScript type definitions, it's actually written in pure TypeScript: https://github.com/pulumi/pulumi-gcp/blob/master/sdk/nodejs/...
Pulumi also does have a number of higher-level constructs, which you can learn about here: https://www.pulumi.com/docs/guides/crosswalk/aws/
Pulumi also does have a number of higher-level constructs, which you can learn about here: https://www.pulumi.com/docs/guides/crosswalk/aws/
Why is every single property of type string - to see what to put there I have to reference a comment in code? I shouldn't have to read comments to see what the magic string is for the f1-micro tier, or the hard disk type or the region. CDK has actual Enums for all of these things that work to make life easier. You're not making it any easier to code up a DatabaseInstance - its simply a bucket of keys and values where I'm looking up what those values should be.
Ah—unions of strings (e.g., `"foo" | "bar"`) effectively are enums in TypeScript and JavaScript. They are autocompleted and type-checked at compile time, so you should get exactly the same behavior. Even exhaustiveness checking. See docs[1] for details.
And, to the specific point: you absolutely should not have to look up the docs. These values definitely will autocomplete.
Anyway, aside from all that, unions of strings is "the idiomatic way" to do this sort of thing in TypeScript and JavaScript.
[1]: https://www.typescriptlang.org/docs/handbook/advanced-types....
And, to the specific point: you absolutely should not have to look up the docs. These values definitely will autocomplete.
Anyway, aside from all that, unions of strings is "the idiomatic way" to do this sort of thing in TypeScript and JavaScript.
[1]: https://www.typescriptlang.org/docs/handbook/advanced-types....
None of the properties in the DatabaseInstance construct example posted above are defined as unions of strings - the TypeScript documentation link you just posted is irrelevant.
This is what one of the property types actually look like:
This is what one of the property types actually look like:
/**
* The MySQL or PostgreSQL version to
* use. Can be `MYSQL_5_6`, `MYSQL_5_7`, `POSTGRES_9_6` or `POSTGRES_11` (beta) for second-generation
* instances, or `MYSQL_5_5` or `MYSQL_5_6` for first-generation instances.
* See [Second Generation Capabilities](https://cloud.google.com/sql/docs/1st-2nd-gen-differences)
* for more information.
*/
readonly databaseVersion?: pulumi.Input<string>;
Oh, I see. Some properties that are not as well-typed as they could be, that's true. We are working on that. But there are lots of properties that do have good typing. Instance types, for example.
I don't think large blocks of comments qualify as strong typing: https://github.com/pulumi/pulumi-gcp/blob/76df2e85214ad13465...
I don't think I understand, but am happy to help clarify. That is a TypeScript file where the type `DatabaseInstance` is defined. It's a completely normal TypeScript type—it will get picked up by editors, verified by the compiler, and so on. It is exactly the sort of type you'll find in CDK.
Parenthetical, one thing that didn't make it into this post that I'm particularly optimistic about is the `watch` command, which will continuously watch your code and deploy it to (say) Kubernetes. It looks like this[1]. This is very handy when you're trying to build a thing that has to integrate with a bunch of other services, but there are too many to develop locally on minikube. It basically lets you see instantly if something crashes, or whatever. In general I'm very optimistic about that sort of experience—another product in that space is Tilt[4] which I am personally a big fan of.
Another thing that gets kind of a passing mention that is cooler than it looks here is our streaming `query` stuff. By exposing Kubernetes `watch`es as streaming queries, it becomes very easy to write very cool apps like this one[2] , which helps you debug services. You can actually see a bug in the Deployment controller there—we create two `Deployment`s with the same labels, and deleting one causes the controller to delete _all_ the pods. This would generate a transient 500 for any request hitting that service. I think there are probably lots of bugs in Kubernetes like this, but people don't notice them because this level of visibility previously was quite hard. :) To get an idea of how hard that sort of app is to write, you can see the code we wrote for the prototype of this SDK here[3].
[1]: https://twitter.com/funcofjoe/status/1195411343966236677?s=2...
[2]: https://drive.google.com/open?id=1Q2f8CZhswMvTliaLlHZK4vItdM...
[3]: https://github.com/carbonql/carbon-ts/blob/master/examples/u...
[4]: https://tilt.dev/