Docker-compose.yml as a universal infrastructure interface(ergomake.dev)
ergomake.dev
Docker-compose.yml as a universal infrastructure interface
https://ergomake.dev/blog/docker-compose-as-a-universal-interface/
105 comments
I thought maybe I was just getting old and crusty, but this really does capture my primary complaint with K8s - everything that isn't standard, is so special and unrelated to the underlying technology that you need to relearn how K8's interacts with every layer of abstraction just to have half a chance of it working.
> everything that isn't standard
Could you give an example? I thought k8s and its YAML/practices was the standard?
Could you give an example? I thought k8s and its YAML/practices was the standard?
YAML practices… YAML is not a language, it’s a syntax tree. It pretends to be configuration and what it really becomes is some kind of a gimped template engine which people put layers and layers of code generators on top of to manage the complexity. XML was better and I’m willing to die on this hill.
He means part of the k8s standard. Once you get to the non-standard, cloud specific configuration, things go haywire.
For example try to configure a udp connection in k8s on aws.
For example try to configure a udp connection in k8s on aws.
oh, like azure k8s vs aws k8s vs gcp k8s? each has their own way to like... attach a volume for block storage or something? i thought that was all abstracted away/standardized?
> Once you get to the non-standard, cloud specific configuration
what's a common use case for when this is hit?
> Once you get to the non-standard, cloud specific configuration
what's a common use case for when this is hit?
Use the service resource in Kubernetes to order a load balancer in different clouds. You need different tags, they function and are configured differently, and you still need to learn how it works in your cloud.
Contrast this with the contract* K8 tries to sell you, which is "describe the intent and we'll figure it out for you" and you'll realize the cloud providers bunged up this whole thing.
That's not to say K8 is perfect, but if you think for even a second that wiring up resources outside your kubernetes cluster will be lower-friction, kubernetes is falling short of it's goals.
Contrast this with the contract* K8 tries to sell you, which is "describe the intent and we'll figure it out for you" and you'll realize the cloud providers bunged up this whole thing.
That's not to say K8 is perfect, but if you think for even a second that wiring up resources outside your kubernetes cluster will be lower-friction, kubernetes is falling short of it's goals.
I've always felt the standards are more about organization and common practices, whereas the implementation is still the wild west. Unfortunately in my experience, abstraction, even if structured, doesn't solve the main issue: it turns out the engineering operation of infra is still pretty hard. At least you know where to look to find the chaos though.
> whereas the implementation is still the wild west.
in my own experience, i would've thought "ok, let me aptinstall k8s and then apply some yaml to it so i can avoid going to the cloud for a massive k8s bill for almost no reason"
standing up kubeadm on a single master + worker node setup on a VM (like a droplet or EC2) is basically like, extremely non-standard + frowned upon
then, does anybody actually write pure k8s yaml? or do they write jsonnet/kustomize/helm charts? do they apply with kubectl or the helm CLI or the argocd CLI?
in my own experience, i would've thought "ok, let me aptinstall k8s and then apply some yaml to it so i can avoid going to the cloud for a massive k8s bill for almost no reason"
standing up kubeadm on a single master + worker node setup on a VM (like a droplet or EC2) is basically like, extremely non-standard + frowned upon
then, does anybody actually write pure k8s yaml? or do they write jsonnet/kustomize/helm charts? do they apply with kubectl or the helm CLI or the argocd CLI?
> then, does anybody actually write pure k8s yaml?
You can think of plain k8s yaml as a sort of machine code for k8s. You _can_ write yaml by hand, but it’s not generally how things are done outside of initially learning how k8s works.
The tools you’ve mentioned all make working with k8s resources a lot more pleasant. Something simple like parameterizing the name of a secret instead of hardcoding it in N different yaml files saves a lot of time if you ever need to refactor, and being able to provide those values via a “standard” CLI tool makes automated deployments a lot easier compared to hacking together some yq commands. helm in particular tracks the history of any charts you’ve installed, and has a very simple “helm rollback” command which can get you back to the last working version of your application if things end up going wrong.
The exact tools used by each team will differ; you could use any of them. But no one in my experience is writing plain yaml manifests and kubectl applying them in production contexts.
You can think of plain k8s yaml as a sort of machine code for k8s. You _can_ write yaml by hand, but it’s not generally how things are done outside of initially learning how k8s works.
The tools you’ve mentioned all make working with k8s resources a lot more pleasant. Something simple like parameterizing the name of a secret instead of hardcoding it in N different yaml files saves a lot of time if you ever need to refactor, and being able to provide those values via a “standard” CLI tool makes automated deployments a lot easier compared to hacking together some yq commands. helm in particular tracks the history of any charts you’ve installed, and has a very simple “helm rollback” command which can get you back to the last working version of your application if things end up going wrong.
The exact tools used by each team will differ; you could use any of them. But no one in my experience is writing plain yaml manifests and kubectl applying them in production contexts.
Yep. And the end result is a hodge podge of labels that make up an API that is much worse than the original interface, and requires significant amount of bounds/error checking in order to provide useful error messages until the codebase for it is no longer simple.
How would you make it less leaky then?
I just give developers access to k8s and teach them about it. That's really just my admission that the Kubernetes API isn't perfect, but it's better than many alternatives. If I were to build something to hide k8s from developers, eventually I'd have to add so many extra options that I'd probably end up re-inventing the Kubernetes API anyway.
That may just be unique to my job. Maybe another company could force all apps into nice boxes that prevents their bespoke platform API from leaking.
That may just be unique to my job. Maybe another company could force all apps into nice boxes that prevents their bespoke platform API from leaking.
Helm charts are pretty standard, git/kube-native, and lead much more nicely into the more sophisticated operator/controller model.
One part I haven't quite put a handle on is where/when to use CRDs in either helm or OLM. The community still seems split on how to use CRDs.
One part I haven't quite put a handle on is where/when to use CRDs in either helm or OLM. The community still seems split on how to use CRDs.
Honestly, I love compose files for local development. Launch and init a database, a DB admin tool, middleware, and a UI server in a single shot.
New team member: "How do I get the stack running?"
Existing member: "Clone this repo and run docker-compose up -d"
20 minutes and a cup of coffee later:
New member: "Cool! Ready to start!"
Does it look like our AWS stack? Only superficially. Which can be a good thing. The compose file shows you the broad strokes, not the extreme details. Compose files help with the basic mental model. Want to know more? Read through the CDK stack code.
But using a compose file for a cloud deployment interface? No thank you.
New team member: "How do I get the stack running?"
Existing member: "Clone this repo and run docker-compose up -d"
20 minutes and a cup of coffee later:
New member: "Cool! Ready to start!"
Does it look like our AWS stack? Only superficially. Which can be a good thing. The compose file shows you the broad strokes, not the extreme details. Compose files help with the basic mental model. Want to know more? Read through the CDK stack code.
But using a compose file for a cloud deployment interface? No thank you.
Yeah recently at work I created a compose.yml for our kafka pipeline. We have a collector and an enricher which produce to and consume from a kafka topic, respectively. The enricher then runs some functions on the data it consumes and produces to another topic. These services are all super microservicey, and so testing locally outside of docker was difficult (not to mention the PITA that is setting up a zookeeper broker on Windows)
The compose.yml runs everything on a docker network and uses redpanda for the kafka broker. Iteration became SUPER easy. Previously, we were just writing config files locally and uploading them to S3, then restarting the dev instance of whatever we were working on in k8s to re-ingest. This was, as I'm sure you can tell by now, tedious
The compose.yml runs everything on a docker network and uses redpanda for the kafka broker. Iteration became SUPER easy. Previously, we were just writing config files locally and uploading them to S3, then restarting the dev instance of whatever we were working on in k8s to re-ingest. This was, as I'm sure you can tell by now, tedious
In theory, it works everywhere. In practice, for any sufficiently large project, teams can't (either in terms in possibility or ability) size down the running services so that the whole stack runs locally.
And then starts the hunt for tradeoffs, with a million pitfalls.
I have seen the pattern fail too often to be only optimistic on the approach. I now tend to favor ephemeral deployments with only one locally running service connected to a bunch of remote services.
And then starts the hunt for tradeoffs, with a million pitfalls.
I have seen the pattern fail too often to be only optimistic on the approach. I now tend to favor ephemeral deployments with only one locally running service connected to a bunch of remote services.
News flash: 90% solution not suitable for folks with 10% of problem domains. Full story at 11 o'clock.
I gave up on compose since I couldnt figure out VSCode autocomplete for Python and JS, with the library files living within the container. Once I needed to run a command on a docker container (wasn't allowed to commit it) to make it work. To update pip/npm stuff, I do it outside of docker and then update the images.
Has anybody found a better way with just open source tools?
Has anybody found a better way with just open source tools?
I typically just map the source directories on the host. Good for --watch options. Works for me with Angular, Django, Svelte, and others.
If this article is any indication of the current state of the industry then the devops movement has busted. We're back to dev and ops.
If anything, situation is much worse than when we used to have "System/Network administrators", the cloud was a lie.
Don't know that I'd go so far as to blame the cloud (maybe the marketing materials but not the actual cloud) but I do share the sentiment.
Yeah. Everything then was a "layer 9" problem.
Little bit of self-promotion here, but seeing this comment is exactly what we felt at Kurtosis: https://docs.kurtosis.com/explanations/what-is-kurtosis
We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.
We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.
"Progress" is that now also ops gets to throw half-functional operators, underdocumented CRDs, and pluggable CRI/CNI/CSI all not quite configured to work together properly back over the wall to devs who have no hope to debug anything anymore!
100%. It's not hard to learn a bit of k8s yaml if you're an appdev FFS.
The problem is YAML, but it's too late to stop that momentum now.
Is anyone using Cue or dhall in a real production scenario? I'd be curious to hear reviews of those, as they look promising.
Is anyone using Cue or dhall in a real production scenario? I'd be curious to hear reviews of those, as they look promising.
I would be more interested in a stripped down JavaScript runtime for configuration. Some tooling already supports this paradigm. Embed something like QuickJS or a Lua interpreter and have the script export a config object. It feels too much like we want these plaintext config files to turn into fully-fledged scripts.
It's not stripped down, but Pulumi is a version of this. Regular code, and all the benefits and drawbacks of that approach, in your language(s) of choice to create infrastructure.
I haven't used Pulumi, but I have used Terraform (years ago). That is pretty much exactly what I am talking about, but instead of creating a new language, just use an already established language meant for scripting.
I've started using this setup for my Nim projects. I created a project that lets me use Nim instead of YAML for a statically checked configuration system. It can be run and output json, yaml, or run as is in Nim code. Its been much nicer than pure YAML! https://github.com/elcritch/ants/tree/main
Not satisfied I used a simple source filter to get a YAML compatible variant (WIP): https://github.com/elcritch/ants/blob/main/tests/cimport_exa...
Our concept of configs languages needs to keep evolving.
Not satisfied I used a simple source filter to get a YAML compatible variant (WIP): https://github.com/elcritch/ants/blob/main/tests/cimport_exa...
Our concept of configs languages needs to keep evolving.
We're trying to do something very similar to what the OP proposes at our current company (Kurtosis), and we explored YAML, Jsonnet, Dhall, CUE, and general-purpose languages and eventually settled on Starlark: https://docs.kurtosis.com/explanations/starlark#why-did-kurt...
BLUF: Infrastructure seems to live in this weird "configuration code" space between pure config (YAML) and actual code. You want reusable config, but you don't want to have to understand the complexities of a real codebase. Jsonnet, Dhall, and CUE all have the simplicity you want, but we found the DSL we'd need to write to be rough. Starlark hits this very nice balance of safety and ease-of-readability (and as a bonus is basically just Python). We've been very happy with it so far.
BLUF: Infrastructure seems to live in this weird "configuration code" space between pure config (YAML) and actual code. You want reusable config, but you don't want to have to understand the complexities of a real codebase. Jsonnet, Dhall, and CUE all have the simplicity you want, but we found the DSL we'd need to write to be rough. Starlark hits this very nice balance of safety and ease-of-readability (and as a bonus is basically just Python). We've been very happy with it so far.
With Lua, you don't even need to export a config object, the Lua script is the config object. I do think there is some promise to this - see Neovim for example.
There are a lot of downsides though, because now you've got a leaky config language instead of a leaky config interpreter.
There are a lot of downsides though, because now you've got a leaky config language instead of a leaky config interpreter.
Like a CDK (cloud development kit)? Those tools already exist - like AWS CDK, Pulumi, CDK-TF - but they are hardly stripped down. AWS CDK for instance transpiles everything down to Typescript and those layers of abstractions make debugging painful.
Yes kind of, but instead of creating a new language (like HCL), just use a language people already know, but strip the runtime to a minimal set of features.
Lua has been created as a config language. I love to see the world coming full circle.
Born in 1996, so I have no clue on the history of Lua, but good to know I am following in the footsteps of gods! :)
I share your, and all Norwegians', distaste for YAML. One saving grace I have found is that YAML is a superset of JSON. JSON isn't great, but it's certainly better than YAML.
I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling.
Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.
I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling.
Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.
Upon further reflection, it's not actually YAML I have a problem with, so much as the (lack of) composability of YAML. As soon as you're dealing with multiple files and want to "include" something in one way or another, it seems like every DevOps software has its own semi-standard way of allowing that.
As annoying as the parsing quirks of YAML are, they don't bite me that often. What bites me is the proliferation of YAML files in an effort to DRY my config files with whatever primitive imitation of a templating language I can use as the common denominator of the various tools consuming them.
I just checked our monorepo, and we have ~93 YAML files related in some way to DevOps.
As annoying as the parsing quirks of YAML are, they don't bite me that often. What bites me is the proliferation of YAML files in an effort to DRY my config files with whatever primitive imitation of a templating language I can use as the common denominator of the various tools consuming them.
I just checked our monorepo, and we have ~93 YAML files related in some way to DevOps.
My main issue is that YAML is used as a [bad] programming language. To make matters worse, in the domains where it's typically used (DevOps) the "developer inner loop" is usually minutes long (e.g. GitHub actions).
Having to actually do the dangerous things that you typically do with YAML in order to test the YAML is insane!
Having to actually do the dangerous things that you typically do with YAML in order to test the YAML is insane!
> JSON isn't great, but it's certainly better than YAML.
This is the first time I've ever seen someone say this. Why do you think it's better?
This is the first time I've ever seen someone say this. Why do you think it's better?
As a human, I'd rather write YAML than JSON. But if I'm writing code, I'd rather write code that parses JSON.
The problem is that the two often conflict with each other, because I'm writing code to parse config written by a human.
My rule of thumb is that any autogenerated communication between services can be done over JSON (assuming some more efficient serialization format isn't available), but if a human needs to maintain it (and check it into a repo with diffs) then it should be YAML.
The problem is that the two often conflict with each other, because I'm writing code to parse config written by a human.
My rule of thumb is that any autogenerated communication between services can be done over JSON (assuming some more efficient serialization format isn't available), but if a human needs to maintain it (and check it into a repo with diffs) then it should be YAML.
I agree with this point. I would also say that while HCL gets a lot of hate from the developers it really is an easy language to learn and use. Yes it's another language but the language itself I found to be very natural for declaring desired state. Last year I spent two half-days training my team on it (I have a lot of experience with it) and they picked it up rather well. Previous teams I worked with had similar results. It may not be their favorite language, but it's not as bad as cloudformation or a heavy, cumbersome CDK.
Not OP but https://noyaml.com/ summarizes most of it.
TL;DR: It has way too many opinionated things builtin, too many ways to do the same thing.
As a additional note, unlike JSON that has the awesome tooling available (`jq` is reasonably widespread, you can count on python/perl/node being almost everywhere), it has nothing that (because of the multiple opinionated ways) will reliably read your files or not mangle your desired output in some way or another -- except if you limit your YAML to just a embedded JSON.
TL;DR: It has way too many opinionated things builtin, too many ways to do the same thing.
As a additional note, unlike JSON that has the awesome tooling available (`jq` is reasonably widespread, you can count on python/perl/node being almost everywhere), it has nothing that (because of the multiple opinionated ways) will reliably read your files or not mangle your desired output in some way or another -- except if you limit your YAML to just a embedded JSON.
There actually is a yq [0] which I've used effectively for basic parsing and manipulation tasks on the command line. But I've no idea how well it handles YAML corner cases or how many advanced features it's missing compared to jq.
[0] https://github.com/mikefarah/yq
[0] https://github.com/mikefarah/yq
We do - https://foundation.namespace.so/docs. Some feature of "Namespace" are kind of similar to Ergomake, but we use own format with CUE instead of docker-compose. Here is comparison to docker-compose - https://namespace.so/blog/from-docker-compose-to-namespace.
CUE [1] is kinda-sorta "transpiled" into by Pulumi's language support facility [2]. In general the greater the domain space being expressed the more I'm leery of this kind of approach; leaky abstractions creep in faster the larger the volume of concepts embraced. Infrastructure is a pretty big concepts volume. Whatever the source system (Pulumi in this case) does not provide a hook for that the target system (CUE in this case) supports for example, hacks tend to start to emerge. If there was both this layer of abstraction and a "break glass" layer available down to something analogous to an Abstract Syntax Tree level of primitives, then the target systems at least have hooks to compose new abstractions via some kind of DSL into the source system perhaps. There be dragons there too, though; no panaceas, but at least it is better than all sorts of hacks.
I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE.
Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale.
One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code.
[1] https://cuelang.org/docs/about/
[2] https://www.pulumi.com/blog/extending-pulumi-languages-with-...
[3] https://dhall-lang.org/
[4] https://news.ycombinator.com/item?id=32102203
I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE.
Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale.
One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code.
[1] https://cuelang.org/docs/about/
[2] https://www.pulumi.com/blog/extending-pulumi-languages-with-...
[3] https://dhall-lang.org/
[4] https://news.ycombinator.com/item?id=32102203
I just want to point out that already exist a tool that convert a docker-compose.yml file to several k8s manifests: https://kompose.io . You have to adjust some details because is not perfect, obviously.
I think the direction of podman running k8s yaml directly makes more sense here.
Why? What's better with podman in this context? Genuinely curious since I can't think of any real advantage.
Podman doesn't support Docker compose. Instead, it uses the same YAML files that are used by Kubernetes.
The only advantage I see is it dosen't require a kube-apiserver. Other than that it feels like just running a quick throwaway cluster with kind and writing actual manifests is more accurate and portable.
Makes a great deal of sense. It was basically the original idea (when compose was fig). Oftentimes though, people cling desperately and aggressively to existing complex systems...
Having a complex system under the hood is not inherently bad, so long as you have non-leaky abstractions that allow that complexity to be reasoned about (pods, services), sane guardrails that let you know when you're throwing away key guarantees (privileged containers), and reasonable defaults for portions of the system that you don't yet need or care to engage with (resource limits, among others).
Kubernetes may be a system that invites overcomplication, but it in no way requires it— in a world where anyone can run k3s or microk8s on their own laptop and a Kubernetes hello-world is like 10 lines of yaml, I don't think it's a stretch to posit that that's the interface that makes sense to use up and down the stack.
Kubernetes may be a system that invites overcomplication, but it in no way requires it— in a world where anyone can run k3s or microk8s on their own laptop and a Kubernetes hello-world is like 10 lines of yaml, I don't think it's a stretch to posit that that's the interface that makes sense to use up and down the stack.
Intriguing idea, but I don't see how a docker compose manifest can be translated to "just work" in a kubernetes cluster without making a long list of non-intuitive implementation decisions. And if devs are supposed to now learn all the intricacies of compose when using it in an kuberentes context then it isn't very different from any other custom kubernetes config abstraction. Worse in fact due to all the confusion it will bring with it.
"Kubernetes manifests are an unnecessarily detailed abstraction for most software developers. Developers don't need as much control. Therefore, they need higher-level abstractions, which hide more detail."
Agreed... Though I think there is a different solution. Instead of coming up with all these hardcoded formats and standards (docker-compose, k8s, etc, helm, ansible) which essentially are just 'functions' that map X to Y.
Instead of defining 'docker-compose' files, just define the containers itself. And if required, map that to a docker-compose, or k8s manifest or whatever. Remove all these unnecessary layers that aren't really that helpful - because each standard or implementation has it's own parser, error handling etc - why bother with all this?
And just use something like Nix for this. Or just define it as a common python value. Otherwise we end up building all sorts of templating and ugly programming-in-yaml. You could take a look at https://docs.hercules-ci.com/arion/#_arion_compose_nix to see what this might look like - uses Nix to do something like docker-compose.
Agreed... Though I think there is a different solution. Instead of coming up with all these hardcoded formats and standards (docker-compose, k8s, etc, helm, ansible) which essentially are just 'functions' that map X to Y.
Instead of defining 'docker-compose' files, just define the containers itself. And if required, map that to a docker-compose, or k8s manifest or whatever. Remove all these unnecessary layers that aren't really that helpful - because each standard or implementation has it's own parser, error handling etc - why bother with all this?
And just use something like Nix for this. Or just define it as a common python value. Otherwise we end up building all sorts of templating and ugly programming-in-yaml. You could take a look at https://docs.hercules-ci.com/arion/#_arion_compose_nix to see what this might look like - uses Nix to do something like docker-compose.
If you say interface there needs to be an implementation; and docker already does that with contexts:
Of course each implementation is opinionated, AWS for example is deploying to ECS with ALBs.
And it is all good for a list of services, but when you add configuration it becomes a mess:
So to keep the interface clean you would need to split it up in a Service model and a Deployment model that exposes the needed configuration for the target platform plus the service configuration for the specific environment.
Or something...
Thinking of something generic that works in all cases is hard and complex...and opinionated hence the many platforms, I think.
I posted this a while ago https://news.ycombinator.com/item?id=34225669
https://docs.docker.com/engine/context/working-with-contexts/Of course each implementation is opinionated, AWS for example is deploying to ECS with ALBs.
And it is all good for a list of services, but when you add configuration it becomes a mess:
secrets:
foo:
name: "arn:aws:secretsmanager:eu-west-3:1234:secret:foo-ABC123"
x-aws-keys:
- "bar"
So to keep the interface clean you would need to split it up in a Service model and a Deployment model that exposes the needed configuration for the target platform plus the service configuration for the specific environment.
Or something...
Thinking of something generic that works in all cases is hard and complex...and opinionated hence the many platforms, I think.
I posted this a while ago https://news.ycombinator.com/item?id=34225669
Cloud technology is so bad. Just run your programs as normal processes. No bloated images and runtimes, no slow virtual bridging layers, no barriers preventing you to access the hardware or communicating with the user.
The hardware has finite resources, and the best way to share them is to run things natively on the operating system.
It's also all simple and straightforward, everyone understands ssh and simple Linux sysadmin.
The hardware has finite resources, and the best way to share them is to run things natively on the operating system.
It's also all simple and straightforward, everyone understands ssh and simple Linux sysadmin.
I disagree. While there are bloated images and runtimes, at least they are separate.
Unfortunately most applications are developed in a way that they believe they are the only one on the system. Ruby gets called, and they expect 2.7.
And there is another one which is using Ruby 3.2. And it's incompatible with 2.7.
Now what? Do you want to spend time
- maintaining 2 versions of ruby on a system
- keeping them separate
- ensuring they don't mix up
OR
Shove them in a container, run them in parallel and all they surface is a port.
I'll take the container.
Unfortunately most applications are developed in a way that they believe they are the only one on the system. Ruby gets called, and they expect 2.7.
And there is another one which is using Ruby 3.2. And it's incompatible with 2.7.
Now what? Do you want to spend time
- maintaining 2 versions of ruby on a system
- keeping them separate
- ensuring they don't mix up
OR
Shove them in a container, run them in parallel and all they surface is a port.
I'll take the container.
Dockerized delivery starts easy unless you realize you need a new container for every small thing you need to run, and you end up with a pile of containers a short while later.
Docker containers, in my experience, have not figured out auto updates yet. You may be running containers with critical system vulnerabilities and is not bothered to check.
So the basic premise of Dockerization 1.0, "I have exactly one thing which I will put in a single container with all of its dependencies, reproducibly built" is wrong.
Docker containers, in my experience, have not figured out auto updates yet. You may be running containers with critical system vulnerabilities and is not bothered to check.
So the basic premise of Dockerization 1.0, "I have exactly one thing which I will put in a single container with all of its dependencies, reproducibly built" is wrong.
But the update problem is also present on direct installs. It's is less bad on Docker.
Example: 2 containers which rely ffmpeg, and there's a vulnerability in ffmpeg.
But container A can't update cause ffmpeg's update breaks something else...
At least with Docker you can update container B, and maybe isolate A.
On a direct install ffmpeg is a shared component. So now both apps are locked to a vulnerable version.
Not to mention that shared components are often outdated for the sake of stability.
So App A is restricted to the same version as App B, even though App A would like version 9, and B is on 8.
Example: 2 containers which rely ffmpeg, and there's a vulnerability in ffmpeg.
But container A can't update cause ffmpeg's update breaks something else...
At least with Docker you can update container B, and maybe isolate A.
On a direct install ffmpeg is a shared component. So now both apps are locked to a vulnerable version.
Not to mention that shared components are often outdated for the sake of stability.
So App A is restricted to the same version as App B, even though App A would like version 9, and B is on 8.
Direct installs have auto update, though, so they may update A and B separately and will often update them along with bundled dependencies.
In the Docker scenario, it's quite often that neither A nor B will be updated for a long time, since there's no mechanism to do so.
In the Docker scenario, it's quite often that neither A nor B will be updated for a long time, since there's no mechanism to do so.
Well you need to do that anyway if you are indeed building those applications.
[deleted]
I think what you really want to say is your last sentence:
> It's also all simple and straightforward
What you're really trying to say is that cloud technologies are complicated for you.
> It's also all simple and straightforward
What you're really trying to say is that cloud technologies are complicated for you.
You must be really young if you do not understand that aversion to complexity is likely a single best trait a developer may have.
Spoken like someone who's never done Linux sysadmin at scale.
I've done hundreds of nodes, all managed to a very low level (I've been to the datacenter and optimized placement of servers within the cabinet to enable shorter cables to the patch panel) which is way beyond what most companies need.
In general it's better to have fewer and more powerful and reliable nodes that you own than to have many disposable small ones, as shared instances just don't work well and aren't cost-effective.
Cloud in general just makes everything more expensive while removing control. The costs on AWS if paying for three years upfront is the same as buying the hardware outright, except you have little say in what the hardware is and how the infrastructure is set up.
In general it's better to have fewer and more powerful and reliable nodes that you own than to have many disposable small ones, as shared instances just don't work well and aren't cost-effective.
Cloud in general just makes everything more expensive while removing control. The costs on AWS if paying for three years upfront is the same as buying the hardware outright, except you have little say in what the hardware is and how the infrastructure is set up.
choosing serverplacement to optimize for cable length does not sound very efficient. usually you get the cables in the length you need and place servers to solve other requirements (heat, direct connections between devices, etc)
> as shared instances just don't work well and aren't cost-effective
of course ymmv from mine, but this statement being true or false depends on too many things to be of any value
> as shared instances just don't work well and aren't cost-effective
of course ymmv from mine, but this statement being true or false depends on too many things to be of any value
hundreds! my stars! so many!
> as shared instances just don't work well and aren't cost-effective.
goodness you are outing yourself here aren't you
> as shared instances just don't work well and aren't cost-effective.
goodness you are outing yourself here aren't you
If anything it's the opposite.
Docker compose universal? Hello kafka-1, kafka-2, kafka-3 and various compose files depending on the cluster size. Thanks, compose is nice for local dev and as a „this is a minimum of a reference to make the stack talk together”, but no. Compose is missing so much. Services, rollouts, instance count, secrets.
At my current company, we use Docker Swarm with multiple compose files to run all of our services. It's incredibly easy to scale up or down services, and if you need to get more intricate with service details/configs, it allows for that. Love it.
If you're going to run in prod in Kubernetes then you need to live with the idea of applying Kubernetes manifests. So, live with it. Docker Desktop made that easy, and some of the alternatives like Rancher and Podman also make it easy. Plus, how are you going to test your ConfigMaps and Roles and RoleBindings if you aren't actually running in Kubernetes locally? How do you know if a change to your Helm chart is going to work?
If you are not running in prod in Kubernetes, but you still run in prod with Docker, what the heck are you doing? ECS? You still need a control plane to deploy the images and Compose isn't really up to that task for production workloads.
If you are not running in prod in Kubernetes, but you still run in prod with Docker, what the heck are you doing? ECS? You still need a control plane to deploy the images and Compose isn't really up to that task for production workloads.
Uhh hellooo Terraform? I was just discussing docker-compose today, Terraform has the advantage of State Management..
Terraform is just as easy, but probably a combination of the two docker-compose and terraform are much more effective.. Universal Infrastructure interface is the name of Terraform's game.. They already done what you hoped docker-compose did.
Terraform deserves serious kudos for interoperation between all the clouds, docker, vmware, and literally tons of API modules.. Their Infrastructure Modules are nothing to laugh about in a JSON compatible example.. Serious, they even give Ansible a run for their money when Red Hat was all the blaze.
Terraform is just as easy, but probably a combination of the two docker-compose and terraform are much more effective.. Universal Infrastructure interface is the name of Terraform's game.. They already done what you hoped docker-compose did.
Terraform deserves serious kudos for interoperation between all the clouds, docker, vmware, and literally tons of API modules.. Their Infrastructure Modules are nothing to laugh about in a JSON compatible example.. Serious, they even give Ansible a run for their money when Red Hat was all the blaze.
After pursuing a very simmilar idea several years ago, even with simple apps I quickly ran into significant complexity, where important (and very consequential) decisions had to made for the developer (aka a "strongly opinionated" framework), particularly around questions of horizontal scalability. That also locked you into a specific cloud/platform/pattern/etc that often just led to teams needing to "eject" back to enough real k8s features that the abstraction became a real source of extra complexity. We ended up abandoning the project.
I do wish them luck though.
I do wish them luck though.
I like the idea of giving developers a template format they could follow, and then those templates would be transpiled into Kubernetes manifests by the platform.
I would not use docker-compose.yaml for that, though.
I would not use docker-compose.yaml for that, though.
Shameless plug for my own attempt at doing something like this, but creating a higher level abstraction on top of helm and terraform for composing different services and projects to quickly get projects up and running on Kubernetes with sensible defaults and the ability to quickly break out into configuring the helm charts if you need to handle something special.
https://github.com/TorbFoundry/torb
https://github.com/TorbFoundry/torb
Interesting idea. I wonder how some examples would look for teams which have a docker compose file which don't include some services - because they are mocked for tests or expected to be run separately etc. Would it still be possible to write a k8s operator to transform the docker compose file perfectly to a k8s manifest or would it be lacking key information?
Helm did something like in the earlier versions. Helm 3 got rid of it to the great cheering of many. It didn't ingest Docker Compose files, but it is the same architectural pattern.
https://www.techtarget.com/searchitoperations/news/252474360...
On the other hand, you have something like CloudFoundery on top of K8S:
https://github.com/cloudfoundry/cf-for-k8s
https://www.techtarget.com/searchitoperations/news/252474360...
On the other hand, you have something like CloudFoundery on top of K8S:
https://github.com/cloudfoundry/cf-for-k8s
One criticism I have regarding this article is the intended audience surely does not need a detailed explanation of what an abstraction is (complete with dog, golden retriever and chihuahua examples).
Kind of a weird set of examples to describe the concept of an abstraction at all... I hate contrived examples. It was my least favorite part of all of my coursework, GenEds included.
If you have application developers that can't grok what a deployment is and what a service is, fire them. Just fire them. The "run a container" part of k8s is a solved problem that we never need to revisit. There are much more interesting areas to tease apart, like networking and service mesh, and there's some room for idiot proofing those for developers (OpenShift has some nice tooling in this area, among others).
This is exactly the attitude that I've been complaining in other threads in HN.
What else should the application developer know? How to fill taxes for the company?
I hope the people who manage your k8s infrastructure know how to develop the application they are deploying. If not, just fire your k8s teams. You cannot became k8s person without knowing how to develop the application, there are more interesting areas...
That sounds dumb, isn't it?
I hope the people who manage your k8s infrastructure know how to develop the application they are deploying. If not, just fire your k8s teams. You cannot became k8s person without knowing how to develop the application, there are more interesting areas...
That sounds dumb, isn't it?
Writing a k8s deployment is like writing fizzbuzz. If your k8s team can't write fizzbuzz, yes, fire them.
If you have application developers that can't grok what a deployment is and what a service is, fire them. Just fire them.
No - fire their managers, and their managers as needed.
By definition, that's where the responsibility lies. Not with the leaf-node developers. Who are doing exactly as they've been trained and/or vetted by their managers.
No - fire their managers, and their managers as needed.
By definition, that's where the responsibility lies. Not with the leaf-node developers. Who are doing exactly as they've been trained and/or vetted by their managers.
Are you aware at some point in your career you didn't know what those things were either?
At some point we also didn't know what a terminal was, how to type, or how to read. That doesn't mean they aren't baseline expectations for a software developer.
I'd bet most developers on this planet have no idea how to maintain and operate Kubernetes.
Nobody's asking them to.
A Deployment, a CronJob, a Secret, etc.: those are the interfaces with Kubernetes. By extension, it is the interface with which you interact with the service (a cluster of machines) that your infra team maintains. (And they are the ones that need to understand how to maintain & operate Kubernetes.)
App devs need to learn the structure of those YAML interfaces. They are incredibly well documented, both in reference material and in example material.
And yet I find the same as the people above you in this thread: app eng have an absolute aversion to wanting to specify, in programmatic form, how to run their app. Yet, by definition, they're the only ones that know how to do that.
(That said, I have been known to reverse engineer applications enough to understand how they intend to be run, and then write the corresponding k8s YAML. This is an organizational anti-pattern, though.)
The YAML isn't complicated. What's complicated is that the set of data that comprises a process is inherently complicated: what user does it run as? what files does it need? what command to execute it? what env vars? etc. k8s's YAML does rather little abstracting over these.
(I sort of disagree with the "fire them" — rather, you need to ensure your onboarding trains them. But a lot of employers these days seem to do zip for training.)
A Deployment, a CronJob, a Secret, etc.: those are the interfaces with Kubernetes. By extension, it is the interface with which you interact with the service (a cluster of machines) that your infra team maintains. (And they are the ones that need to understand how to maintain & operate Kubernetes.)
App devs need to learn the structure of those YAML interfaces. They are incredibly well documented, both in reference material and in example material.
And yet I find the same as the people above you in this thread: app eng have an absolute aversion to wanting to specify, in programmatic form, how to run their app. Yet, by definition, they're the only ones that know how to do that.
(That said, I have been known to reverse engineer applications enough to understand how they intend to be run, and then write the corresponding k8s YAML. This is an organizational anti-pattern, though.)
The YAML isn't complicated. What's complicated is that the set of data that comprises a process is inherently complicated: what user does it run as? what files does it need? what command to execute it? what env vars? etc. k8s's YAML does rather little abstracting over these.
(I sort of disagree with the "fire them" — rather, you need to ensure your onboarding trains them. But a lot of employers these days seem to do zip for training.)
What the actual problem is, in my considerable experience building teams that encountered this problem, is that they lack basic understanding of OS concepts.
K8s includes a lot of Unix. If you don't know what a user or perm is, or a mount point, you just blame k8s.
K8s includes a lot of Unix. If you don't know what a user or perm is, or a mount point, you just blame k8s.
How are the developers writing Dockerfiles if they don't understand them? A deployment is a declarative way to run a (group of) container(s). There is no extra abstraction required of the developers in the trivial usecase which is what the article is suggesting is too difficult.
> A deployment is a declarative way to run a (group of) container(s). There is no extra abstraction required of the developers
You have skipped over the Pod and ReplicaSet abstractions, and the Pod part is definitely not ignorable even in the most trivial case. The container part is probably more ignorable than the pod part, since pods are what actually gets exposed for most operations.
You have skipped over the Pod and ReplicaSet abstractions, and the Pod part is definitely not ignorable even in the most trivial case. The container part is probably more ignorable than the pod part, since pods are what actually gets exposed for most operations.
> Nobody's asking them to.
They might very soon. https://archive.ph/YWp4O
They might very soon. https://archive.ph/YWp4O
grok means 'to come to understand' as in to learn.
So if your developers can't learn this stuff...
So if your developers can't learn this stuff...
Perhaps they were fired for it and hold some fetishised, eternal grudge.
I have to chuckle that they're still referring to docker-compose.yml as opposed to compose.yaml, and docker-compose as opposed to docker compose. The legacy files and commands will be EOL in June.
https://docs.docker.com/compose/compose-file/
https://docs.docker.com/compose/compose-file/
This article is not even wrong.
[deleted]
Even their example of using Ingresses is already leaky because there is no similar concept of Ingresses in docker-compose, so they're hacking it with labels. What if I need to set a higher request timeout for my app? I have to set a docker-compose label that converts into an Ingress annotation (the leaky part of the Ingress abstraction)? And this is somehow less cognitive load?
Like all similar attempts to simplify deploying apps, it only works if all your apps fit in a nice box. Anything slightly non-standard (like needing different classes of storage volumes) will expose the leakiness. Even Kubernetes kinda still expects all your apps to be stateless web services and will fight you otherwise.