Connecting Kubernetes services with linkerd(lwn.net)
lwn.net
Connecting Kubernetes services with linkerd
https://lwn.net/SubscriberLink/719282/d8444c95089cd014/
48 comments
> linkerd is written in Scala [1] and linkerd is written in Rust [2]
Though your links are correct your statement isn't. Linkerd is written in Scala, but the Rust bit is linkerd-tcp, a TCP loadbalancer, which linkerd (the Scala one) doesn't handle as that covers HTTP and gRPC with optionally encrypting connections between linkers using TLS.
> If he is optimizing for other people’s contributions, is this really the best way to go?
But is he optimising for other people's contributions? Using a C/Rust for something like a TCP proxy makes a lot more sense to me than having a JVM do that. And if he perceives Scala as "not for everyone", maybe adding Rust means he might get more contributions to those components?
I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two. By spinning it out as a standalone component that acts as a proxy, like linkerd is doing, Scala doesn't get in the way and what Finagle offers can be used with a mesh of microservices written in a myriad of languages.
Though your links are correct your statement isn't. Linkerd is written in Scala, but the Rust bit is linkerd-tcp, a TCP loadbalancer, which linkerd (the Scala one) doesn't handle as that covers HTTP and gRPC with optionally encrypting connections between linkers using TLS.
> If he is optimizing for other people’s contributions, is this really the best way to go?
But is he optimising for other people's contributions? Using a C/Rust for something like a TCP proxy makes a lot more sense to me than having a JVM do that. And if he perceives Scala as "not for everyone", maybe adding Rust means he might get more contributions to those components?
I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two. By spinning it out as a standalone component that acts as a proxy, like linkerd is doing, Scala doesn't get in the way and what Finagle offers can be used with a mesh of microservices written in a myriad of languages.
Sorry, not typing -tcp on linkerd-tcp was completely a typo on my end.
> I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two.
Thanks, this is the answer I have been looking for.
> I think that statement actually refers to Finagle itself being written in Scala, so if you want to use it you have to at least hoist your own apps on the JVM and integrate Finagle in them. Finagle is essentially client-side and requires integration into your app tightly coupling the two.
Thanks, this is the answer I have been looking for.
Yep! That is exactly what I meant by that ;)
Maybe it means at runtime, as opposed to at contribution-time?
Lyft has a tech similar to linkerd called Envoy: https://github.com/lyft/envoy. It is a bit lower level than linkerd (based on APIs rather than already integrated with common services) and written in C++.
I'm working on an adapter which should offer a better integration with Kubernetes. The end goal is to have envoy be able to replace our Ingress controllers. Not having to run the JVM on every machine is also nice.
I'm working on an adapter which should offer a better integration with Kubernetes. The end goal is to have envoy be able to replace our Ingress controllers. Not having to run the JVM on every machine is also nice.
> Not having to run the JVM on every machine is also nice.
I am definitely going to take a look at Envoy. Thanks for referencing it!
Folks at Lyft/IBM/Google are building a service mesh partly based on Envoy over here: https://istio.io
The content of the article does not have anything to do with the title. There is not mention of how linkerd and k8s work together.
> Facilities that linkerd provides to assist with this include service discovery, load balancing, encryption, tracing and logging, handling retries, expiration and timeouts, back-offs, dynamic routing, and metrics.
Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly.
Also, regarding https://linkerd.io/, what's with landing pages having almost no useful information on them. "linker∙d is a transparent proxy that adds service discovery, routing, failure handling, and visibility to modern software applications" what's that even mean?
> Fast, lightweight, and performant > Handles tens of thousands of requests per second per instance with minimal latency overhead. Scales horizontally with ease.
Well, since you're calling yourself "transparent" I would hope it'd be fast!
> Any language, any environment > Runs as a transparent proxy alongside existing applications, integrates with existing infrastructure.
So, it's just proxying and retrying things and doing tls for me? Why isn't this better handled in the library I'm using to make the call in the first place?
> Latency-aware load-balancing > Balances request traffic using real-time performance, reducing tail latencies across your application.
Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
> Runtime traffic routing > Provides dynamic, scoped, logical routing rules, enabling blue-green deployments, staging, canarying, failover, and more.
OK, some of that does sound useful, but some of it still feels like something the caller should handle.
> Drop-in service discovery > Integrates with most service discovery systems, decoupling applications from specific implementations.
What's this mean? Why not just use SRV records?
> Production-tested and proven at scale > Powers the production infrastructure of banks, artificial intelligence companies, social networks, government labs, and more.
I know why this is here, but it feels like _everyone_ claims the same thing, so, to me, it just becomes noise.
> "linkerd allows you to drop in a transparent layer of application resilience and provides the operational affordances critical for modern, cloud native environments." > —Oliver Gould, CTO, Buoyant
This sounds doesn't make sense to me, you can't "drop in" resiliency. It needs to be built. The whole thing just feels like it's all marketing and doesn't tell me exactly what problems it's solving for me as a developer.
------------
Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly.
Also, regarding https://linkerd.io/, what's with landing pages having almost no useful information on them. "linker∙d is a transparent proxy that adds service discovery, routing, failure handling, and visibility to modern software applications" what's that even mean?
> Fast, lightweight, and performant > Handles tens of thousands of requests per second per instance with minimal latency overhead. Scales horizontally with ease.
Well, since you're calling yourself "transparent" I would hope it'd be fast!
> Any language, any environment > Runs as a transparent proxy alongside existing applications, integrates with existing infrastructure.
So, it's just proxying and retrying things and doing tls for me? Why isn't this better handled in the library I'm using to make the call in the first place?
> Latency-aware load-balancing > Balances request traffic using real-time performance, reducing tail latencies across your application.
Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
> Runtime traffic routing > Provides dynamic, scoped, logical routing rules, enabling blue-green deployments, staging, canarying, failover, and more.
OK, some of that does sound useful, but some of it still feels like something the caller should handle.
> Drop-in service discovery > Integrates with most service discovery systems, decoupling applications from specific implementations.
What's this mean? Why not just use SRV records?
> Production-tested and proven at scale > Powers the production infrastructure of banks, artificial intelligence companies, social networks, government labs, and more.
I know why this is here, but it feels like _everyone_ claims the same thing, so, to me, it just becomes noise.
> "linkerd allows you to drop in a transparent layer of application resilience and provides the operational affordances critical for modern, cloud native environments." > —Oliver Gould, CTO, Buoyant
This sounds doesn't make sense to me, you can't "drop in" resiliency. It needs to be built. The whole thing just feels like it's all marketing and doesn't tell me exactly what problems it's solving for me as a developer.
------------
Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
> > Latency-aware load-balancing > Balances request traffic using real-time performance, reducing tail latencies across your application.
> Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.
> Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
So? If you're happy with your solution, keep it! But maybe others haven't solved all these problems, or don't like your approach to how you've solved them for other reasons or don't even want to spend time solving these problems in the first place. For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
> Sounds like it requires a lot of cross-talk on my network, eating up bandwidth. That doesn't sound lightweight.
Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.
> Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
So? If you're happy with your solution, keep it! But maybe others haven't solved all these problems, or don't like your approach to how you've solved them for other reasons or don't even want to spend time solving these problems in the first place. For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
> Why does that require eating bandwidth? It's a proxy. It can derive statistics from past and current connections and make decisions based on that data for new ones without talking to anyone else.
Wouldn't those stats get out of date really quickly if you're not constantly talking to that machine?
> For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
But now you have a new component that you need to test and get to know. You didn't get rid of those 10 things, you just compartmentalized them in something else that needs to be maintained.
> Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
Very true, I just don't see the particular advantages to using more basic and well-known tools like DNS and TLS.
Wouldn't those stats get out of date really quickly if you're not constantly talking to that machine?
> For many a component like this means it's 10 less things to deal with and more time to focus on building features that'll matter to their end users.
But now you have a new component that you need to test and get to know. You didn't get rid of those 10 things, you just compartmentalized them in something else that needs to be maintained.
> Besides, just because it can already be solved doesn't mean it's not worth trying out other approaches.
Very true, I just don't see the particular advantages to using more basic and well-known tools like DNS and TLS.
The whole point of Linkerd is moving logic out of your app. Linkerd is like a library but, instead of being in-process, it inserts itself between the app and whatever the app needs to talk to. The app just dials http://foo.svc/, and Linkerd sends the connection to the right place; it doesn't need to know where foo.svc is.
Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices. Linkerd replaces this with something that is explicitly designed to deal with inter-app communication. (I believe one way to run Linkerd is as a small sidecar app inside the same Docker container the man app, as opposed to a big central server, which is also a neat way of doing it.)
Re "why not a library" — sure, you can build all sorts of discovery, distribution and resilience into the client itself. But that consolidates all of this intelligence into just that client. What if you use multiple languages? What about third-party client libs (e.g. Memcached, Redis, PostgreSQL) that you don't control and that you don't want to reimplement? You'll find yourself reimplementing the same thing over and over.
> you can't "drop in" resiliency
I recommend reading the documentation, because drop-in resilience is really the whole point. The guys who built Linkerd previously developed Finagle [1], which is Twitter's RPC library, and when they talk about network resilience, they actually know what they're talking about [2].
[1] https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rp...
[2] https://monkey.org/~marius/funsrv.pdf
Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices. Linkerd replaces this with something that is explicitly designed to deal with inter-app communication. (I believe one way to run Linkerd is as a small sidecar app inside the same Docker container the man app, as opposed to a big central server, which is also a neat way of doing it.)
Re "why not a library" — sure, you can build all sorts of discovery, distribution and resilience into the client itself. But that consolidates all of this intelligence into just that client. What if you use multiple languages? What about third-party client libs (e.g. Memcached, Redis, PostgreSQL) that you don't control and that you don't want to reimplement? You'll find yourself reimplementing the same thing over and over.
> you can't "drop in" resiliency
I recommend reading the documentation, because drop-in resilience is really the whole point. The guys who built Linkerd previously developed Finagle [1], which is Twitter's RPC library, and when they talk about network resilience, they actually know what they're talking about [2].
[1] https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rp...
[2] https://monkey.org/~marius/funsrv.pdf
> The whole point of Linkerd is moving logic out of your app. Linkerd is like a library but, instead of being in-process, it inserts itself between the app and whatever the app needs to talk to. The app just dials http://foo.svc/, and Linkerd sends the connection to the right place; it doesn't need to know where foo.svc is.
That's just DNS.
> Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices.
Why not DNS round-robin?
> I recommend reading the documentation, because drop-in resilience is really the whole point.
Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
That's just DNS.
> Many companies, including mine, already run an internal load-balancer (such as HAProxy) that takes care of routing traffic between microservices.
Why not DNS round-robin?
> I recommend reading the documentation, because drop-in resilience is really the whole point.
Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
> That's just DNS.
That's not just DNS. Well, it might be - it depends on your environment.
For example: In a world where you have some kind of containerization, or scheduler responsible for deploying your services you typically to have services (or different instances of your services), running on ports determined at runtime. You might solve this with an off-host load balancer, so your frontend applications don't need to worry about it, but now you need to worry about those other things: circuit breaking, retry mechanisms, etc between your application and its middle-tear load balancer.
A common deployment strategy for something like linkerd (or Envoy) would be as a sidecar process, on the same host where you don't need to worry about the reliability of the network to talk to a remote service, because something else is taking care of 90% of your concerns for you.
> Why not DNS round-robin?
There's functionality offered by load balancing at L7 that you can't do with round-robin DNS. You end up having to push that complexity into each service you develop - retry mechanisms, circuit breaking, and routing decisions if you're going to try do green/blue or canary deployments.
You might say "solve this with a library", but the reality is you have to reinvent the wheel, you have to keep that library in sync to ensure the behaviour is consistent, and if you're in an organization that develops services in "N" languages, that challenge can be even greater.
> Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
Absolutely. Your services still need to be able to fail gracefully, but this just moves the complexity of a whole host of other issues to a single, and common spot.
That's not just DNS. Well, it might be - it depends on your environment.
For example: In a world where you have some kind of containerization, or scheduler responsible for deploying your services you typically to have services (or different instances of your services), running on ports determined at runtime. You might solve this with an off-host load balancer, so your frontend applications don't need to worry about it, but now you need to worry about those other things: circuit breaking, retry mechanisms, etc between your application and its middle-tear load balancer.
A common deployment strategy for something like linkerd (or Envoy) would be as a sidecar process, on the same host where you don't need to worry about the reliability of the network to talk to a remote service, because something else is taking care of 90% of your concerns for you.
> Why not DNS round-robin?
There's functionality offered by load balancing at L7 that you can't do with round-robin DNS. You end up having to push that complexity into each service you develop - retry mechanisms, circuit breaking, and routing decisions if you're going to try do green/blue or canary deployments.
You might say "solve this with a library", but the reality is you have to reinvent the wheel, you have to keep that library in sync to ensure the behaviour is consistent, and if you're in an organization that develops services in "N" languages, that challenge can be even greater.
> Fundamentally, the application itself needs to be resilient. No proxy will make it so. It could help, but it can't do it.
Absolutely. Your services still need to be able to fail gracefully, but this just moves the complexity of a whole host of other issues to a single, and common spot.
>That's not just DNS. Well, it might be - it depends on your environment.
In short, another problem introduced by containers and not relevant to non-containers. Why can't a container get its own IP and port like everything else? FreeBSD jails do. Why make it opaque and put the "ingress" component inside of k8s or Docker? It'd be reasonable for k8s to act as a DHCP/DNS server, but not to swallow the whole world.
In short, another problem introduced by containers and not relevant to non-containers. Why can't a container get its own IP and port like everything else? FreeBSD jails do. Why make it opaque and put the "ingress" component inside of k8s or Docker? It'd be reasonable for k8s to act as a DHCP/DNS server, but not to swallow the whole world.
> Why not DNS round-robin?
Round-robin may be sufficient for your needs. However, the load balancing techniques that linkerd employs solved real production issues for us. These techniques allowed us to be paged less frequently for minor errors and we could more reliably make changes to the system without causing user-facing issues.
See https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-b...
Round-robin may be sufficient for your needs. However, the load balancing techniques that linkerd employs solved real production issues for us. These techniques allowed us to be paged less frequently for minor errors and we could more reliably make changes to the system without causing user-facing issues.
See https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-b...
> That's just DNS.
No, it's a bit more complicated than that, as chrisboulton explains (so I won't go into detail).
You could use a dynamic DNS server such as Consul for discovery, and people do. Last I checked, you still had to fight eagerly caching clients, but that's probably doable. But there's still a host of other issues that Linkerd addresses.
Again, I strongly recommend reading about Finagle to understand what this is trying to solve.
Another thing Linkerd brings to the mix is instrumentation and tracing. Instead of building this into every single sender and receiver, Linkerd can collect tracing data for you and make it available for other subsystems to ingest, such as Zipkin or Prometheus.
No, it's a bit more complicated than that, as chrisboulton explains (so I won't go into detail).
You could use a dynamic DNS server such as Consul for discovery, and people do. Last I checked, you still had to fight eagerly caching clients, but that's probably doable. But there's still a host of other issues that Linkerd addresses.
Again, I strongly recommend reading about Finagle to understand what this is trying to solve.
Another thing Linkerd brings to the mix is instrumentation and tracing. Instead of building this into every single sender and receiver, Linkerd can collect tracing data for you and make it available for other subsystems to ingest, such as Zipkin or Prometheus.
No programming language's HTTP/RPC stack supports SRV records. They are just not a thing. Changing your microservices to know about SRV is not a great solution either. Using something like linkerd frees your application from concerns like how discovery and transport is done. SRV is also not compatible with most components out there. e.g. you can't tell memcached/redis to resolve SRV records. You should take a look at how Kubernetes also does not do SRV records, instead it creates an IP address per service for intra-cluster communication. A/AAAA names work with legacy applications and are backwards compatible. That's pretty much why open source community has not been rolling out a major solution based on SRV.
Multiple A records are also fine. Also more supported as you pointed out.
Unless you're constantly talking to a service, your stats will get old quickly. It could also cause everyone to target a specific host creating problems.
I'm still not sure what service discovery is required beyond knowing the name of a service.
Linkerd doesn't free me from anything. I still need to know how it's done, how it's configured, and how to fix it when it breaks.
Unless you're constantly talking to a service, your stats will get old quickly. It could also cause everyone to target a specific host creating problems.
I'm still not sure what service discovery is required beyond knowing the name of a service.
Linkerd doesn't free me from anything. I still need to know how it's done, how it's configured, and how to fix it when it breaks.
I don't think SRV records are the right answer; your networking layer should be k8s aware and issue a watch command on Endpoints; it'll be updated immediately-ish when the servers changes. This is similar to how finagle's zookeeper server set is supposed to work.
What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production. But I think it's straddling a very narrow section; small users shouldn't care about this and just rely on the round robin, zipkin is a PITA to run anyway. Large users will probably want to write their own libraries (zipkin is a lot better if you put traces in your process)
What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production. But I think it's straddling a very narrow section; small users shouldn't care about this and just rely on the round robin, zipkin is a PITA to run anyway. Large users will probably want to write their own libraries (zipkin is a lot better if you put traces in your process)
> I don't think SRV records are the right answer;
OK, A/AAAA are more common.
> your networking layer should be k8s aware and issue a watch command on Endpoints;
Or, I could use standard networking concepts and not build my entire app to suit one new thing that I may not want to stay with forever, or even more than just test.
> What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production
There are already TCP/IP, DNS, DHCP, &c libraries for all languages, and that aren't tied to a specific stack. Why should I build into this stack and not into a more common subset?
OK, A/AAAA are more common.
> your networking layer should be k8s aware and issue a watch command on Endpoints;
Or, I could use standard networking concepts and not build my entire app to suit one new thing that I may not want to stay with forever, or even more than just test.
> What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production
There are already TCP/IP, DNS, DHCP, &c libraries for all languages, and that aren't tied to a specific stack. Why should I build into this stack and not into a more common subset?
DNS for service discovery is fraught with perils. Many implementations don't respect TTLs, or don't actually use multiple records, or don't do something smart like power of two.
Even if you pick all your impls carefully, you have to wait for your TTL instead of a push mechanism for changes.
If you wanted to implement the watch as a library, code wise, it's maybe ~500 lines per language, using the k8s client lib. I could hammer it out in two days.
Even if you pick all your impls carefully, you have to wait for your TTL instead of a push mechanism for changes.
If you wanted to implement the watch as a library, code wise, it's maybe ~500 lines per language, using the k8s client lib. I could hammer it out in two days.
[deleted]
Disclaimer: I'm still researching linkerd and haven't tried to use it.
One component we're interested in is its "automatic" tracing. It claims to be able to annotate requests with Zipkin stats without requiring any instrumentation at the software level.
You are correct that other solutions exist for service discovery, but that's an in-demand feature right now as well, and afaik the main reason that people use linkerd.
>Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
Quick, turn back now from this whole containerization thing, because I assure you this is basically the answer for everything in this space. And you will be aggressively accused of being an old fogie for resisting it. Simplicity is just not appreciated anymore; at this point, "containers" may as well be an antonym for it (and by extension, an antonym for "sane engineering").
One component we're interested in is its "automatic" tracing. It claims to be able to annotate requests with Zipkin stats without requiring any instrumentation at the software level.
You are correct that other solutions exist for service discovery, but that's an in-demand feature right now as well, and afaik the main reason that people use linkerd.
>Maybe I'm just an old fogie, or being dense, but I still don't see how this solves any problems I can't and aren't already solving.
Quick, turn back now from this whole containerization thing, because I assure you this is basically the answer for everything in this space. And you will be aggressively accused of being an old fogie for resisting it. Simplicity is just not appreciated anymore; at this point, "containers" may as well be an antonym for it (and by extension, an antonym for "sane engineering").
I'm personally a big BSD jails fan. The idea of containerization is nice, but people often forget that it requires tooling and care, just like anything. And isn't a magic bullet
I appreciate the sentiment behind linkerd et al but I think we are really over-complicating things and tying ourselves into knots by going head-first into this world. I can't help but think we're going to regret it in a few years.
The complexity introduced not only by Docker, but also k8s and all of the other things competing for a spot in the middle of your cluster, is immense and widely underestimated.
Your applications need to be aware and adapted to the k8s model of automatic pod termination, no hard storage, etc. Google has had over 10 years with the best computer scientists money can buy to perfect their stack. Most companies don't.
For most companies, k8s/linkerd/Docker deployments are a case of overengineering run frighteningly amok.
There are many-a-snafu lying down this path. I'm very worried about how casually everyone is willing to jump aboard.
One particular worry: most Dockerfiles are built on opaque images registered with an image registry, and many of the publishers are not well-known. If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance). These are frequently distributed by complete strangers. Building "from scratch" off the base image distributed by the distro itself is considered "the old way", at least among those I've worked with.
My understanding is that most public image repositories are unmoderated. The moment a widely-used image gets pwned, there is going to be panic.
Source: been converting entire company to Docker/k8s
The complexity introduced not only by Docker, but also k8s and all of the other things competing for a spot in the middle of your cluster, is immense and widely underestimated.
Your applications need to be aware and adapted to the k8s model of automatic pod termination, no hard storage, etc. Google has had over 10 years with the best computer scientists money can buy to perfect their stack. Most companies don't.
For most companies, k8s/linkerd/Docker deployments are a case of overengineering run frighteningly amok.
There are many-a-snafu lying down this path. I'm very worried about how casually everyone is willing to jump aboard.
One particular worry: most Dockerfiles are built on opaque images registered with an image registry, and many of the publishers are not well-known. If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance). These are frequently distributed by complete strangers. Building "from scratch" off the base image distributed by the distro itself is considered "the old way", at least among those I've worked with.
My understanding is that most public image repositories are unmoderated. The moment a widely-used image gets pwned, there is going to be panic.
Source: been converting entire company to Docker/k8s
> If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance)
I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1].
> My understanding is that most public image repositories are unmoderated.
To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both.
> most Dockerfiles are built on opaque images registered with an image registry
Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on.
0. https://hub.docker.com/_/ruby/
1. https://github.com/docker-library/official-images/tree/maste...
I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1].
> My understanding is that most public image repositories are unmoderated.
To the best of my understanding, the official images are maintained by Docker folks, or people associated with the upstream projects, or some mixture of both.
> most Dockerfiles are built on opaque images registered with an image registry
Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
So yes, while it's true there are a lot of public images out there with questionable provenance, there's usually a solid, well-supported foundation to base your own images on.
0. https://hub.docker.com/_/ruby/
1. https://github.com/docker-library/official-images/tree/maste...
>I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images.
Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict.
>Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is
a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort.
Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
[0] https://github.com/docker-library/ruby/blob/752c5f7cf44870ce...
Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
While it is nice that there are a few known-good repositories for projects popular enough to secure the approval of Docker Inc., the habit of casually importing images is still rampant, afaict.
>Far from opaque, the official images are typically well-documented, and the source code is available from the various Git repositories linked to from the library.
Well, here's an example [0] of a base Dockerfile. I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something. After chasing the tree up, this is
a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
and b) a good example of the "brilliance" of the Dockerfile format. That massive set of line continuations is necessary because if you implement the Dockerfile "naively", by making each modification a "RUN" command, you end up with an image that is very bloated due to the way Docker's caching layer functions. Bonus feature: if any of those commands is to return a non-0 exit code, the line will abort.
Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
[0] https://github.com/docker-library/ruby/blob/752c5f7cf44870ce...
I believe most of your questions have a (mostly) positive answer. Admittedly the Docker project moves fast, so I've tried to provide a link or two for each:
> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
The official images are all signed with Notary[0] (released <2 years ago). The "Content Trust" feature makes use of TLS plus a number of keys[1]. It can be enabled in required environments to prevent the pulling of non-signed images (`export DOCKER_CONTENT_TRUST=1`).
> I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something.
There is a helpful Chrome extension, OctoLinker[2], that makes the `FROM parent` clickable (among other non-Docker things). I'm sure there are similar extensions, but this is the one I currently use.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
The `FROM` line in the Dockerfile would likely be the first thing to undergo scrutiny, for obvious reasons. Also, all the layers go through security scanning[3], some issues can be fixed[4] and others are tracked upstream (e.g. by Debian in the case of the Ruby image you linked[5]).
> you end up with an image that is very bloated due to the way Docker's caching layer functions
You can now use `docker build --squash`[6] to combine multiple image layers while still benefiting from layer caching during builds. Also, the final image won't contain files that are added in previous layers and removed in later ones. (And with multi-stage builds recently merged, soon multiple inheritance will be easier.)
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it
I would say groups or sole developers that want others to use their images will usually provide documentation and links to the Dockerfiles. Of course, this isn't always possible or desired (e.g. see closed-source projects like Windows Server Core[7], or the countless people using Docker Hub as a free image host.)
[0]: https://github.com/docker/notary
[1]: https://docs.docker.com/engine/security/trust/trust_key_mng/
[2]: https://octolinker.github.io/
[3]: https://docs.docker.com/docker-hub/official_repos/#how-do-i-...
[4]: https://github.com/docker-library/official-images/pulls?utf8...
[5]: https://github.com/docker-library/ruby/issues/117
[6]: https://docs.docker.com/engine/reference/commandline/build/#...
[7]: https://hub.docker.com/r/microsoft/windowsservercore/
> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
The official images are all signed with Notary[0] (released <2 years ago). The "Content Trust" feature makes use of TLS plus a number of keys[1]. It can be enabled in required environments to prevent the pulling of non-signed images (`export DOCKER_CONTENT_TRUST=1`).
> I had to go 3 levels up from the link on the Docker Hub page to get to this one because most of the images are FROM:something.
There is a helpful Chrome extension, OctoLinker[2], that makes the `FROM parent` clickable (among other non-Docker things). I'm sure there are similar extensions, but this is the one I currently use.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
The `FROM` line in the Dockerfile would likely be the first thing to undergo scrutiny, for obvious reasons. Also, all the layers go through security scanning[3], some issues can be fixed[4] and others are tracked upstream (e.g. by Debian in the case of the Ruby image you linked[5]).
> you end up with an image that is very bloated due to the way Docker's caching layer functions
You can now use `docker build --squash`[6] to combine multiple image layers while still benefiting from layer caching during builds. Also, the final image won't contain files that are added in previous layers and removed in later ones. (And with multi-stage builds recently merged, soon multiple inheritance will be easier.)
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it
I would say groups or sole developers that want others to use their images will usually provide documentation and links to the Dockerfiles. Of course, this isn't always possible or desired (e.g. see closed-source projects like Windows Server Core[7], or the countless people using Docker Hub as a free image host.)
[0]: https://github.com/docker/notary
[1]: https://docs.docker.com/engine/security/trust/trust_key_mng/
[2]: https://octolinker.github.io/
[3]: https://docs.docker.com/docker-hub/official_repos/#how-do-i-...
[4]: https://github.com/docker-library/official-images/pulls?utf8...
[5]: https://github.com/docker-library/ruby/issues/117
[6]: https://docs.docker.com/engine/reference/commandline/build/#...
[7]: https://hub.docker.com/r/microsoft/windowsservercore/
Thanks for this post. It does indeed answer most of my follow-up questions.
I wanted to let you know that I gave `docker build --squash` a try today. This is the output I got:
Also, I encountered another issue today when trying to do a `docker push`. It retried a cache layer multiple times before conceding with `open /dev/mapper/docker-254:0-xxx-yyy-zzz: no such file or directory` on one of the layers. I had to rebuild the image with `--no-cache` to get it to push.
Not doing anything unusual/fancy, just an ordinary docker push, which worked fine on other images before and after. No auto-cleanup scripts running in the background or anything that should cause a layer to go mysteriously missing.
I wanted to let you know that I gave `docker build --squash` a try today. This is the output I got:
"--squash" is only supported on a Docker daemon with experimental features enabled
Also, I encountered another issue today when trying to do a `docker push`. It retried a cache layer multiple times before conceding with `open /dev/mapper/docker-254:0-xxx-yyy-zzz: no such file or directory` on one of the layers. I had to rebuild the image with `--no-cache` to get it to push.
Not doing anything unusual/fancy, just an ordinary docker push, which worked fine on other images before and after. No auto-cleanup scripts running in the background or anything that should cause a layer to go mysteriously missing.
> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?
I'm not sure. You can find more information about them here[0].
> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
I'm not sure about this either. It looks like Rancher and OpenShift support whitelisting registries, so this may have to be handled at a higher level. I believe you can always tell whether something being pulled from Docker Hub is official because it won't have a leading path component indicating the contributor (e.g. mwpmaybe/ruby:2.4-alpine is a non-official image whereas ruby:2.4-alpine is official).
> the habit of casually importing images is still rampant
This is just my two cents, but that seems like a policy problem, not a technology problem. Enforce a rule for your project/team/organization that containers have to use an official image, or a private image based on an official image. I do agree that a simple per-registry namespace whitelist would be a nice feature to have in dockerd.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
I believe official images must be based on other official images.
> b) a good example of the "brilliance" of the Dockerfile format
I don't think it was ever meant to be brilliant, just simple. I'm definitely looking forward to the next generation of Dockerfile formats that ameliorate this issue and many others.
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
I believe so. This is a complaint I have with Docker Hub. I personally feel that the Dockerfiles used to create public images should always be published along with the images.
0. https://docs.docker.com/docker-hub/official_repos/
I'm not sure. You can find more information about them here[0].
> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
I'm not sure about this either. It looks like Rancher and OpenShift support whitelisting registries, so this may have to be handled at a higher level. I believe you can always tell whether something being pulled from Docker Hub is official because it won't have a leading path component indicating the contributor (e.g. mwpmaybe/ruby:2.4-alpine is a non-official image whereas ruby:2.4-alpine is official).
> the habit of casually importing images is still rampant
This is just my two cents, but that seems like a policy problem, not a technology problem. Enforce a rule for your project/team/organization that containers have to use an official image, or a private image based on an official image. I do agree that a simple per-registry namespace whitelist would be a nice feature to have in dockerd.
> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)
I believe official images must be based on other official images.
> b) a good example of the "brilliance" of the Dockerfile format
I don't think it was ever meant to be brilliant, just simple. I'm definitely looking forward to the next generation of Dockerfile formats that ameliorate this issue and many others.
> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).
I believe so. This is a complaint I have with Docker Hub. I personally feel that the Dockerfiles used to create public images should always be published along with the images.
0. https://docs.docker.com/docker-hub/official_repos/
>This is just my two cents, but that seems like a policy problem, not a technology problem.
Software that makes dangerous things too easy is a technology problem. This is why Linux distros require signed packages by default.
Furthermore, one of Docker's popular appeals is "never get stuck in dependency hell again: just install someone's container!" You see more and more projects saying "The best way to install this is just to use our Docker container." The culture is definitely not leading to a security-minded perspective.
Compare with something like Ansible Playbooks, where the commands executed are always visible.
>I don't think it was ever meant to be brilliant, just simple.
Requiring long continuations for reasonable image sizes is not simple. Without the continuations, final image sizes regularly bloat to be larger than a typical distro ISO. On the other hand, exchanging the cached layers for size nullifies some of Docker's claimed value, specifically wrt atomicity and build time.
Software that makes dangerous things too easy is a technology problem. This is why Linux distros require signed packages by default.
Furthermore, one of Docker's popular appeals is "never get stuck in dependency hell again: just install someone's container!" You see more and more projects saying "The best way to install this is just to use our Docker container." The culture is definitely not leading to a security-minded perspective.
Compare with something like Ansible Playbooks, where the commands executed are always visible.
>I don't think it was ever meant to be brilliant, just simple.
Requiring long continuations for reasonable image sizes is not simple. Without the continuations, final image sizes regularly bloat to be larger than a typical distro ISO. On the other hand, exchanging the cached layers for size nullifies some of Docker's claimed value, specifically wrt atomicity and build time.
> Requiring long continuations for reasonable image sizes is not simple.
We're just arguing semantics at this point. I mean "simple" like Assembler is simple: rudimentary instructions requiring a certain amount of understanding and sophistication to wield correctly. Not "simple" as in easy, intuitive, and footgun-free.
We're just arguing semantics at this point. I mean "simple" like Assembler is simple: rudimentary instructions requiring a certain amount of understanding and sophistication to wield correctly. Not "simple" as in easy, intuitive, and footgun-free.
> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?
In Kubernetes land, you can use the ImagePolicyWebhook admission controller. https://kubernetes.io/docs/admin/admission-controllers/#imag...
I agree that Dockerfiles are awful.
In Kubernetes land, you can use the ImagePolicyWebhook admission controller. https://kubernetes.io/docs/admin/admission-controllers/#imag...
I agree that Dockerfiles are awful.
It's perfectly normal to use base images and build your own. We do this for most things.
One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily.
Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one.
Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.
One simple reason is control: You want to be able to follow CVEs and upgrade your images in line with patches. If every image branches off some random Docker Hub image, the only way to do that would be some kind of image vulnerability scanner, which I believe exists, but wouldn't help you actually patch anything easily.
Public images are still super nice to bootstrap things. You can start with a public image, then move to a custom-built one.
Lastly: I wouldn't conflate the Docker image challenge with the operational complexity of using Kubernetes and microservices. They are completely orthogonal in my mind. Docker has all sorts of issues that we need to solve sooner rather than later; Kubernetes' design, on the other hand, is sound.
I recognize that Docker and k8s are distinct, and I think rkt looks much smarter than Docker. Now that k8s has released a stable version capable of running non-Docker containers (as of last week), it's fair to separate them.
However, I'm still struggling to see the real value proposition offered by either.
The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.
However, I'm still struggling to see the real value proposition offered by either.
The story on Docker or Docker-style containers without orchestration is that they're supposed to be easy to ensure you get an identical environment everywhere and more fully utilize your hardware resources. While that does seem to happen if you incorporate docker-compose or some other metadata about what ports to map and volumes to mount, the pains associated with using Docker have far, far outweighed the one-time cost of configuring a new application or getting a new build to work for me. And we already had pretty good tooling for this pre-container, with things like Ansible. It wasn't exact, but it was much more comfortable.
Being a Puppet user (until we've finally retired that nightmare), I couldn't be happier with this migration to containers.
Docker itself is problematic for all sorts of reasons, but it at least does the most important thing, which is to run pre-packaged code in isolated containers. With Puppet, as with Ansible, you just can't guarantee an isolated environment, nor is the environment easily "replicable", in the sense that if you need more instances of the exact same thing (e.g. more data processing workers), there's nothing in the system built to treat things as N-of-anything. The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.
Meanwhile, Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes, where in Docker Engine they're just low-level primitives you need to manage manually. It also handles exec'ing into an existing container. The pod as the unit of orchestration is also much nicer than Docker's single-process containers.
For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally). Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing, I rest assured that what is running is what I deployed.
Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc. We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.
Docker, to me, isn't particularly interesting without Kubernetes, although it has its uses. It's terrific to be able to "boot" a temporary Ubuntu or Elasticsearch or whatever in order to do something that I can just tear down afterwards. Build/test containers are great, both locally and in a CI environment.
What pains are you referring to, specifically?
Docker itself is problematic for all sorts of reasons, but it at least does the most important thing, which is to run pre-packaged code in isolated containers. With Puppet, as with Ansible, you just can't guarantee an isolated environment, nor is the environment easily "replicable", in the sense that if you need more instances of the exact same thing (e.g. more data processing workers), there's nothing in the system built to treat things as N-of-anything. The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.
Meanwhile, Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes, where in Docker Engine they're just low-level primitives you need to manage manually. It also handles exec'ing into an existing container. The pod as the unit of orchestration is also much nicer than Docker's single-process containers.
For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally). Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing, I rest assured that what is running is what I deployed.
Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc. We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.
Docker, to me, isn't particularly interesting without Kubernetes, although it has its uses. It's terrific to be able to "boot" a temporary Ubuntu or Elasticsearch or whatever in order to do something that I can just tear down afterwards. Build/test containers are great, both locally and in a CI environment.
What pains are you referring to, specifically?
I know I was just downvoted for answering your questions before, but I thought I'd reply here just to share what I've hit so far during today's work day using Docker. Just today.
1. `docker push` fails because of a mysteriously-vanished cache layer, forcing `docker build --no-cache` to work.
2. `docker-compose stop [service]` broke in a weird way, failing to detect the service's stoppage, creating a duplicate service of the same name that I couldn't do anything with. Had to clear out the duplicate container with `docker rm -fv ...` and now trying to figure out how to make `docker-compose` happy again.
These are totally normal, conventional operations that should not be choking for unexplained reasons.
And people suggest running production workloads on this, for some reason.
and bonus:
3. Tried the `docker build --squash` feature suggested by another commenter, to find it requires "experimental support" to be enabled in the Docker daemon. I have enough problems with docker without that, so I'm going to pass.
1. `docker push` fails because of a mysteriously-vanished cache layer, forcing `docker build --no-cache` to work.
2. `docker-compose stop [service]` broke in a weird way, failing to detect the service's stoppage, creating a duplicate service of the same name that I couldn't do anything with. Had to clear out the duplicate container with `docker rm -fv ...` and now trying to figure out how to make `docker-compose` happy again.
These are totally normal, conventional operations that should not be choking for unexplained reasons.
And people suggest running production workloads on this, for some reason.
and bonus:
3. Tried the `docker build --squash` feature suggested by another commenter, to find it requires "experimental support" to be enabled in the Docker daemon. I have enough problems with docker without that, so I'm going to pass.
>The only unit of provisioning is the VM. Puppet/Salt/Ansible are highly static systems.
If you're operating the same script from the same image (say, a distro's official ISO), they should work effectively the same way. It is true that bootstrapping nodes may break, but unlikely. It shouldn't be common.
There's no reason something like Ansible couldn't be extended to build containers. Playbooks effectively perform the same function as Dockerfiles: they list commands to execute off of a base image. Playbooks are somewhat more flexible in that they're frequently used to address already-running machines.
For the record, push-centric configuration management like Chef and Puppet is much worse than Salt or Ansible, which can connect to arbitrary nodes and execute commands.
>Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes
Maybe we adopted too early but none of these things are "neatly" captured by Kubernetes. We have complex superstructures over all of them. They may be neater than running them in Docker alone, but they're much more complex than running against bare nodes.
Particularly, addressing application instances and getting stateful storage have been challenges in Kubernetes, just as they are in Docker. StatefulSets went stable in December (3 months ago) and I haven't tried them yet, so maybe those concerns are assuaged.
The ideal interface for k8s would be, essentially, an aware hypervisor. If each pod acted as a subnet and each container acted like a machine in that subnet, meaning it had an IP that was accessible according to external firewall and DNS rules and otherwise acted like a normal machine, that'd be much easier to accept. Neither Docker nor Kubernetes are amenable to this.
>It also handles exec'ing into an existing container.
The interface for this is practically the same as it is on Docker. You do `kubectl exec -it <pod> /bin/ash` instead of `docker exec -it <container_id> /bin/ash`. The only extra thing it handles is making the container appear local to the master, instead of requiring one to get on the correct kubelet.
>For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally).
The value in this being running more applications on a single cloud instance? If you have the instance/VM as the unit of orchestration, it works the same way: a coordinator (load balancer) divides the traffic among the units, and then stops transparently when you deregister or when a health check fails.
>Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing
"Orchestration, scheduling, autoscaling etc." refer, more or less, to the same thing: getting your applications running somewhere. Containers and k8s in no way have a monopoly on that functionality, and my experience has not been that they've significantly simplified it.
>I rest assured that what is running is what I deployed.
If running a playbook over a fresh node isn't good enough, you can use an image system to ensure it's a binary copy of what you built. There were several competing APIs for managing this in a platform-agnostic manner.
>Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc.
I know that k8s includes some logic to attempt to automate the failover/deployment process, and that is a nice bonus. Not without equivalents, but I believe that k8s is positioned to do this more reliably than other systems.
Otherwise, same story here; containers and k8s did not introduce these capabilities, and I'm not sure how it claims to improve them.
k8s seems necessary because it makes all of this that already had reasonable solutions on non-containers work well with containers. Thus, no containers, no real reason to use k8s.
>We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.
Setting request quotas and node affinities is not fun. In a busy environment, it's like managing a shared host; you have to fine-tune each application so that it doesn't go outside of its box, and this is not completely reliable if anything else is consuming resources (say, dockerd, which was recently running at 450% utilization on one of our instances).
>What pains are you referring to, specifically?
a) Redoing all applications so they can run without a permanent filesystem, so they can be prepared for pod termination at any moment, so they don't care about being individually addressable, and so on. This by itself is a big deal. Containers etc don't make sense for a stateful project, and there's no reason to assume that every project needs to be stateless. "Operators" have been suggested as CoreOS as a way to handle this, but that's a whole 'nother headache to add to the pile, requiring registration of third-party resource types and listeners to gracefully handle pod termination.
b) Creating usable Dockerfiles for everything and maintaining them. In our case, we converted to Alpine because it "made smaller images", which was also a pain point due to lots of things not being friendly with musl. This includes tons of troubleshooting, weird little tricks, injecting things into the right place in the Dockerfile to keep the image size minimized, trying to map ports and do all the esoteric invocations required by the Docker daemon when running and linking the containers, clearing out images and containers before they make your disk full since Docker doesn't know how to do this intelligently, and so forth.
c) Getting everything to talk to the things it needs through Kubernetes, which involves a lot of config and port mapping, whereas previously we only needed to say "Go to DNS name" within our LAN (firewall managed separately). Kubernetes sort of allows this, for a subset of containers, if they're in the right cluster and the right label has been applied or something. Ingress resources and controllers are filed under this umbrella.
d) Troubleshooting or analyzing the live environment is much more difficult due to all of the weird network shenanigans going on, the restricted context available within the container, lack of diagnostic tools, etc.
e) Kubernetes configs are long, esoteric, and rely on a complex web of interdependent k8s-specific objects. Deployment success/failure takes multiple steps to ascertain (deploy; get pods; describe pod if failure; look for relevant message and respond, which requires repeating the process).
f) Docker hanging and breaking all containers until host reboot; Docker eating all disk space during development (for images/builds, usually); Docker crashing randomly; etc.
Say you want to interactively test a change to something that's running in a Docker container. You can get into the container and edit the file in place (if you install vim or similar), but beware that restarting the process will delete your container and revert back to the image. Your options are a) edit the file in your local environment, rebuild the Dockerfile locally, re-run the image, re-trigger the event, observe bugs, and repeat; or b) circumvent by doing docker start <container_id>; docker exec -it <container_id> /your/shell; and running the processes, detaching from them, and editing the process in-container, retriggering the processes that would be fired by the CMD or ENTRYPOINT each time to reload the application; hope all of this works fine in the limited context provided by the container. Then, exiting the container, committing the image, pulling out your changes to your local environment where you can commit them, pushing to git and the image registry, and hoping that the Docker container, which hypothetically is supposed to work automatically everywhere, actually ends up doing that, which seems not to happen pretty often for something that's supposed to work everywhere.
OR you could spin up a test instance/VM/whatever, or set up a test version of the application on your local environment, BOTH of which can usually be done with the same Ansible playbook, and then you can edit normally without worrying about the filesystem disappearing or the repeated builds taking all your space and test normally without worrying about esoteric error messages due to the limited container environment, restricted sandbox permissions, or incorrect network flags on the container invocation. You run tests, commit the change, which is not at risk of disappearing, and move on.
Anyway, if you're already committed to Docker for whatever reason, Kubernetes is probably an OK way to generalize it and try to make it less bad. I just see no compelling reason to commit to either other than a significant concern of maximum utilization of cloud instances, which is a fine concern, but not in exchange for the extra cost incurred in k8s.
Maybe my opinion would change if I stepped away from our environment and tried to bootstrap my own "clean" k8s cluster and Dockerfiles, but my experiments with minikube have featured a lot of the same frustrations.
Everything just seems to take a lot longer this way, which makes sense because we've added many layers of complication on top of what was a previously well-defined workflow. Containers could be nice if they were more mature, but for now, there is every reason to stay away.
If you're operating the same script from the same image (say, a distro's official ISO), they should work effectively the same way. It is true that bootstrapping nodes may break, but unlikely. It shouldn't be common.
There's no reason something like Ansible couldn't be extended to build containers. Playbooks effectively perform the same function as Dockerfiles: they list commands to execute off of a base image. Playbooks are somewhat more flexible in that they're frequently used to address already-running machines.
For the record, push-centric configuration management like Chef and Puppet is much worse than Salt or Ansible, which can connect to arbitrary nodes and execute commands.
>Kubernetes smoothes over a lot of the rough parts of Docker: Ports, IPs, DNS/discovery, monitoring and volume management are all neatly captured by Kubernetes
Maybe we adopted too early but none of these things are "neatly" captured by Kubernetes. We have complex superstructures over all of them. They may be neater than running them in Docker alone, but they're much more complex than running against bare nodes.
Particularly, addressing application instances and getting stateful storage have been challenges in Kubernetes, just as they are in Docker. StatefulSets went stable in December (3 months ago) and I haven't tried them yet, so maybe those concerns are assuaged.
The ideal interface for k8s would be, essentially, an aware hypervisor. If each pod acted as a subnet and each container acted like a machine in that subnet, meaning it had an IP that was accessible according to external firewall and DNS rules and otherwise acted like a normal machine, that'd be much easier to accept. Neither Docker nor Kubernetes are amenable to this.
>It also handles exec'ing into an existing container.
The interface for this is practically the same as it is on Docker. You do `kubectl exec -it <pod> /bin/ash` instead of `docker exec -it <container_id> /bin/ash`. The only extra thing it handles is making the container appear local to the master, instead of requiring one to get on the correct kubelet.
>For me, the value proposition is the cloud. When I boot up N nodes, I know that Kubernetes will optimally pack them with my container, and if I boot up N+1 nodes, I know that the containers will spread to the new node, and back down again if I go down to N (either intentionally or accidentally).
The value in this being running more applications on a single cloud instance? If you have the instance/VM as the unit of orchestration, it works the same way: a coordinator (load balancer) divides the traffic among the units, and then stops transparently when you deregister or when a health check fails.
>Orchestration, scheduling, autoscaling etc. take care of this "cloud of containers", if you will, and since every container is an isolated, identical, versioned, stateless (for the most part!), fully redundant thing
"Orchestration, scheduling, autoscaling etc." refer, more or less, to the same thing: getting your applications running somewhere. Containers and k8s in no way have a monopoly on that functionality, and my experience has not been that they've significantly simplified it.
>I rest assured that what is running is what I deployed.
If running a playbook over a fresh node isn't good enough, you can use an image system to ensure it's a binary copy of what you built. There were several competing APIs for managing this in a platform-agnostic manner.
>Then there are all the other nice things, like rolling deploys, cordoning, self-healing, job scheduling, secret management, being able to boot up a parallel, different version of a stack without clobbering everything else, etc.
I know that k8s includes some logic to attempt to automate the failover/deployment process, and that is a nice bonus. Not without equivalents, but I believe that k8s is positioned to do this more reliably than other systems.
Otherwise, same story here; containers and k8s did not introduce these capabilities, and I'm not sure how it claims to improve them.
k8s seems necessary because it makes all of this that already had reasonable solutions on non-containers work well with containers. Thus, no containers, no real reason to use k8s.
>We run production and staging in the same Kubernetes cluster (different namespaces), and we can use things like resource requests/limits and node affinities to ensure that they're scheduled differently, but still using the same underlying hardware resources.
Setting request quotas and node affinities is not fun. In a busy environment, it's like managing a shared host; you have to fine-tune each application so that it doesn't go outside of its box, and this is not completely reliable if anything else is consuming resources (say, dockerd, which was recently running at 450% utilization on one of our instances).
>What pains are you referring to, specifically?
a) Redoing all applications so they can run without a permanent filesystem, so they can be prepared for pod termination at any moment, so they don't care about being individually addressable, and so on. This by itself is a big deal. Containers etc don't make sense for a stateful project, and there's no reason to assume that every project needs to be stateless. "Operators" have been suggested as CoreOS as a way to handle this, but that's a whole 'nother headache to add to the pile, requiring registration of third-party resource types and listeners to gracefully handle pod termination.
b) Creating usable Dockerfiles for everything and maintaining them. In our case, we converted to Alpine because it "made smaller images", which was also a pain point due to lots of things not being friendly with musl. This includes tons of troubleshooting, weird little tricks, injecting things into the right place in the Dockerfile to keep the image size minimized, trying to map ports and do all the esoteric invocations required by the Docker daemon when running and linking the containers, clearing out images and containers before they make your disk full since Docker doesn't know how to do this intelligently, and so forth.
c) Getting everything to talk to the things it needs through Kubernetes, which involves a lot of config and port mapping, whereas previously we only needed to say "Go to DNS name" within our LAN (firewall managed separately). Kubernetes sort of allows this, for a subset of containers, if they're in the right cluster and the right label has been applied or something. Ingress resources and controllers are filed under this umbrella.
d) Troubleshooting or analyzing the live environment is much more difficult due to all of the weird network shenanigans going on, the restricted context available within the container, lack of diagnostic tools, etc.
e) Kubernetes configs are long, esoteric, and rely on a complex web of interdependent k8s-specific objects. Deployment success/failure takes multiple steps to ascertain (deploy; get pods; describe pod if failure; look for relevant message and respond, which requires repeating the process).
f) Docker hanging and breaking all containers until host reboot; Docker eating all disk space during development (for images/builds, usually); Docker crashing randomly; etc.
Say you want to interactively test a change to something that's running in a Docker container. You can get into the container and edit the file in place (if you install vim or similar), but beware that restarting the process will delete your container and revert back to the image. Your options are a) edit the file in your local environment, rebuild the Dockerfile locally, re-run the image, re-trigger the event, observe bugs, and repeat; or b) circumvent by doing docker start <container_id>; docker exec -it <container_id> /your/shell; and running the processes, detaching from them, and editing the process in-container, retriggering the processes that would be fired by the CMD or ENTRYPOINT each time to reload the application; hope all of this works fine in the limited context provided by the container. Then, exiting the container, committing the image, pulling out your changes to your local environment where you can commit them, pushing to git and the image registry, and hoping that the Docker container, which hypothetically is supposed to work automatically everywhere, actually ends up doing that, which seems not to happen pretty often for something that's supposed to work everywhere.
OR you could spin up a test instance/VM/whatever, or set up a test version of the application on your local environment, BOTH of which can usually be done with the same Ansible playbook, and then you can edit normally without worrying about the filesystem disappearing or the repeated builds taking all your space and test normally without worrying about esoteric error messages due to the limited container environment, restricted sandbox permissions, or incorrect network flags on the container invocation. You run tests, commit the change, which is not at risk of disappearing, and move on.
Anyway, if you're already committed to Docker for whatever reason, Kubernetes is probably an OK way to generalize it and try to make it less bad. I just see no compelling reason to commit to either other than a significant concern of maximum utilization of cloud instances, which is a fine concern, but not in exchange for the extra cost incurred in k8s.
Maybe my opinion would change if I stepped away from our environment and tried to bootstrap my own "clean" k8s cluster and Dockerfiles, but my experiments with minikube have featured a lot of the same frustrations.
Everything just seems to take a lot longer this way, which makes sense because we've added many layers of complication on top of what was a previously well-defined workflow. Containers could be nice if they were more mature, but for now, there is every reason to stay away.
Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the space is way out of wack.
I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important.
My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere.
Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines.
I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.
My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here and there, there was demand for it because there's demand for anything and everything with Google's name attached to it, regardless of applicability or propriety or any good sense (case in point: companies pretending to be smart by "integrating" TensorFlow), and they released k8s in response to the clamor for the mysterious Google-backed orchestration platform, much to the detriment of non-Googlers everywhere.
Maybe it's a coordinated plan to harm young startups that may challenge them. Or, alternate theory: perhaps it's a coordinated plan to get people to buy far more cloud instances than they actually need, since k8s literally cannot run on a single instance; to run it locally, you have to run virtual machines.
I understand that k8s has improved and that many people use it (myself among them), but it maintains that vibe of an internal tool; something so convoluted and complex and inside baseball that you'd only voluntarily undertake it if someone said your paycheck was at risk.
As a Kubernetes user, I don't understand the "vibe of an internal tool" comment. It feels nothing like that to me.
Kubernetes is big and complex, to be sure, but to me its state of being unfinished is not because it's "internal" anything, but because — since Google couldn't just open-source Borg and Omega, which presumably have a decade of Google-specific cruft that wouldn't translate to a reusable tool — it had to be built from scratch, and is only slowly accruing the various features needed to approximate the maturity of a full-blown orchestration system of the kind that Google uses already.
In other words, Kubernetes works great and all, but it's also immature in many ways: Ingress is painful, secrets management not secure enough; we're only now getting stateful sets, affinities, federation, role-based security, etc.; we're still waiting for things like scheduled jobs and custom metrics support in the autoscaler; Kubelet is still an awkward SPOF; etc. But those shortcomings have nothing to do with it being an internal tool, but rather because there's just a lot of work left to be done.
Kubernetes is big and complex, to be sure, but to me its state of being unfinished is not because it's "internal" anything, but because — since Google couldn't just open-source Borg and Omega, which presumably have a decade of Google-specific cruft that wouldn't translate to a reusable tool — it had to be built from scratch, and is only slowly accruing the various features needed to approximate the maturity of a full-blown orchestration system of the kind that Google uses already.
In other words, Kubernetes works great and all, but it's also immature in many ways: Ingress is painful, secrets management not secure enough; we're only now getting stateful sets, affinities, federation, role-based security, etc.; we're still waiting for things like scheduled jobs and custom metrics support in the autoscaler; Kubelet is still an awkward SPOF; etc. But those shortcomings have nothing to do with it being an internal tool, but rather because there's just a lot of work left to be done.
Just FYI, over 50% of contributions come from !Google, and that's part of the reason that so many people (CoreOS, Redhat, Apprenda, Deis, thousands of internal folks) have rebased on it.
I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
Disclosure: I work at Google on Kubernetes
I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
Disclosure: I work at Google on Kubernetes
So I actually answered this question for you when you asked it in December: https://news.ycombinator.com/item?id=13241681 . ;)
Not to toot the same horn.
I have more experience with Kubernetes now than I did then, though I think a lot of those interface snafus, which is most of what the old post discussed, are still valid complaints.
You mentioned back then that if people aren't aware of some kube featureset, they end up reimplementing it. You mentioned logging. Can you tell me what you recommend for logging?
Right now, we have everything writing logs to stdout in the container, which gets recorded by kube in something like /var/log/kube/containers/ * . Then we have a fluentd container that reads the logs and uploads them to an external machine, which is running fluentd containers that receive the input stream, transform it according to rules, and pipe it out to a cloud-based log aggregator.
Is that how it's supposed to work? I know we used to do it somewhat differently, but had trouble with resource consumption that was disrupting our pods.
And here's a potentially legitimate Kubernetes issue: the scheduler is only reading the load generated from containers administered by Kubernetes, so even though the box is being thrashed, k8s will not be aware of the problem and continues to report the node's resources as healthy. In our case, the dockerd process was using 4.5 cores. Kubernetes seemed unaware and it was affecting our performance because another busy pod was on the server and was not getting clamped by its quota (since it, technically, was not hitting the quota for itself). Shouldn't these be defined relative to overall system resources, and not just the containers that kube can see?
It's possible Kubernetes was aware, but that we didn't know how to see it. A colleague told me the interface I was looking at for pod performance data reflects only the quotas, not the actual utilization, so maybe k8s knew (why isn't there a `kubetop` that can be run on the host to supervise?). I also hear that automatic rescheduling is still in draft stage, so if k8s detects high load on a kubelet, instead of moving the pod, it will just not assign new pods to it.
About addressing. Right now, we have dev, stage, and prod. We have a container that takes all traffic targeting the kubelet on port 80 and proxies it out to the pod using servicelb. It's set up to forward from http://app-label.env-name.example.com to a pod matching that label. It's annoying that we have to run an external container for this in the first place and do extra configuration on it, but it's also annoying that since servicelb is not aware of the beta label attempting to simulate hostnames, we can't individually address instances of apps. Is that how this is supposed to work, or is there a ready-built solution that makes this much easier that we're overlooking?
----
>I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
The best way to improve Kubernetes would be to:
a) simplify the terminology;
b) simplify the kubectl command, and particularly make it follow semi-standard Unix utility conventions (examples: -f should mean force or not be used (and yes, I know some other tools commit this sin too, but the association with "force" is nerve-wracking on a production environment), should probably be -r or -o; --from-file is ambiguous, and should be --read-literal or -r --raw or something like that);
c) simplify the configuration syntax and files;
d) provide simple ways to get real storage and real IPs (I haven't thoroughly investigated StatefulSets, so it may do this). I know that Google has long ago gone into a plane where these are irrelevant, but normal companies haven't, and IMO there's no reason they necessarily should;
e) provide simple backup and restoration methods (afaik, this involves multiple steps, including backing up the etcd cluster (not allowed to run on the master, or just conventionally doesn't?)).
My suggestion for distributed workloads is:
a) script your machine in Ansible or similar, and make a base image from that;
b) use your cloud provider of choice to deploy new instances under the circumstances considered necessary;
c) register the new node with haproxy or whatever sits at your front end (often a cloud-provided LB);
d) use conventional administrative and monitoring tooling to administer.
All of that is fully scriptable, in far less time than it takes to convert to Docker/k8s.
-----
>TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
Why is the above-recommended distributed systems strategy inadequate, even for people with more than 3 nodes who care about downtime? What's fundamentally wrong with it? What unique value does k8s bring to the table?
What's the k8s replacement for htop? tcpdump? df? and all the other utilities, and why are they better, and what benefit does k8s provide, that we didn't already have, that warrants giving up everything? I know it's not TRYING to replace all of that per se, but the practicalities make kubernetes the authority for such information, and you're not supposed to have to get onto the kubelet itself to diagnose these types of issues so you can't even use root on the Docker host to try and do some of this.
In our deployment, the main k8s jockey just kills pods if they're acting up and hopes that will magically fix it, because the devs don't want to get near the setup with a 10-foot pole. Before, they would SSH in and go over the issues. Now, since not only can't you SSH in but there's a convoluted process to get a shell, and then the container will not have any diagnosis tools in it anyway, it's very hard to have them collaboratively troubleshoot a problem we're seeing in the wild.
I can't tell you how many times I've had someone tell me "They must have a better answer for that; Google uses it after all...". These people are under the mistaken impression that if Google is using it, it must be robust and stable, when in fact the opposite is usually true, and I don't think Google tries to pretend it's not. It's probably just misinterpreted the signal from the developer community as meaningful approbation, when it's really just blindly following the cool kid on the block.
It's not that k8s isn't a neat thing. It's just that it's solving a problem that pretty much only Google had, and now everyone else is plummeting down the rabbit hole.
We've spent probably 1.5 man-years getting our infrastructure containerized and kubed up. We're not at 100% in prod yet but getting pretty close. It's clear that I'm really dumb but I haven't even been the main one doing this, so can't blame it on me. That's a huge project, and how does k8s/Docker justify the cost? It lets us cut down on AWS usage? Sure, but nowhere near the proportion of cost. It "makes it easier" to administer the cluster? Nope, not a chance. It's fun to kill pods but that's living on the edge, most apps written by non-Google can't take that. Solution? Write an "operator", which appears to be any random program that intercepts the k8s API and does stuff to support pods, etc. That sucks, why don't I just keep the init script that works fine?
One could argue that k8s is about resource utilization, which is sort-of-but-not-really-true. And that has a downside; we've essentially turned our environment into a shared host now, and despite k8s quotas, we still have pods that are bad neighbors; this is turning out to be quite difficult to control.
So I'm just really at a loss here. Google wanted it and they built it and it's working for them, that's great. That's an internal tool. How does this help anyone who wants to run a normal site this way? I haven't heard a single success story that had a moral or goal behind it other than "I'm cool like Google now too."
And on top of all of this, before k8s even comes into the picture, you have to make a Dockerfile and run your application in Docker, which is a PITA itself, in terms of stability, security, and configuration (see above-mentioned dockerd hammering the box, dockerd hangs or breakages that render a box useless, having to deal with port forwards, name collisions, large Docker caches, docker's inability to remove old images or containers on its own, Dockerfile lameness, culture of importing unknown images, etc.).
I like the ideas of containers (and have liked them when they were called "jails" in past eras ;) ) and I like the idea of orchestrators like kubernetes. But I think it's very early days right now for both, and that they will need radically improved simplification and stability before they are appropriate for general use.
Not to toot the same horn.
I have more experience with Kubernetes now than I did then, though I think a lot of those interface snafus, which is most of what the old post discussed, are still valid complaints.
You mentioned back then that if people aren't aware of some kube featureset, they end up reimplementing it. You mentioned logging. Can you tell me what you recommend for logging?
Right now, we have everything writing logs to stdout in the container, which gets recorded by kube in something like /var/log/kube/containers/ * . Then we have a fluentd container that reads the logs and uploads them to an external machine, which is running fluentd containers that receive the input stream, transform it according to rules, and pipe it out to a cloud-based log aggregator.
Is that how it's supposed to work? I know we used to do it somewhat differently, but had trouble with resource consumption that was disrupting our pods.
And here's a potentially legitimate Kubernetes issue: the scheduler is only reading the load generated from containers administered by Kubernetes, so even though the box is being thrashed, k8s will not be aware of the problem and continues to report the node's resources as healthy. In our case, the dockerd process was using 4.5 cores. Kubernetes seemed unaware and it was affecting our performance because another busy pod was on the server and was not getting clamped by its quota (since it, technically, was not hitting the quota for itself). Shouldn't these be defined relative to overall system resources, and not just the containers that kube can see?
It's possible Kubernetes was aware, but that we didn't know how to see it. A colleague told me the interface I was looking at for pod performance data reflects only the quotas, not the actual utilization, so maybe k8s knew (why isn't there a `kubetop` that can be run on the host to supervise?). I also hear that automatic rescheduling is still in draft stage, so if k8s detects high load on a kubelet, instead of moving the pod, it will just not assign new pods to it.
About addressing. Right now, we have dev, stage, and prod. We have a container that takes all traffic targeting the kubelet on port 80 and proxies it out to the pod using servicelb. It's set up to forward from http://app-label.env-name.example.com to a pod matching that label. It's annoying that we have to run an external container for this in the first place and do extra configuration on it, but it's also annoying that since servicelb is not aware of the beta label attempting to simulate hostnames, we can't individually address instances of apps. Is that how this is supposed to work, or is there a ready-built solution that makes this much easier that we're overlooking?
----
>I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
The best way to improve Kubernetes would be to:
a) simplify the terminology;
b) simplify the kubectl command, and particularly make it follow semi-standard Unix utility conventions (examples: -f should mean force or not be used (and yes, I know some other tools commit this sin too, but the association with "force" is nerve-wracking on a production environment), should probably be -r or -o; --from-file is ambiguous, and should be --read-literal or -r --raw or something like that);
c) simplify the configuration syntax and files;
d) provide simple ways to get real storage and real IPs (I haven't thoroughly investigated StatefulSets, so it may do this). I know that Google has long ago gone into a plane where these are irrelevant, but normal companies haven't, and IMO there's no reason they necessarily should;
e) provide simple backup and restoration methods (afaik, this involves multiple steps, including backing up the etcd cluster (not allowed to run on the master, or just conventionally doesn't?)).
My suggestion for distributed workloads is:
a) script your machine in Ansible or similar, and make a base image from that;
b) use your cloud provider of choice to deploy new instances under the circumstances considered necessary;
c) register the new node with haproxy or whatever sits at your front end (often a cloud-provided LB);
d) use conventional administrative and monitoring tooling to administer.
All of that is fully scriptable, in far less time than it takes to convert to Docker/k8s.
-----
>TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
Why is the above-recommended distributed systems strategy inadequate, even for people with more than 3 nodes who care about downtime? What's fundamentally wrong with it? What unique value does k8s bring to the table?
What's the k8s replacement for htop? tcpdump? df? and all the other utilities, and why are they better, and what benefit does k8s provide, that we didn't already have, that warrants giving up everything? I know it's not TRYING to replace all of that per se, but the practicalities make kubernetes the authority for such information, and you're not supposed to have to get onto the kubelet itself to diagnose these types of issues so you can't even use root on the Docker host to try and do some of this.
In our deployment, the main k8s jockey just kills pods if they're acting up and hopes that will magically fix it, because the devs don't want to get near the setup with a 10-foot pole. Before, they would SSH in and go over the issues. Now, since not only can't you SSH in but there's a convoluted process to get a shell, and then the container will not have any diagnosis tools in it anyway, it's very hard to have them collaboratively troubleshoot a problem we're seeing in the wild.
I can't tell you how many times I've had someone tell me "They must have a better answer for that; Google uses it after all...". These people are under the mistaken impression that if Google is using it, it must be robust and stable, when in fact the opposite is usually true, and I don't think Google tries to pretend it's not. It's probably just misinterpreted the signal from the developer community as meaningful approbation, when it's really just blindly following the cool kid on the block.
It's not that k8s isn't a neat thing. It's just that it's solving a problem that pretty much only Google had, and now everyone else is plummeting down the rabbit hole.
We've spent probably 1.5 man-years getting our infrastructure containerized and kubed up. We're not at 100% in prod yet but getting pretty close. It's clear that I'm really dumb but I haven't even been the main one doing this, so can't blame it on me. That's a huge project, and how does k8s/Docker justify the cost? It lets us cut down on AWS usage? Sure, but nowhere near the proportion of cost. It "makes it easier" to administer the cluster? Nope, not a chance. It's fun to kill pods but that's living on the edge, most apps written by non-Google can't take that. Solution? Write an "operator", which appears to be any random program that intercepts the k8s API and does stuff to support pods, etc. That sucks, why don't I just keep the init script that works fine?
One could argue that k8s is about resource utilization, which is sort-of-but-not-really-true. And that has a downside; we've essentially turned our environment into a shared host now, and despite k8s quotas, we still have pods that are bad neighbors; this is turning out to be quite difficult to control.
So I'm just really at a loss here. Google wanted it and they built it and it's working for them, that's great. That's an internal tool. How does this help anyone who wants to run a normal site this way? I haven't heard a single success story that had a moral or goal behind it other than "I'm cool like Google now too."
And on top of all of this, before k8s even comes into the picture, you have to make a Dockerfile and run your application in Docker, which is a PITA itself, in terms of stability, security, and configuration (see above-mentioned dockerd hammering the box, dockerd hangs or breakages that render a box useless, having to deal with port forwards, name collisions, large Docker caches, docker's inability to remove old images or containers on its own, Dockerfile lameness, culture of importing unknown images, etc.).
I like the ideas of containers (and have liked them when they were called "jails" in past eras ;) ) and I like the idea of orchestrators like kubernetes. But I think it's very early days right now for both, and that they will need radically improved simplification and stability before they are appropriate for general use.
Well you can simply use GKE and that is all managed for you. Seemed to work quite well for Pokemon Go as they scaled out massively.
"Have we made an unnecessarily contorted solution that you don't need to solve any of your real-world problems? Don't worry, you can still claim to be cool by paying us money." What a solution!
> Building "from scratch" off the base image distributed by the distro itself is considered "the old way"
Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.
Then I don't want to be new. I started using a few of the Docker Hub images (mostly `openjdk`), but I've transitioned completely to building all runtime images off of `debian:jessie`. And even that image I may want to remove eventually, just so that our CI system isn't affected by Docker Hub outages.
linkerd is written in Scala [1] and linkerd is written in Rust [2]. If he is optimizing for other people’s contributions, is this really the best way to go?
[1] https://github.com/linkerd/linkerd [2] https://github.com/linkerd/linkerd-tcp