euank · HackerTrans

10 ay öncediscuss

I can give an n=1 anecdote here: the dns resolver used to have hard-coded caching which meant that it would be unresponsive to pod updates, and cause mini 30s outages.

The code in question was: https://github.com/grpc/grpc-go/blob/b597a8e1d0ce3f63ef8a7b6...

That meant that deploying a service which drained in less than 30s would have a little mini-outage for that service until the in-process DNS cache expired, with of course no way to configure it.

Kuberesolver streams updates, and thus lets clients talk to new pods almost immediately.

I think things are a little better now, but based on my reading of https://github.com/grpc/grpc/issues/12295, it looks like the dns resolver still might not resolve new pod names quickly in some cases.

euank

4 yıl öncediscuss

Coincidentally, Google has sorta done that too!

ChromiumOS/ChromeOS is built using portage, and is certainly rolling release. It's not really a traditional linux distro, but it's something.

euank

4 yıl öncediscuss

> Does anyone know if and where I could file a bug report?

The linux kernel does take bug reports: https://docs.kernel.org/admin-guide/reporting-issues.html

However, that bug probably isn't specific enough as you've described it, unless you can find the commit causing it (such as via a git bisect https://docs.kernel.org/admin-guide/bug-bisect.html), or come up with a clearer repro.

Alternatively, if you're seeing the issue on a distro-maintained kernel (such as on fedora/ubuntu/debian with their kernel package), reporting the issue to the distro maintainers may be more appropriate.

euank

5 yıl öncediscuss

> Email has never had explicit threading

The "In-Reply-To" header is described in rfc2822. It is an explicit header in the RFC that is how you create threads.

Every mail client I've used correctly understands how to thread reply-chains using In-Reply-To.

The thing you're talking about, steam receipts grouping, is not a feature of email, but a specific feature of gmail's web view which is not mandated by any RFC and indeed is not explicit threading...

But there is a real way to thread which is defined in the RFC, and if you use a reasonable email client (aka not gmail), every mailing list's threading will work for you.

euank

6 yıl öncediscuss

The kubelet can talk to containerd's cri endpoint, yes, but there's one additional bit of complexity.

If someone wants to use kubelet + docker so that they can, for example, ssh into a node and type 'docker ps' to see containers, or have something else using the docker api see the containers the kubelet started, that won't work after re-pointing the kubelet from docker to containerd.

The difference here is namespacing[0], but not the linux-kernel-container-namespace, rather the containerd concept by the same name to allow "multi-tenancy" of a single containerd daemon.

In addition, I don't think you could have docker + cri run in the same containerd namespace since they end up using different networking and storage containerd plugins. I think that terminology is right.

So yeah, repointing the kubelet to containerd directly works fine, but it won't be the same thing as running docker containers.

[0]: https://github.com/containerd/containerd/blob/9561d938/docs/...

euank

6 yıl öncediscuss

For what it's worth, there are a few cases where docker vs some other runtime does make a difference.

One difference is that if you 'docker build' or 'docker load' an image on a node, with docker as a runtime a pod could be started using that image, but if containerd is the runtime it would have had to be 'ctr image import'ed instead.

I know that minikube, at some point, suggested people use 'DOCKER_HOST=..' + 'docker build' to make images available to that minikube node, which this would cause to not work.

It would be nice if k8s had its own container image store so you could 'kubectl image load' in a runtime agnostic way, but unfortunately managing the fetching of container images has ended up as something the runtime does, and k8s has no awareness of above the runtime.

Oh, and for production clusters, a distribution moving from dockerd to containerd could break a few things, like random gunk in the ecosystem that tries to find kubernetes pods by querying the docker api and checking labels. I think there's some monitoring and logging tools that do that.

If distributions move from docker to docker-via-a-cri-shim, that won't break either of those use cases of course.

euank

6 yıl öncediscuss

containerd can serve CRI requests itself. This has been the case since the containerd v1.1.0 release[0], which included the cri "plugin" as an in-process part of the containerd binary. For a while, to keep up the plugin idea, it was in a separate github repo too, but these days it's in the main containerd repo directly[1].

[0]: https://github.com/containerd/containerd/releases/tag/v1.1.0

[1]: https://github.com/containerd/containerd/tree/9561d9389d/pkg...

euank

6 yıl öncediscuss

I think that the title of this is a bit misleading.

Kubernetes is removing the "dockershim", which is special in-process support the kubelet has for docker.

However, the kubelet still has the CRI (container runtime interface) to support arbitrary runtimes. containerd is currently supported via the CRI, as is every runtime except docker. Docker is being moved from having special-case support to being the same in terms of support as other runtimes.

Does that mean using docker as your runtime is deprecated? I don't think so. You just have to use docker via a CRI layer instead of via the in-process dockershim layer. Since there hasn't been a need until now for an out-of-process cri->docker-api translation layer, there isn't a well supported one I don't think, but now that they've announced the intent to remove dockershim, I have no doubt that there will be a supported cri -> docker layer before long.

Maybe the docker project will add built-in support for exposing a CRI interface and save us an extra daemon (as containerd did).

In short, the title's misleading from my understanding. The Kubelet is removing the special-cased dockershim, but k8s distributions that ship with docker as the runtime should be able to run a cri->docker layer to retain docker support.

For more info on this, see the discussion on this pr: https://github.com/kubernetes/kubernetes/pull/94624

euank

6 yıl öncediscuss

> is there a name for this bias?

I think it's response bias. https://en.wikipedia.org/wiki/Response_bias

euank

6 yıl öncediscuss

You're right that nixos / all nix packages isn't/aren't perfectly reproducible.

In practice, most of the packages in the nixos base system seem to be reproducible, as tested here: https://r13y.com/

Naturally, that doesn't prove they are perfectly reproducible, merely that we don't observe unreproducibility.

Nix has tooling, like `nix-build --check`, the sandbox, etc which make it much easier to make things likely to be reproducible.

I'm actually fairly confident that the redis package is reproducible (having run `nix-build --check` on it, and seen it have identical outputs across machines), which is part of why I picked it as my example above.

However, I think my point stands. Dockerfiles make no real attempt to enforce reproducibility, and rarely are reproducible.

Nix packages push you in the right direction, and from practical observation, usually are reproducible.

euank

6 yıl öncediscuss

A Dockerfile is not a reproducible set of build instructions in most cases. I'd guess that the vast majority of Dockerfiles are not reproducible.

Let's look at an example dockerfile for redis (based on [0])

    FROM debian:buster-slim
    RUN apt-get update; apt-get install -y --no-install-recommends gcc
    RUN wget http://download.redis.io/releases/redis-6.0.6.tar.gz && tar xvf redis* && cd redis-6.0.6 && make install

(Note, modified from upstream for this example; won't actually build)

The unreproducible bits are the following:

1. FROM debian:buster-slim -- unreproducible, the base image may change

2. apt-get update && apt-get install -- unreproducible, will give a different version of gcc and other apt packages at different times

Those two bits of unreprodicble-ness are so core to the image, that they result in every other step not being reproducible either.

As a result, when you 'docker build' that over time, it's very unlikely you'll get a bit-for-bit identical redis binary at the other end. Even a minor gcc version change will likely result in a different binary.

As a contrast to this, let's look at a reproducible build of redis using nix. In nixpkgs, it looks like so [1].

If I want a reproducible shell environment, I simply have to pin down its dependencies, which can be done by the following:

    let
      pkgs = import (builtins.fetchTarball {
        url = "https://github.com/NixOS/nixpkgs/archive/48dfc9fa97d762bce28cc8372a2dd3805d14c633.tar.gz";
        sha256 = "0mqq9hchd8mi1qpd23lwnwa88s67ac257k60hsv795446y7dlld2";
      }) {};
    in pkgs.mkShell {
      buildInputs = [ pkgs.redis];
    }

If I distribute that nix expression, and say "I ran it with nix version 2.3", that is sufficient for anyone to get a bit-for-bit identical redis binary. Even if the binary cache (which lets me not compile it) were to go away, that nixpkgs revision expresses the build instructions, including the exact version of gcc. Sure, if the binary cache were deleted, it would take multiple hours for everything to compile, but I'd still end up with a bit-for-bit identical copy of redis.

This is true of the majority of nix packages. All commands are run in a sandbox with no access to most of the filesystem or network, encouraging reproducibility. Network access is mediated by special functions (like fetchTarball and fetchGit) which require including a sha256.

All network access going through those specially denoted means of network IO means it's very easy to back up all dependencies (i.e. the redis source code referenced in [1]), and the sha256 means it's easy to use mirrors without having to trust them to be unmodified.

It's possible to make an unreproducible nix package, but it requires going out of your way to do so, and rarely happens in practice. Conversely, it's possible to make a reproducible dockerfile, but it requires going out of your way to do so, and rarely happens in practice.

Oh, and for bonus points, you can build reproduible docker images using nix. This post has a good intro to how to play with that [2].

[0]: https://github.com/docker-library/redis/blob/bfd904a808cf68d...

[1]: https://github.com/NixOS/nixpkgs/blob/a7832c42da266857e98516...

[2]: https://christine.website/blog/i-was-wrong-about-nix-2020-02...

comments