Let Google do the patching with new managed base images(cloud.google.com)
cloud.google.com
Let Google do the patching with new managed base images
https://cloud.google.com/blog/products/containers-kubernetes/exploring-container-security-let-google-do-the-patching-with-new-managed-base-images
74 comments
I'm Tech Lead/maintainer for these images. Happy to answer any questions!
Will it be 18 or 24 months before Google discontinues this product offering? How long will customers who chose to use this offering have to move to another solution once Google discontinues it?
Genuinely curious here -- what alternatives are there to this service that are guaranteed beyond 18-24 months?
Why is 18.04 not an available-at-launch image choice?
This was a mistake - we have it ready and just forgot to publish it. I'll get it out ASAP!
I think Ubuntu 16.04 is used internally a lot with Google, as they have hardened it themselves. I believe GKE On-Prem will also be based on Ubuntu 16.04 for this reason.
Not that much anymore. https://www.zdnet.com/article/google-moves-to-debian-for-in-...
What do you all think of Cloud Native Buildpacks?* A slightly different approach to solving some of the same problems of handling dependency and security updates off-cycle from application updates.
* https://buildpacks.io/
* https://buildpacks.io/
I'm a pretty big fan of the new build pack spec. I haven't gotten a chance to really try it out in depth yet, but the layer-aware caching is a pretty elegant way to make fast builds possible.
There is a fair amount of ongoing contact between Google teams (like dlor's) and the folks working on/around Cloud Native Buildpacks at Pivotal and Heroku (including me). We are continuing to compare notes.
I think these images will fold up neatly into CNBs without much fuss, if they are packaged as stacks.
I think these images will fold up neatly into CNBs without much fuss, if they are packaged as stacks.
Great, I can see lots of potential.
What's the difference between this and say:
FROM ubuntu:16.04
Won't that pull the latest official Ubuntu image with that tag?
FROM ubuntu:16.04
Won't that pull the latest official Ubuntu image with that tag?
There are a few reasons you might want to use our version of these images:
- If you're running on GCP, pulling images from GCR will likely be much faster than Dockerhub given network locality.
- Content Trust. DockerHub images are maintained by a large team of upstream maintainers and volunteers. These images are fully managed by Google, and you can examine and audit the full supply chain from source to built image.
- If you're running on GCP, pulling images from GCR will likely be much faster than Dockerhub given network locality.
- Content Trust. DockerHub images are maintained by a large team of upstream maintainers and volunteers. These images are fully managed by Google, and you can examine and audit the full supply chain from source to built image.
I think you're listing wrong reasons to prefer Google's images, while there are some right reasons.
* Faster fetch - not a big deal IMO (edit: you could just mirror, you don't have to do the patching)
* Content trust - but trust to e.g. Canonical's Ubuntu is required anyway, right? And then, you have to trust Google.
Examples of reasons I would take as good:
* Does Google routinely do audit of original distro images, so that it's less of a risk to use community distro like Debian? It could.
* Is Google "manning" the work on the actual distributed software, accepting users' bugreports and then working with upstream to fix it? It could.
Even-more-downstream-distro is actually not a bad idea.
* Faster fetch - not a big deal IMO (edit: you could just mirror, you don't have to do the patching)
* Content trust - but trust to e.g. Canonical's Ubuntu is required anyway, right? And then, you have to trust Google.
Examples of reasons I would take as good:
* Does Google routinely do audit of original distro images, so that it's less of a risk to use community distro like Debian? It could.
* Is Google "manning" the work on the actual distributed software, accepting users' bugreports and then working with upstream to fix it? It could.
Even-more-downstream-distro is actually not a bad idea.
If you really wanted to contribute to the ecosystem you could have offered another vulnerability list to work against, work with upstream to deal with issues, and mirror the existing bits on GCR for your customers.
--- edit add disclosure statement
Disclosure: I work at Microsoft on Azure, previously at Docker on the core engine.
--- edit add disclosure statement
Disclosure: I work at Microsoft on Azure, previously at Docker on the core engine.
There's nothing about this that excludes that work as well.
But, there's nothing to say that the upstream distro would accept the changes. Google has more stringent requirements that a general distro, and you're choosing to follow those requirements (at the benefit of Google's automated scanning). It's totally your choice.
Disclosure: I work at Microsoft on Azure.
But, there's nothing to say that the upstream distro would accept the changes. Google has more stringent requirements that a general distro, and you're choosing to follow those requirements (at the benefit of Google's automated scanning). It's totally your choice.
Disclosure: I work at Microsoft on Azure.
> But, there's nothing to say that the upstream distro would accept the changes
If upstream didn't accept it, then it shouldn't be masquerading as what upstream is... which is what changing `FROM $image` to point to Google's image does.
> and you're choosing to follow those requirements
From what I gathered from the article, Google is changing the meaning of `FROM $image` to mean Google's build of the image. There is no choice in the matter outside of using Google's services.
Disclosure (and sorry for not mentioning in the original comment, I've updated): I work at Microsoft on Azure, and formerly at Docker on the core engine.
If upstream didn't accept it, then it shouldn't be masquerading as what upstream is... which is what changing `FROM $image` to point to Google's image does.
> and you're choosing to follow those requirements
From what I gathered from the article, Google is changing the meaning of `FROM $image` to mean Google's build of the image. There is no choice in the matter outside of using Google's services.
Disclosure (and sorry for not mentioning in the original comment, I've updated): I work at Microsoft on Azure, and formerly at Docker on the core engine.
Nothing is changing with FROM - not sure what gave you that impression. We're simply offering some new base images people can choose to use if they want to.
These also happen to be the ones that have been in use by some of our "more serverless" offerings, where users don't get to supply a Dockerfile, for some time now.
These also happen to be the ones that have been in use by some of our "more serverless" offerings, where users don't get to supply a Dockerfile, for some time now.
> As long as the FROM field in your Dockerfile points to `$distro:latest` from Cloud Marketplace
I read this perhaps the wrong way, in which case I apologize on that front.
I read this perhaps the wrong way, in which case I apologize on that front.
Is this similar to what Red Hat does with OpenShift?
They maintain official RHEL-based images available on their own catalog [1]
Some images are "s2i" (Source-to-Image) which are automatically built by merging a vetted base image and source code from a git repo [2].
[1] https://access.redhat.com/containers/
[2] https://docs.openshift.com/container-platform/3.11/architect...
They maintain official RHEL-based images available on their own catalog [1]
Some images are "s2i" (Source-to-Image) which are automatically built by merging a vetted base image and source code from a git repo [2].
[1] https://access.redhat.com/containers/
[2] https://docs.openshift.com/container-platform/3.11/architect...
I think it's closer to saying that Google will stand behind Centos and Ubuntu images without you needing to sign contracts with Red Hat or Canonical.
I work for Pivotal. We have a contract with Canonical that leads us to base all our work on Ubuntu, including derived base images, analogous to how RHEL is bundled into OpenShift.
If you're buying from us, this news is neutral. If you're buying from Red Hat, this news is neutral. If you're buying from neither, it's good news.
I work for Pivotal. We have a contract with Canonical that leads us to base all our work on Ubuntu, including derived base images, analogous to how RHEL is bundled into OpenShift.
If you're buying from us, this news is neutral. If you're buying from Red Hat, this news is neutral. If you're buying from neither, it's good news.
Hi, you have covered base images vs distroless images in your article.
A lot of organizations don't yet have the tooling required to maintain distroless images (since that tooling extends all the way to developer machines).
Have you considered a hybrid option, where in GKE, you automatically keep this base image layer auto-updated when running on cos? It could give organizations that don't have distroless capability to still use vanilla docker tooling, but still get the same benefits of distroless at runtime (and potential of increased adoption it might drive to GKE)?
A lot of organizations don't yet have the tooling required to maintain distroless images (since that tooling extends all the way to developer machines).
Have you considered a hybrid option, where in GKE, you automatically keep this base image layer auto-updated when running on cos? It could give organizations that don't have distroless capability to still use vanilla docker tooling, but still get the same benefits of distroless at runtime (and potential of increased adoption it might drive to GKE)?
Is there a dockerhub like interface where I can see all of the repos and available tags? I clicked several links in the blog but didn't see anything similar.
You can put the image tag URL into your browser's address bar and it'll redirect to the Cloud Console page listing tags for that image, e.g.:https://marketplace.gcr.io/google/ubuntu16_04
From there you can go up a level to see all images in the /google/ namespace: https://console.cloud.google.com/gcr/images/cloud-marketplac...
From there you can go up a level to see all images in the /google/ namespace: https://console.cloud.google.com/gcr/images/cloud-marketplac...
I'm trying to understand the value proposition here. Is it:
Today, these images are already safer than other images
or
In the future, these images have an SLA for updates and patches in case of future problems
(or, both?)
Today, these images are already safer than other images
or
In the future, these images have an SLA for updates and patches in case of future problems
(or, both?)
Just the fact that google is gatekeeping the images instead of dockerhub makes them a better security proposition.
It's 2018 and dockerhub _still_ doesn't have 2FA. All it takes is one cred-stuffed account belonging to a debian maintainer to take you down if you don't pin your images.
It's 2018 and dockerhub _still_ doesn't have 2FA. All it takes is one cred-stuffed account belonging to a debian maintainer to take you down if you don't pin your images.
Do you think this strategy could be successful/have a future for non-container based Operating Systems?
4.4.7 (unable to deliver this
What about windows containers?
Alpine?
"The CentOS managed base image uses `yum` and `rpm` for package management, and these pull RPM files only over HTTPS connections."
That's interesting. Is there a reason for that?
IIRC the stock CentOS doesn't use HTTPS for its yum/rpm repos and I figured it wasn't necessary to use HTTPS since the package signature is verified.
That's interesting. Is there a reason for that?
IIRC the stock CentOS doesn't use HTTPS for its yum/rpm repos and I figured it wasn't necessary to use HTTPS since the package signature is verified.
a MITM can trick your server into thinking there are no updates, and no error will be raised
That, and an attacker could do a replay attack, and potentially send you to an earlier (vulnerable) version.
What's the difference between this and pulling an official image from dockerhub? For example https://hub.docker.com/_/ubuntu/
It's betting that Google are better package maintainers than Debian/CentOS/Ubuntu, especially when it comes to maintaining container images. From a purely technical, and financial, point of view, it's likely true.
Of course, they could have bolstered the distro teams. And the fact that the repositories are within GCR, not Docker, is just convenience.
It brings you closer into Googles warm, technototalitarian embrace, is what I mean.
Of course, they could have bolstered the distro teams. And the fact that the repositories are within GCR, not Docker, is just convenience.
It brings you closer into Googles warm, technototalitarian embrace, is what I mean.
I would suggest you look a little deeper into Debian's history and work on reproducible builds: https://wiki.debian.org/ReproducibleBuilds
They've been at it (looks like) since ~2014, and their goals and motivations seem 100% in line with google, but at the package/distro level, not the container-base-image level (highly related): https://tests.reproducible-builds.org/debian/unstable/amd64/...
They've been at it (looks like) since ~2014, and their goals and motivations seem 100% in line with google, but at the package/distro level, not the container-base-image level (highly related): https://tests.reproducible-builds.org/debian/unstable/amd64/...
Exactly! This work builds on top of the Debian team's work on reproducible package builds, providing a way to combine all of these packages into a reproducible container image.
This is difficult with Docker directly today because timestamps appear all over the place. We designed and built a custom set of bazel rules to make this possible.
This is difficult with Docker directly today because timestamps appear all over the place. We designed and built a custom set of bazel rules to make this possible.
I came across a tool that will generate a bit-for-bit identical Debian container [1], using Debian package snapshots.
[1] https://github.com/debuerreotype/debuerreotype
[1] https://github.com/debuerreotype/debuerreotype
> goals and motivations seem 100% in line with google,
It's been a long time since I've believed in their idealism, yes, but I still think accusing Debian of being in the advertising and surveillance business is a bit harsh.
It's been a long time since I've believed in their idealism, yes, but I still think accusing Debian of being in the advertising and surveillance business is a bit harsh.
> Google are better package maintainers than Debian
My intent was to challenge your statement that google are better package maintainers than Debian, specifically w.r.t. reproducibility of builds.
It's disrespectful to Debian to think that they haven't been pushing for secure, auditable, trusted software running on trusted computers.
It's practically their reason for existing: taking open, auditable software, packaging it in reproducible fashion for use by anyone who wants it.
https://www.debian.org/social_contract.html#guidelines
Imagine if Debian had similar financial support available compared to Google/RedHat? The best info I could find is here: https://www.spi-inc.org/corporate/annual-reports/2017.pdf
`This covers the Period January 1, 2017 – December 31, 2017`
`Gross Income -------- 635,311.59`
...and in that way, yes: google can afford more package maintainers, more scrutiny, but if they are "better package maintainers" it's at those margins and due to economics rather than ability or desire.
My intent was to challenge your statement that google are better package maintainers than Debian, specifically w.r.t. reproducibility of builds.
It's disrespectful to Debian to think that they haven't been pushing for secure, auditable, trusted software running on trusted computers.
It's practically their reason for existing: taking open, auditable software, packaging it in reproducible fashion for use by anyone who wants it.
https://www.debian.org/social_contract.html#guidelines
Imagine if Debian had similar financial support available compared to Google/RedHat? The best info I could find is here: https://www.spi-inc.org/corporate/annual-reports/2017.pdf
`This covers the Period January 1, 2017 – December 31, 2017`
`Gross Income -------- 635,311.59`
...and in that way, yes: google can afford more package maintainers, more scrutiny, but if they are "better package maintainers" it's at those margins and due to economics rather than ability or desire.
> My intent was to challenge your statement that google are better package maintainers than Debian, specifically w.r.t. reproducibility of builds.
Ah, I believe there's a misunderstanding coming from misreading of my statement: what I wrote is that the _reason to upgrade_ would be the _belief_ that they are better, and that if you restrict yourself to _certain measures_, they probably can, by throwing more money at the problem. I hoped that he latter part of my comment probably makes it rather clear that Google wouldn't actually be better as far as I'm concerned. Even from purely technical perspective, I'm fairly sure Debian is willing to support many architectures Google will ignore.
Ah, I believe there's a misunderstanding coming from misreading of my statement: what I wrote is that the _reason to upgrade_ would be the _belief_ that they are better, and that if you restrict yourself to _certain measures_, they probably can, by throwing more money at the problem. I hoped that he latter part of my comment probably makes it rather clear that Google wouldn't actually be better as far as I'm concerned. Even from purely technical perspective, I'm fairly sure Debian is willing to support many architectures Google will ignore.
[deleted]
This is awesome - is there any thing similar for Azure? Or possible 3rd party solutions that do the same? We don’t leverage GCP but I am very envious of this feature. Would love the community to help point me in the right direction to get same functionality - mainly not having to maintain and patch Ubuntu 16.0.4 images
The good news is you don't have to use GCP! The very nice people over there have made them available for anyone -
https://cloud.google.com/container-registry/docs/managed-bas...
So, to be clear, no matter what cloud you use, if you create a Dockerfile like the following:
Disclosure: I work at Microsoft on Azure.
https://cloud.google.com/container-registry/docs/managed-bas...
So, to be clear, no matter what cloud you use, if you create a Dockerfile like the following:
FROM launcher.gcr.io/google/ubuntu16_04:latest
CMD echo "foobar"
Disclosure: I work at Microsoft on Azure.
You moved to MS?!? Your about section still mentions your Google address. Good luck!
Whoops! Updated :)
Thanks!
Thanks!
Thanks for pointing this out! Great news!
Yeah, it's called Red Hat Enterprise Linux ;-) It's available on all the cloud providers and has a better API/ABI guarantee as well as very, very well defined commitments to patching and lifecycle :-)
Does Google (or any of the other cloud vendors) audit/review the actual source code of packages used in the images? such as apache, nginx, openjdk, etc? or do they just run a scanner that test for known vulnerabilities?
wow. why would anyone want this? in a production environment you want tight control over software versions, not surprise updates.
I generally agree with you. However I also feel that the ABI guarantees by Red Hat, Canonical and the Debian project are fairly trustworthy. Rolling forward automatically is much less risky, taken overall, than a rotting pin.
"Let Google do the patching"
Also replace your shepherd dogs with wolves, wolves are bigger, faster, and will do it for free[*].
Also replace your shepherd dogs with wolves, wolves are bigger, faster, and will do it for free[*].
I don't understand this comment, what are you suggesting Google will do? Surreptitiously insert code into your binary during a patch?
If you are running in Google cloud, its their machines and they have power to do pretty much whatever they want anyway. How would this feature affect anything?
If you are running in Google cloud, its their machines and they have power to do pretty much whatever they want anyway. How would this feature affect anything?
Once you're dependent on their packaging process, why would they ever have to introduce whatever tracking or restriction they want "surreptitiously"? It'll be a part of their internal monitoring and diagnostics package.
This doesn't make any sense. What motives would Google have for providing an insecure service on their cloud hosting.
I'm all for Google-bashing in areas that actually make some sense but this is nonsensical.
I'm all for Google-bashing in areas that actually make some sense but this is nonsensical.
If you are into conspiracy theory's and such, then you can question if google and NSA really have a partnership and PRISM and mass surveillance and all that stuff, then question if the images are tainted or whatnot. I am not saying it is so by any stretch. I am just saying how i can understand the paranoid not trusting this 100%.
Sidestepping the conspiracy questions, these images are built reproducibly, from sources available on GitHub. You can clone the repo, audit the build tools and inputs, and build the image yourself. You'll get the exact same sha256 as the one on our registry.
Still makes no sense. If you're paranoid about it, why are you running on google's cloud?
LaGrange(1)