A Linux sysadmin's introduction to cgroups(redhat.com)
redhat.com
A Linux sysadmin's introduction to cgroups
https://www.redhat.com/sysadmin/cgroups-part-one
73 comments
cgroups were invented at Google in the early 2000's.
In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time.
Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.
In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time.
Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.
I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google.
[1] https://en.wikipedia.org/wiki/Cgroups
[2] https://lwn.net/Articles/199643/
[1] https://en.wikipedia.org/wiki/Cgroups
[2] https://lwn.net/Articles/199643/
That's interesting. OpenVZ (CC'd on the patch) and FreeBSD jails also pre-date mainstream containers on Linux. However, having been using Linux since the mid 1990s I feel confident in saying public use and development of container-like technologies on mainstream linux definitely dates from circa the LXC userspace and not before. This was ~2009-2010.
It's probably true that public use of such technology can be dated to LXC. But cgroups, and the initial (private) userspace implementations, definitely come from Google.
There were many efforts to get some kind of container technology into Linux, from the early 2000s (and probably earlier). VServer, OpenVZ, etc, were all trying to support full server virtualization.
Cgroups was different from most of them in that it wasn't trying to present a virtual server environment, just facilitate resource control. At Google, all the jobs we ran on Borg (the internal system that later inspired Kubernetes) knew that they were in a shared environment, so there was much less need to fake a private virtual environment. The low-level libraries linked into all Google binaries did a lot of coordination with Borg to make the sharing fairly painless. But we were trying to run tens of jobs on each of our increasingly-larger machines, and some of the batch jobs would inevitably end up hogging resources, which would hurt the important latency-sensitive jobs. CPU and memory isolation were the main things that we cared about, with network/disk as secondary elements.
For a while (in 2005-2007) in Borg we approximated this by using the kernel's "fake NUMA" support (originally intended for testing the NUMA code) to break up the system into a bunch of fake NUMA nodes, and using cpusets to reserve CPUs and memory chunks to important jobs. It was pretty ugly and rather coarse-grained, but it worked rather well and was the first widespread (running on millions of Google Linux servers) userspace system for controlling a cgroups-like system.
Essentially, cgroups piggy-backed on to the existing cpusets mechanism/API which had already been accepted into the kernel, and expanded it into a more generic way of creating a hierarchy of groups and mapping a process to a group. This was a lot simpler to get accepted than an entire virtualization system. Given that mapping, making other resource scheduling be group-aware rather than just process-aware was much more straightforward. The same userspace support in Borg that had been used for controlling cpusets worked pretty much as-is with cgroups, since the basic API was the same (just more resources were supported).
There were many efforts to get some kind of container technology into Linux, from the early 2000s (and probably earlier). VServer, OpenVZ, etc, were all trying to support full server virtualization.
Cgroups was different from most of them in that it wasn't trying to present a virtual server environment, just facilitate resource control. At Google, all the jobs we ran on Borg (the internal system that later inspired Kubernetes) knew that they were in a shared environment, so there was much less need to fake a private virtual environment. The low-level libraries linked into all Google binaries did a lot of coordination with Borg to make the sharing fairly painless. But we were trying to run tens of jobs on each of our increasingly-larger machines, and some of the batch jobs would inevitably end up hogging resources, which would hurt the important latency-sensitive jobs. CPU and memory isolation were the main things that we cared about, with network/disk as secondary elements.
For a while (in 2005-2007) in Borg we approximated this by using the kernel's "fake NUMA" support (originally intended for testing the NUMA code) to break up the system into a bunch of fake NUMA nodes, and using cpusets to reserve CPUs and memory chunks to important jobs. It was pretty ugly and rather coarse-grained, but it worked rather well and was the first widespread (running on millions of Google Linux servers) userspace system for controlling a cgroups-like system.
Essentially, cgroups piggy-backed on to the existing cpusets mechanism/API which had already been accepted into the kernel, and expanded it into a more generic way of creating a hierarchy of groups and mapping a process to a group. This was a lot simpler to get accepted than an entire virtualization system. Given that mapping, making other resource scheduling be group-aware rather than just process-aware was much more straightforward. The same userspace support in Borg that had been used for controlling cpusets worked pretty much as-is with cgroups, since the basic API was the same (just more resources were supported).
But if your talking about system virtualization, System 370 invented that way before FreeBSD invented jails...and speaking of jails..even chroot on BSD came before that, but he is talking about cgroups.
If you're talking about what I think you're talking about, IBM's VM is a hypervisor, which is a world away from cgroups, chroot, jails, or, say, the Java VM, before anyone brings that up.
I talk about the CP/CMS Operating-system, CP-370 later VM/370 "virtualized or containerized" just the OS and not Hardware (for accounting and Security reasons):
https://en.wikipedia.org/wiki/CP/CMS#Overview
that's exactly what jails/zones do (System, NOT Hardware virtualization)
https://en.wikipedia.org/wiki/CP/CMS#Overview
that's exactly what jails/zones do (System, NOT Hardware virtualization)
[deleted]
No, it isn't, and you don't understand one or the other or both.
Indeed from the OpenVZ wiki's history section:
>"Nov 1999: Alexander Tormasov visited Singapore and proposed a new direction to Sergey Beloussov: container virtualization. He formulated three main components: containers as a set of processes with namespace isolation, file system to share code/ram and isolation in resources. Indeed it was 1999 when our engineers started adding bits and pieces of containers technology to Linux kernel 2.2. Well, not exactly "containers", but rather "virtual environments" at that time -- as it often happens with new technologies, the terminology was different (the term "container" was coined by Sun only five years later, in 2004)."[1]
[1] https://wiki.openvz.org/History
>"Nov 1999: Alexander Tormasov visited Singapore and proposed a new direction to Sergey Beloussov: container virtualization. He formulated three main components: containers as a set of processes with namespace isolation, file system to share code/ram and isolation in resources. Indeed it was 1999 when our engineers started adding bits and pieces of containers technology to Linux kernel 2.2. Well, not exactly "containers", but rather "virtual environments" at that time -- as it often happens with new technologies, the terminology was different (the term "container" was coined by Sun only five years later, in 2004)."[1]
[1] https://wiki.openvz.org/History
There's also Linux Vserver (2001), which used to be included in Debian, but is still in use today.
It's important to realize that Linux offers a-la-carte container features which evolved over time. Some of the features include chroot, process groups, user/group permissions, cgroups, network multiplexing (e.g. inetd, nat), namespaces (network, mount, pid, user, etc.) and unshare/setns, union file systems like unionfs/aufs/overlayfs, etc.. There are a lot of ways you can implement container orchestration systems on Linux.
And as noted, OpenVZ and Linux Vserver were earlier container subsystems for Linux.
And as noted, OpenVZ and Linux Vserver were earlier container subsystems for Linux.
And HP-UX Vaults (I used them in 2000) and Tru64 containers as well.
Yeah, I think you're right about the BSD jails point. This totally pre-dates the Linux piece. I was only chatting about the Linux piece.
I am pretty sure that OpenVZ also predates the Google work in a Linux context.
Basically saying that 'containers on Linux' are 'made at Google' is not true.
Basically saying that 'containers on Linux' are 'made at Google' is not true.
[deleted]
But the OP doesn't state containers on linux were invented at Google. It says cgroups were.
[deleted]
BSD jails date back to FreeBSD 4, which was released in 2000.
Edit: I started questioning my memory of this, so I poked around some. Poul-Henning Kamp did the initial work in April, 1999, here is the commit:
https://svnweb.freebsd.org/base?view=revision&revision=46155
More details from him about it:
http://phk.freebsd.dk/sagas/jails/
Edit: I started questioning my memory of this, so I poked around some. Poul-Henning Kamp did the initial work in April, 1999, here is the commit:
https://svnweb.freebsd.org/base?view=revision&revision=46155
More details from him about it:
http://phk.freebsd.dk/sagas/jails/
Awesome, thanks _jal!
Mid 90s ran some vm system of MSDOS. It added threading and was for phone systems.
Solaris Zones is another Unix container system, ~2004-2005.
Intersting Linux history - Do you by a chance have a link to the commit ?
I picked a file from the patch index mail linked above, mm_inline.h, and went scrolling through its history at https://github.com/torvalds/linux/commits/master/include/lin... but didn't see a corresponding change there. I guess the patches might have gotten refactored before merging or something too but would be nice to have a pointer in the Linux tree history that would work as a reference.
edit: tried looking for linux/container.h too but that just brings up a newer acpi container related container.h: https://github.com/torvalds/linux/commits/master/include/lin...
edit 2: history for linux/cgroup.c starts in a year later in 2007, with this commit: https://github.com/torvalds/linux/commit/ddbcc7e8e50aefe467c... - it has mentions of people from OpenVZ, IBM and a bunch of unaffiliated domains in addition to the signed-off-by line from a Googler (Paul Menage).
I picked a file from the patch index mail linked above, mm_inline.h, and went scrolling through its history at https://github.com/torvalds/linux/commits/master/include/lin... but didn't see a corresponding change there. I guess the patches might have gotten refactored before merging or something too but would be nice to have a pointer in the Linux tree history that would work as a reference.
edit: tried looking for linux/container.h too but that just brings up a newer acpi container related container.h: https://github.com/torvalds/linux/commits/master/include/lin...
edit 2: history for linux/cgroup.c starts in a year later in 2007, with this commit: https://github.com/torvalds/linux/commit/ddbcc7e8e50aefe467c... - it has mentions of people from OpenVZ, IBM and a bunch of unaffiliated domains in addition to the signed-off-by line from a Googler (Paul Menage).
I think the patch never made it trough mainline and the work on the cgroups framework started. with this patch.
It basically says:
It basically says:
Based originally on the cpuset system, extracted by Paul Menage
* Copyright (C) 2006 Google, Inc
* Copyright notices from the original cpuset code:
* --------------------------------------------------
* Copyright (C) 2003 BULL SA.
* Copyright (C) 2004-2006 Silicon Graphics, Inc.
*
* Portions derived from Patrick Mochel's sysfs code.
* sysfs is Copyright (c) 2001-3 Patrick Mochel
*
* 2003-10-10 Written by Simon Derr.
* 2003-10-22 Updates by Stephen Hemminger.
* 2004 May-July Rework by Paul Jackson.
I believe this is the original whitepaper from 2007.
"Adding Generic Process Containers to the Linux Kernel" https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pd...
Although the paper was written by Paul B. Menage it looks like Balbir Singh and Srivatsa Vaddagiri from IBM also made contributions to it.
"Adding Generic Process Containers to the Linux Kernel" https://www.kernel.org/doc/ols/2007/ols2007v2-pages-45-58.pd...
Although the paper was written by Paul B. Menage it looks like Balbir Singh and Srivatsa Vaddagiri from IBM also made contributions to it.
Kubernetes is just another layer on top of Docker. It can use other container execution environments, but Kubernetes itself doesn't manage the container runtime; it just orchestrates containers at a higher level. If you're running Kubernetes, you're almost always still running Docker.
Yeah, funny how it all worked out. cgroups/namespaces powered simple LXC containers. Docker comes along and makes a nice workflow and package management system (wrapping cgroups/namespaces). Kubernetes comes along and makes a nice workflow/cluster management layer (wrapping Docker). Cloud providers come along and make a nice Kubernetes management layer (wrapping Kubernetes). Pretty crazy to see the evolution over the past few years.
This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement.
Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.
This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement.
Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.
"threw tons of gas on the whole devops movement"
I'm happy to find that I agree with you on at least one area of this new ecosystem :)
I'm happy to find that I agree with you on at least one area of this new ecosystem :)
This is literally my job and I kind of hate how unnecessarily complex it all is. Not in the sense of "distributed systems are hard", but that we don't actually need the onion to have 20 layers, we really only need 2 or 3.
The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves)
All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months".
The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
The reason we need an orchestration system for our orchestration system, is the orchestration system is a snowflake. We need an orchestration system because the container system is a snowflake. We need a container system because containers are a snowflake. None of it is really standard or easy to implement, because none of it was designed in the spirit of Unix. It was all just some random companies who threw some crap together so they could start making money selling ads or hosting services. (I don't mean cgroups, I mean the tools that use them, although cgroups are kind of warty in themselves)
All most people need to do is run a couple processes on a couple hosts, send packets between services (and networks), and have something start/stop/restart those processes. You can have all of that with, say, systemd (ugh), or three different standard interfaces that any program can use with standard libraries/ABIs. Notice I didn't say "with 5 different daemons running on 3 hosts that need complex configuration, constant maintenance, and a migration effort every 3 months".
The Open Container Initiative seems close to getting the first thing done. If we get the other two standardized, we can lose a whole bunch of the onion layers. Consequently a lot of people will need to find a new way to make money, because there'll be not nearly as much need to pay people to deal with the onion.
The way you described it one has to wonder what would the tech landscape look like had Plan 9 received adoption instead of Linux, so you didn’t have to invent all those layers of complexity to manage services on multiple hosts. Just mount the network interface, cpu, ram and start multiple processes...
Nah, we'd still find ways to make it complex. I mean, yes, I think Plan 9 would've been a nicer base to work from and maybe reduced it a bit, but people are excellent at making things more complex, sometimes for legitimate reasons. Off the top of my head, I don't think Plan 9 has a good "orchestration" story itself (ex. "run 17 instances of this process wherever there's enough free resources in our 100-server cluster"), and while the "Plan 9 k8s" of this alternate universe would be simpler than the k8s we have in reality, it would still exist and add one or more layers of abstraction, and then people would stick management layers in front of that, and...
> I don't think Plan 9 has a good "orchestration" story itself
I'm not super familiar with plan 9, but given what I do know, I'm pretty sure that could be done with an rc script. I'm pretty sure that a) you can fork processes after starting them on remote systems, and b) you can query remote systems to figure out which ones have free resources. Given that, it's just a shell / rc script away. Then you just need another script to do the management. ;)
I'm not super familiar with plan 9, but given what I do know, I'm pretty sure that could be done with an rc script. I'm pretty sure that a) you can fork processes after starting them on remote systems, and b) you can query remote systems to figure out which ones have free resources. Given that, it's just a shell / rc script away. Then you just need another script to do the management. ;)
As another example of something orchestration-like ("run 17 processes where possible on this 100 node cluster"), there were also OpenMOSIX and Beowulf clusters on Linux, which if I remember right would load balance work across available systems, in the same way that the kernel's scheduler balances work across CPU cores.
https://en.wikipedia.org/wiki/OpenMosix
https://en.wikipedia.org/wiki/OpenMosix
Yes. Mosix was the kind of black magic that makes you think software today is still several generations behind the times (like Plan9 before it). Even its distributed network filesystem is superior in ease of use to all others. But the devil's in the details, and distributing threads and shared memory is just really difficult, and the patchsets never made it into vanilla. So there goes yet another amazing design down the drain.
It reminds me of how electric cars are a 190 year old technology whose actual peak popularity was 120 years ago. Yet here we are, with manufacturers predicting they'll finally be popular again, in 10 to 15 years.
It reminds me of how electric cars are a 190 year old technology whose actual peak popularity was 120 years ago. Yet here we are, with manufacturers predicting they'll finally be popular again, in 10 to 15 years.
Here is the secret, one also doesn't need to follow IT fashion with every fad that pops up.
Still delivering on VMs, might be out of fashion, but it works, everyone gets it, and I get back home on time.
Still delivering on VMs, might be out of fashion, but it works, everyone gets it, and I get back home on time.
The funny thing is that one of the complaints many people have about the cgroups API (text-based control files in a virtual filesystem, which came originally from the cpusets system) is very much in style of Plan 9.
> All most people need to do is run a couple processes on a couple hosts … You can have all of that with, say, systemd (ugh)
Why ugh? I do have all of that with systemd, and it’s great. Rather than running each service in a separate filesystem and requiring them to communicate over a virtual network, I use ProtectSystem and ProtectHome to keep services as isolated as they need to be. Rather than creating a network bridge and doing NAT inside my server, if services require network isolation, I use NetworkNamespacePath to assign them to namespaces that contain different VLAN interfaces, which are configured in my router just like physical interfaces. Rather than building a huge container image for every application, I use my OS’s package manager to install apps and manage their dependencies. The skills required to do this can be applied to any systemd-based Linux distribution, and are needed to properly understand and troubleshoot systemd-based Linux distributions running in containers anyway. I’d need some new tools (eg. Ansible) to scale this setup past one or two hosts, but I’m baffled by the people setting up complex Docker/Kubernetes systems just to self-host a few web apps.
Why ugh? I do have all of that with systemd, and it’s great. Rather than running each service in a separate filesystem and requiring them to communicate over a virtual network, I use ProtectSystem and ProtectHome to keep services as isolated as they need to be. Rather than creating a network bridge and doing NAT inside my server, if services require network isolation, I use NetworkNamespacePath to assign them to namespaces that contain different VLAN interfaces, which are configured in my router just like physical interfaces. Rather than building a huge container image for every application, I use my OS’s package manager to install apps and manage their dependencies. The skills required to do this can be applied to any systemd-based Linux distribution, and are needed to properly understand and troubleshoot systemd-based Linux distributions running in containers anyway. I’d need some new tools (eg. Ansible) to scale this setup past one or two hosts, but I’m baffled by the people setting up complex Docker/Kubernetes systems just to self-host a few web apps.
Systemd is basically another Kubernetes. The Unix way does not require a patriarchal set of daemons and a proprietary framework to manage the interoperability of everything else in a way that's incompatible with anything else. That said, it's still several layers less than K8s and its dependencies are today.
I'd render a guess that the 'ugh' has to do with the general assessment that systemd falls prey to the same flaws that GP identified in the onion layers of orchestration and containerization systems: it's unnecessarily complicated.
> If you're running Kubernetes, you're almost always still running Docker.
Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.
Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.
Podman is meant as a replacement for the docker command as it’s typically used by developers and for bespoke services that can use systemd for lifecycle management. podman is an ergonomic runc wrapper.
Redhat’s answer to capital D Docker is cri-o.
Redhat’s answer to capital D Docker is cri-o.
To add to this, cri-o and libpod/podman are increasingly sharing code.
> If you're running Kubernetes, you're almost always still running Docker.
I'm not sure that's true - a lot of Kubernetes system use containerd these days.
I'm not sure that's true - a lot of Kubernetes system use containerd these days.
I think a lot of documentation still tends to suggest docker as a default, but containerd actually makes more sense.
Why use something higher level than you need that wraps containerd (namely Docker) instead of something that is exactly at the level you want: containerd itself? The only real advantage of wrapping docker is that doing so ensures the docker cli works for introspecting containers.
The Docker folks don't want people to be wrapping dockerd, because then those infrastructure folk complain about changes/additional features in dockerd. Even dockerd is (glacially) moving in the direction of move heavily using containerd, for example using it to pull/store images.
The longer-term intent appears to be for dockerd being more for implementing docker specific stuff like its networking, swarm/stack, and various features needed by the front-end.
Why use something higher level than you need that wraps containerd (namely Docker) instead of something that is exactly at the level you want: containerd itself? The only real advantage of wrapping docker is that doing so ensures the docker cli works for introspecting containers.
The Docker folks don't want people to be wrapping dockerd, because then those infrastructure folk complain about changes/additional features in dockerd. Even dockerd is (glacially) moving in the direction of move heavily using containerd, for example using it to pull/store images.
The longer-term intent appears to be for dockerd being more for implementing docker specific stuff like its networking, swarm/stack, and various features needed by the front-end.
The 4th part[1] of this tutorial series also add a sysadmincasts.com video in the `Sources` section
[1]https://www.redhat.com/sysadmin/cgroups-part-four
[1]https://www.redhat.com/sysadmin/cgroups-part-four
Fantastic videos! Thanks for the links!
Yes, Kubernetes uses cgroup too.
Systemd units are also based on cgroups.
Yes, Kubernetes uses cgroup too.
Systemd units are also based on cgroups.
Man your site is GREAT. Super clear videos. Good stuff.
These videos were absolutely amazing. Thank you.
Dude, those videos are awesome!
Thank you.
Thank you.
These things should strive for accuracy and this article is not accurate. The cgroups facility does not control "the number of CPU shares per process." Although you can put such a thing into effect with control groups, it's more accurate to say that a control group limits the resources of a set of tasks. Those tasks may be from one or several processes, and it's also the case that a single process can divide its own tasks into several cgroups.
Hey webster, whats a task?
While I agree that we should strive for accuracy, you're really splitting hairs there. Plus the author puts a disclaimer in there "NOTE: This is a gross simplification and is NOT technically accurate should you be looking to get involved with the underlying cgroup code. The above explanation is meant for clarity of understanding."
But the explanation does not contribute to clarity of understanding. This has nothing to do with hacking on cgroups itself. As a user of cgroups you should know that one group may control multiple processes, and one process may be controlled by multiple groups! There certainly is not a 1:1 correspondence.
It’s not an internal detail, it’s fundamental to understanding it from a user perspective.
This is Part I from a series.
Part II: https://www.redhat.com/sysadmin/cgroups-part-two
Part III: https://www.redhat.com/sysadmin/cgroups-part-three
Part IV: https://www.redhat.com/sysadmin/cgroups-part-four
Someone managed to apply part three ? Systemd gets in the way for me.
Edit: well, at least part four, finally, properly document how to cope with systemd. And guess what, it is different than 90% of documentation on cgroups on the internet.
Edit: well, at least part four, finally, properly document how to cope with systemd. And guess what, it is different than 90% of documentation on cgroups on the internet.
Found an interesting ongoing blogpost series that also tries to explain all the other low level kernel mechanism that make up Docker and other container technologies on Linux. They haven't reach the topic of control groups though.
https://www.schutzwerk.com/en/43/posts/linux_container_intro...
https://www.schutzwerk.com/en/43/posts/linux_container_intro...
Still work in progress and the topics are not too in-depth... but it's interesting, thanks.
I wrote a blog post on cgroups a couple years ago that's still accurate and goes further into depth and gives workable examples of using them both inside and outside of containers: https://www.grant.pizza/blog/understanding-cgroups/
[deleted]
I like the cgroups approach because it seems to logically build on the established idea of process groups.
I also tend to think that processes in cgroups are a sweet spot of lightweight containerization that can do quite a bit.
I also tend to think that processes in cgroups are a sweet spot of lightweight containerization that can do quite a bit.
systemd uses cgroups so nearly every Linux box is using them.
so great info
Conspiracy theory. our kubernetes clusters as a service are not really running on VMs.
Ah yes, "cgroups", according to notable no-nonsense kernel hacker Al Viro:
"it's not just badly written kernel/cgroup.c - the interfaces on both sides (userland and the rest of kernel) are seriously misdesigned. As far as I'm concerned, configuring it out solves my problem nicely."
That was in 2011, so things might have improved. What remains however is that cgroups was added to the kernel, by Googlers, for easier maintenance, but with an implicit understanding that no sane person would actually make use of it to do something important.
... enter SystemD.
"it's not just badly written kernel/cgroup.c - the interfaces on both sides (userland and the rest of kernel) are seriously misdesigned. As far as I'm concerned, configuring it out solves my problem nicely."
That was in 2011, so things might have improved. What remains however is that cgroups was added to the kernel, by Googlers, for easier maintenance, but with an implicit understanding that no sane person would actually make use of it to do something important.
... enter SystemD.
The whole google is built on cgroups v1, which seems to stand as an existence proof that these interfaces are not useless.
"bad design" and "useful" can be overlapping groups
so when someone is using gmail on a macOS system, how are they using cgroups in any way? how is "the whole google" built on this? perhaps you means "google's web-facing servers using cgroups extensively" ?
That is a somewhat weird and dare I say pointless take. Thousands of machines are involved in serving every request from a web browser to gmail. Of those, only the web browser might not be in linux cgroups.
It's true that a better userspace API for cgroups might have involved something more like ioctl(), rather than a filesystem with text-based control files. But people had been trying to get such APIs into the kernel for years with no success. Which is why we tried to piggyback on something existing.
Cpusets, which had already been accepted into the kernel a year or two previously, provided the basic userspace API for cgroups. It was just expanded to support control files for more resource types. (And multiple resource hierarchies, although that's gone in cgroups v2 I think).
Cpusets, which had already been accepted into the kernel a year or two previously, provided the basic userspace API for cgroups. It was just expanded to support control files for more resource types. (And multiple resource hierarchies, although that's gone in cgroups v2 I think).
A personal note here too. cgroups were invented at Google in the early 2000's. If you're using search today, gmail, docs, maps, etc. You're using cgroups. It sounds simple but this tech really powered a whole wave of innovation/startups/projects that almost everyone interacts with on a daily basis. Either through touching a Google service or interacting with anyone using Docker or Kubernetes (running on cgroups/namespaces). Pretty impressive.
#14 - Introduction to Linux Control Groups (Cgroups)
https://sysadmincasts.com/episodes/14-introduction-to-linux-...
#24 - Introduction to Containers on Linux using LXC
https://sysadmincasts.com/episodes/24-introduction-to-contai...