Linux Kernel 5.14(lwn.net)
lwn.net
Linux Kernel 5.14
https://lwn.net/Articles/867706/
23 comments
It's worth noting that it isn't a magic bullet - it only hardens against certain classes of kernel flaw. If the kernel is compromised, it is trivial to extract the secrets (this may seem obvious, but the strength of the feature has been over-hyped by some publications)
https://github.com/JonathonReinhart/nosecmem
https://github.com/JonathonReinhart/nosecmem
Yeah, I'm kinda unclear on exactly what it prevents. Shouldn't the kernel, if not compromised, be preventing these mappings anyway?
I'm speculating a bit here, but I think it could guard against side-channel attacks against the kernel.
e.g. if you had a sidechannel that leaks data from the kernel, the kernel can't disclose data that it can't actually read.
e.g. if you had a sidechannel that leaks data from the kernel, the kernel can't disclose data that it can't actually read.
Linux has a "direct map" of all physical memory in kernelspace, which I guess makes things easier. See https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt "direct mapping of all physical memory (page_offset_base)". (I think this is not uncommon: JOS, which my undergrad operating systems class used, does this.)
Memory allocated with memfd_secret gets removed from the direct map, which means that from any other process's address space, there is no virtual address (regardless of memory protection) that corresponds to those physical addresses. So any exploit that gets only gets you memory read/write but not kernelspace code execution (various types of wild pointers, etc., as well as whatever exciting speculative execution vulnerability will undoubtedly show up next year) cannot compromise the data in a memfd_secret mapping.
Memory allocated with memfd_secret gets removed from the direct map, which means that from any other process's address space, there is no virtual address (regardless of memory protection) that corresponds to those physical addresses. So any exploit that gets only gets you memory read/write but not kernelspace code execution (various types of wild pointers, etc., as well as whatever exciting speculative execution vulnerability will undoubtedly show up next year) cannot compromise the data in a memfd_secret mapping.
> any exploit that gets only gets you memory read/write but not kernelspace code execution
That's an odd assumption. There are still a lot of universal tricks for turning a truly arbitrary R/W primitive into code execution, e.g. just write to the page table.
OTOH, this does guard against arbitrary read primitives, which, as you mentioned, is especially useful in post-Spectre days.
That's an odd assumption. There are still a lot of universal tricks for turning a truly arbitrary R/W primitive into code execution, e.g. just write to the page table.
OTOH, this does guard against arbitrary read primitives, which, as you mentioned, is especially useful in post-Spectre days.
Ah, okay. I get it. There has been some loose talk to reduce the amount of memory mapped in the direct map for openbsd.
Attacks on s/w and h/w systems are asymmetric warfare. Until the ptrace or other attacks are scripted, the barrier to entry is usefully high. Once they are scripted, it can be close to zero, depending. OTOH if the ptrace and other approaches leave a signature, then you remain at risk but able to know. And, it has wheel-of-life qualities as your attacker codes to hide the signature, you code to keep the visibility open, and to try and defeat the underlying attack. And so it goes.
Is there something bad happening with kernelnewbies? They're behind on 5.13 and 5.14 is just a placeholder at the moment.
I think it is often a kernel version or two behind.
Hopefully just taking some time off for the holidays
I'm very excited for the better CFS quota enforcement mechanism in 5.14 (https://lwn.net/Articles/844976/).
Kubernetes implements CPU and memory limits for containers (https://kubernetes.io/docs/concepts/configuration/manage-res...). The memory limit works the way you'd expect: there's a memory cgroup, and Linux implements the memory-allocation algorithm based on the memory cgroup's limit and not the total system memory. Wildly-overboard allocations get denied; somewhat-overboard allocations are permitted if you have overcommit turned on, but cause an OOM kill if you try to use all of it. This is at least expected behavior for existing non-Kubernetes applications and (more importantly) for people used to non-Kubernetes application development/deployment: it's just like if you were running on a Linux system with the limited memory.
CPU limits are much wackier. Because CPUs are a discrete resource and Kubernetes supports fractional CPU allocations, they can't use CPU pinning. Instead, they use the CFS quota controls, which aren't commonly used outside Kubernetes as far as I can tell. The quota is expressed as basically a fraction: for a process with, say, 0.5 vCPUs allocated, it gets to use 50 ms of CPU time every 100 ms of real time. The way it works at the kernel level, as best as I've been able to understand, is that after a process runs on the CPU for a bit, the kernel looks at the quota and sees if it has exceeded the quota. If it has, the process becomes unschedulable until it has "re-earned" sufficient quota. It does not slow down or proactively de-schedule the process, analogous to how it would have run on a slower or fewer-CPU computer.
So, if you have multiple processes or threads running in your container, and there are free cores for them to run, you can easily use up multiple seconds of CPU time in those 100 ms. Then you have to wait to re-earn your quota in 50 ms chunks before your process can run. And all the while the process appears to be runnable, not stuck, it just seems like the kernel is never choosing to schedule it.
We have seen, in production, processes stalled for minutes because they evidently ran so far past their quota. We have seen entire system deadlocks when such a stalled process is holding some kernel lock for network I/O, in our case, Ceph (if the kernel releases the process into interruptible sleep, then that process can still hold locks; it will release those locks if signaled and rescheduled, but in this case the process isn't being scheduled at all).
And we're far from the only people who have seen awful behavior from CFS quotas. Tim Hockin, Kubernetes co-creator and tech lead, advises people to just never use the CPU limit feature! https://twitter.com/thockin/status/1134193838841401345 We have done as he advises and just turned it off cluster-wide with --cpu-cfs-quota=false - it helps the Kubernetes scheduler if you set your CPU/memory limits equal to your requests, so this gets us the right Kubernetes behavior without incurring the wacky kernel behavior. If you Google that flag you can find many others who have done the same in their public configurations. There's a Kubernetes feature to use CPU pinning instead, which mentions improved performance as an advantage https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-... , but it only works if you're scheduling whole numbers of CPUs always.
A whole lot of applications on our Kubernetes clusters - lightly-accessed web servers, monitoring daemons, etc. - are best described as wanting 0.1 or 0.5 CPUs, in the sense that on (say) an 8-core machine, you could run many dozens of instances of these applications just fine. But that's not how the existing CFS quota mechanism interprets 0.1 or 0.5 CPUs. It seems like the "burst" mechanism here is likely to handle processes that, in a random 100 ms period, actually do run for more than 10 ms, but on an ongoing basis don't average out to using more than one tenth of a CPU. We might be able to turn --cpu-cfs-quota back on as a result.
Kubernetes implements CPU and memory limits for containers (https://kubernetes.io/docs/concepts/configuration/manage-res...). The memory limit works the way you'd expect: there's a memory cgroup, and Linux implements the memory-allocation algorithm based on the memory cgroup's limit and not the total system memory. Wildly-overboard allocations get denied; somewhat-overboard allocations are permitted if you have overcommit turned on, but cause an OOM kill if you try to use all of it. This is at least expected behavior for existing non-Kubernetes applications and (more importantly) for people used to non-Kubernetes application development/deployment: it's just like if you were running on a Linux system with the limited memory.
CPU limits are much wackier. Because CPUs are a discrete resource and Kubernetes supports fractional CPU allocations, they can't use CPU pinning. Instead, they use the CFS quota controls, which aren't commonly used outside Kubernetes as far as I can tell. The quota is expressed as basically a fraction: for a process with, say, 0.5 vCPUs allocated, it gets to use 50 ms of CPU time every 100 ms of real time. The way it works at the kernel level, as best as I've been able to understand, is that after a process runs on the CPU for a bit, the kernel looks at the quota and sees if it has exceeded the quota. If it has, the process becomes unschedulable until it has "re-earned" sufficient quota. It does not slow down or proactively de-schedule the process, analogous to how it would have run on a slower or fewer-CPU computer.
So, if you have multiple processes or threads running in your container, and there are free cores for them to run, you can easily use up multiple seconds of CPU time in those 100 ms. Then you have to wait to re-earn your quota in 50 ms chunks before your process can run. And all the while the process appears to be runnable, not stuck, it just seems like the kernel is never choosing to schedule it.
We have seen, in production, processes stalled for minutes because they evidently ran so far past their quota. We have seen entire system deadlocks when such a stalled process is holding some kernel lock for network I/O, in our case, Ceph (if the kernel releases the process into interruptible sleep, then that process can still hold locks; it will release those locks if signaled and rescheduled, but in this case the process isn't being scheduled at all).
And we're far from the only people who have seen awful behavior from CFS quotas. Tim Hockin, Kubernetes co-creator and tech lead, advises people to just never use the CPU limit feature! https://twitter.com/thockin/status/1134193838841401345 We have done as he advises and just turned it off cluster-wide with --cpu-cfs-quota=false - it helps the Kubernetes scheduler if you set your CPU/memory limits equal to your requests, so this gets us the right Kubernetes behavior without incurring the wacky kernel behavior. If you Google that flag you can find many others who have done the same in their public configurations. There's a Kubernetes feature to use CPU pinning instead, which mentions improved performance as an advantage https://kubernetes.io/blog/2018/07/24/feature-highlight-cpu-... , but it only works if you're scheduling whole numbers of CPUs always.
A whole lot of applications on our Kubernetes clusters - lightly-accessed web servers, monitoring daemons, etc. - are best described as wanting 0.1 or 0.5 CPUs, in the sense that on (say) an 8-core machine, you could run many dozens of instances of these applications just fine. But that's not how the existing CFS quota mechanism interprets 0.1 or 0.5 CPUs. It seems like the "burst" mechanism here is likely to handle processes that, in a random 100 ms period, actually do run for more than 10 ms, but on an ongoing basis don't average out to using more than one tenth of a CPU. We might be able to turn --cpu-cfs-quota back on as a result.
This is EXACTLY the issue we are currently running into while hosting our gameservers on kubernetes via agones https://agones.dev/site/. While our gameserver processes don't really need a full core, they do need to be scheduled regularly to meet their tick simulation deadlines. A gameserver that ticks at 60hz absolutely needs CPU time at the very least every few milliseconds. Not getting scheduled back in time for a game tick means more input latency, and wreaks havoc on game clients trying to adjust their prediction buffers to optimally time their input receipt server side before the next server tick.
Bad, bad, bad scenario all around. A gameserver process really needs to get scheduled like a heartbeat. Ticks can't be late on the server as it really screws everything else up big time. We've seen these "bursting" effects really start to take it's toll, to the effect that we're now just setting a request and limit of 1CPU for each game server process (even though we could pack a lot more punch here).
Bad, bad, bad scenario all around. A gameserver process really needs to get scheduled like a heartbeat. Ticks can't be late on the server as it really screws everything else up big time. We've seen these "bursting" effects really start to take it's toll, to the effect that we're now just setting a request and limit of 1CPU for each game server process (even though we could pack a lot more punch here).
If you're using Linux, you could try the deadline scheduler policy (SCHED_DEADLINE, see sched(7) [1]), especially if the tick runtime can be estimated to an extent. Essentially, you set the task period (60 Hz in your case), the task runtime (a rough upper bound for the task execution time) and a deadline, and the Linux scheduler will make sure your task runs at an appropriate time every period. It will also throttle tasks that exceed their budget (by running over their allotted runtime) and it won't admit tasks if it's impossible to run them (since the sum of task runtimes exceeds the available runtime).
[1] https://www.man7.org/linux/man-pages/man7/sched.7.html
[1] https://www.man7.org/linux/man-pages/man7/sched.7.html
>While our gameserver processes don't really need a full core, they do need to be scheduled regularly to meet their tick simulation deadlines.
I think you need a core. If you don’t have one you’ll never know when the scheduler runs the process. The core scheduler feature seems like a good way to now do this.
We used to run hpux in “semi real time mode” for processes on tight timelines. The OS would let us put one process on the core that had interrupts disabled (only certain system calls were allowed). It worked well. Of course if the process had an issue the machine required a reboot.
Hpux also had processor sets which would allow us to tell the os to run certain processes on certain cores. This seems similar to the “core cookie” but I think that seems to have more to do with resource control.
This is really good progress.
I think you need a core. If you don’t have one you’ll never know when the scheduler runs the process. The core scheduler feature seems like a good way to now do this.
We used to run hpux in “semi real time mode” for processes on tight timelines. The OS would let us put one process on the core that had interrupts disabled (only certain system calls were allowed). It worked well. Of course if the process had an issue the machine required a reboot.
Hpux also had processor sets which would allow us to tell the os to run certain processes on certain cores. This seems similar to the “core cookie” but I think that seems to have more to do with resource control.
This is really good progress.
What's the advantage of using the CFS quota mechanism over cgroups cpulimit? I've been using cgroups to successfully make heavy compile jobs unnoticeable by limiting CPU shares to "0" (essentially idle priority) and limiting RAM to 2GB (again, soft limit, uses as much as is available), plus 16GB swap.
It has worked surprisingly well, I can compile Firefox on all cores while playing a game and it's completely imperceptible that anything else is happening, both in terms of performance and interactiveness/latency. No issues with this "burstiness" or stuck/deadlocked processes, either. "Nice" is basically useless in comparison.
It has worked surprisingly well, I can compile Firefox on all cores while playing a game and it's completely imperceptible that anything else is happening, both in terms of performance and interactiveness/latency. No issues with this "burstiness" or stuck/deadlocked processes, either. "Nice" is basically useless in comparison.
Kubernetes supports this too - CPU "requests" are implemented as a) input to the scheduler (it will not over-schedule a box) b) configuration of CPU shares in the kernel CPU cgroup. So that's effectively what we're relying on now with --cpu-cfs-quota. If you have an 8-hyperthread box and you have three jobs requesting 4, 2, and 2 CPUs each, they will use up the entire machine from the point of view of Kubernetes' scheduler and the first job will get twice as many CPU shares as the second two.
The problem is that we're running services etc., not batch jobs, so we do want them to make meaningful forward progress. So we can't set the shares to zero. We just don't want a misconfigured / runaway job to starve out the rest of the machine, even when the other jobs are not trying to use 100% CPU.
A specific sub-case here is capacity planning - right now, even with CPU shares, you can request one CPU for a computationally-intensive multi-thread/process task, and if the rest of the box is running internal web services with sporadic traffic, it will easily be able to use the whole machine. But then if you launch eight instances of that same job on the machine, they'll all perform much worse. So ideally we want to proactively limit CPU usage so that application developers/operators get realistic expectations about performance, and in turn, we get realistic information about how heavily our cluster is actually used.
The problem is that we're running services etc., not batch jobs, so we do want them to make meaningful forward progress. So we can't set the shares to zero. We just don't want a misconfigured / runaway job to starve out the rest of the machine, even when the other jobs are not trying to use 100% CPU.
A specific sub-case here is capacity planning - right now, even with CPU shares, you can request one CPU for a computationally-intensive multi-thread/process task, and if the rest of the box is running internal web services with sporadic traffic, it will easily be able to use the whole machine. But then if you launch eight instances of that same job on the machine, they'll all perform much worse. So ideally we want to proactively limit CPU usage so that application developers/operators get realistic expectations about performance, and in turn, we get realistic information about how heavily our cluster is actually used.
Excellent summary and exciting that we may be able to finally use CPU limits in k8s! I remember running into this in the early k8s days and being pretty disappointed that such a fundamental feature wasn't recommended or effective in real-world usage. Hope it doesn't take too long for this to roll out to all the k8s cloud offerings.
Excellent perspective, thanks for writing it up.
... So let me ask you this, by when do you get this kernel available to you in production? ;)
... So let me ask you this, by when do you get this kernel available to you in production? ;)
This is a good writeup but why bother with limits at all? Except core pinning there should be no gain in setting limits for non-metered workloads and it’s most likely just letting compute go unused. As far as I can tell this kernel change just fixes the tail latency problem
From my understanding the other goal that is often overlooked is that limiting resources on unmetered workloads prevents those processes from depriving others (and the host) of resources. It's mostly just a catch-all situation, but I would expect there to be ways to fine-tune this behavior.
This is not the case if you set cpu request the kubelet (actually cri) will configure cpu.shares controller
If you are having trouble with suspend/resume on your Ryzen 4xxx or 5xxx laptop, you might be in luck with this version. See https://gitlab.freedesktop.org/drm/amd/-/issues/1230 for details.
A better source for these changes are the two merge window roundups from LWN:
- https://lwn.net/Articles/861248/
- https://lwn.net/Articles/861695/