Use `nproc` and not grep /proc/cpuinfo(flamingspork.com)
flamingspork.com
Use `nproc` and not grep /proc/cpuinfo
https://www.flamingspork.com/blog/2020/11/25/why-you-should-use-nproc-and-not-grep-proc-cpuinfo/
50 comments
LXCFS can be used with Docker as well (although the integration is manual) which can be quite helpful (eg we're using it with docker-compose and find it very helpful)
Over time more things tend to get containerized/virtualized properly so these sorts of issues can be fixed.
My understanding is that it's unlikely cpuinfo and meminfo will be made cgroup-aware due to procfs-specific conventions on magic files. Making global procfs files change based on the current process is seen as a bad design these days -- should it be based on the process reading or the process opening the file (in this instance, both solutions are bad but for different reasons)? From memory, the topic was brought up early on in cgroup development.
IIRC this utility is a wrapper around glibc get_nprocs(), which on Linux will ultimately read from /sys/devices/system/cpu/online. If you need this information from a compiled language, you should probably use the glibc wrapper.
We usually use
np = sysconf(_SC_NPROCESSORS_ONLN);
Even this I feel doesn't work all the time. At least from memory, I remember some problems where on kubernetes if you look at nproc or cpuinfo, that will give you the number of processors on the host machine, and not necessarily the CPU limit which is set per container. Although maybe I'm mis-remembering. I guess nproc could look at something else as well to filter based on that, and I agree having a nice portable commandline is way better than looking at /proc.
This is 100% a problem I was working on for a customer recently. nproc specifically does not currently check your CFS CPU cgroup quota, i.e. you may have 12 active cores but only a budget of 4 CPUs worth of CPU time. This can be an issue.
It would be ideal to have a standard utility or library that would also check that.
If you want to understand more about these problems this article has some good background, although they fixed a real bug in the scheduler here the background will give you the general idea and it's still a problem to some extent if all of those threads/processes are trying to contend on a single shared lock.. which is precisely what happens with multi threaded Python programs (they sometimes contend on the GIL) that use data science modules which do a lot of processing in C without the GIL lock held and only take it occasionally. However the problem comes if you have 12 threads consuming your 4 threads worth of CPU time they'll end up block for 5-10 or more milliseconds at a time where they may well be holding that lock and preventing the other threads from running.
You may then ask why not reduce the CPU set for those containers, the problem is there is no way (at least currently) to say 'this process can only run on 4 CPUs, but any of them'. You have to reduce the CPU set down to 4 specific CPUs and now if all containers scheduled to those specific 4 CPUs are busy you will be losing time to those other containers when a different 4 CPUs may be totally idle. It would be nice to advance the scheduler to support such a concept without using the quotas specifically.
As a final side note.. LXD cheats here. It uses 'lxcfs' to bind mount onto /proc/cpuinfo and modifies the output to only show the number of CPU cores you actually have. Which suffices for most programs.
It would be ideal to have a standard utility or library that would also check that.
If you want to understand more about these problems this article has some good background, although they fixed a real bug in the scheduler here the background will give you the general idea and it's still a problem to some extent if all of those threads/processes are trying to contend on a single shared lock.. which is precisely what happens with multi threaded Python programs (they sometimes contend on the GIL) that use data science modules which do a lot of processing in C without the GIL lock held and only take it occasionally. However the problem comes if you have 12 threads consuming your 4 threads worth of CPU time they'll end up block for 5-10 or more milliseconds at a time where they may well be holding that lock and preventing the other threads from running.
You may then ask why not reduce the CPU set for those containers, the problem is there is no way (at least currently) to say 'this process can only run on 4 CPUs, but any of them'. You have to reduce the CPU set down to 4 specific CPUs and now if all containers scheduled to those specific 4 CPUs are busy you will be losing time to those other containers when a different 4 CPUs may be totally idle. It would be nice to advance the scheduler to support such a concept without using the quotas specifically.
As a final side note.. LXD cheats here. It uses 'lxcfs' to bind mount onto /proc/cpuinfo and modifies the output to only show the number of CPU cores you actually have. Which suffices for most programs.
If you just run Kubernetes with the default settings, that’s correct. You will need to enable the so-called static cpu manager policy on kubelets to make it use cpusets based on CPU requests & limits.
https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...
A somewhat annoying thing about this option is that even though it can create cpusets for every container individually, it only gets enabled in case all containers in the pod use an integer number of CPUs, which seems unnecessary.
https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana...
A somewhat annoying thing about this option is that even though it can create cpusets for every container individually, it only gets enabled in case all containers in the pod use an integer number of CPUs, which seems unnecessary.
> So, what this will end up doing is just increase the number of context switches, possibly also adding a performance degradation.
I think this is a jumping to conclusions a bit. You’d need to be reading /proc/cpuinfo in a fairly tight loop in order for the time spend doing any context switching to dominate the runtime cost of a program. Is this a common practice in some types of computation? Or am I misunderstanding this quote from the article?
I think this is a jumping to conclusions a bit. You’d need to be reading /proc/cpuinfo in a fairly tight loop in order for the time spend doing any context switching to dominate the runtime cost of a program. Is this a common practice in some types of computation? Or am I misunderstanding this quote from the article?
I think that the full context of the sentence you quoted will help:
> In this case, if you base your number of threads off grepping lscpu you take another dependency (on the util-linux package), which isn’t needed. You also get the wrong answer, as you do by grepping /proc/cpuinfo. So, what this will end up doing is just increase the number of context switches, possibly also adding a performance degradation.
The idea is that a common way to size an application that exploits concurrency is to base the number of application threads on the hardware threads available. This portion of the article is saying that if you do so on the basis of `lscpu` or grepping `/proc/cpuinfo` you could mis-size relative to the actual number of cores available to schedule the threads of your application on. Doing so will lead to excessive context switching for your application, and that can hurt.
> In this case, if you base your number of threads off grepping lscpu you take another dependency (on the util-linux package), which isn’t needed. You also get the wrong answer, as you do by grepping /proc/cpuinfo. So, what this will end up doing is just increase the number of context switches, possibly also adding a performance degradation.
The idea is that a common way to size an application that exploits concurrency is to base the number of application threads on the hardware threads available. This portion of the article is saying that if you do so on the basis of `lscpu` or grepping `/proc/cpuinfo` you could mis-size relative to the actual number of cores available to schedule the threads of your application on. Doing so will lead to excessive context switching for your application, and that can hurt.
I wasn't talking about the reading of the number of CPUs you could use, but rather how many threads you'd spawn based on that information.
There may be 10 CPUs in the machine, but if your in your container which is restricted to using only 1 of them, it's no use launching 10 threads in that container over just having one.
There may be 10 CPUs in the machine, but if your in your container which is restricted to using only 1 of them, it's no use launching 10 threads in that container over just having one.
Oh I see. My mistake. This makes more sense.
The article is saying if you make too many processes (more than the number of hardware threads available) the OS will have to context switch between them to keep them all running.
They mean that you read it once, then spawn more threads than you have actual cores. Those threads then need to context switch back and forth to do work.
It's amazing how much knowledge is being lost with the advent of container revolutions and orchestrator Os'es. Cgroups is _still_ opt in not opt out...though those who never have worked in HPC on bare metal with parallel libraries and approaches for cpu like openmp are loudly complaining about control groups that don't control or provide consistent view of userland like there isn't a host system underneath. It's like this (https://www.gnu.org/software/libc/manual/html_node/CPU-Affin...) wasn't the way things were done (and still are to this day) in everything but the container world. I know it's hard to grasp but not everyone is containerizing everything.
I see suggestions to use ‘make -j$(nproc)’. However, I was taught when learning MPI programming 20 years ago, you want think at least “one to load, one to run”. The idea being that processes and threads waiting on IO will sleep/yield while another process or thread can continue executing. An especially useful strategy if you can afford the RAM (trading RAM for CPU). Consequently, I tend to run twice as many ‘make’ jobs as threads on my systems.
I think that would justify adding a single extra, not an entire slate.
I think logic still holds. There’s an order magnitude more cache in today’s server CPUs. There’s higher likelihood today that your thread or process will remain in the cache, keeping it performant.
Yeah I think so I was taught to run make -j(number of threads + 1) by taught I mean I saw it in a lot of tutorials during my early years of learning Linux, C and make.
I’ve heard about this trick/incantation as well, but haven’t managed to measure a significant performance gain (though it makes sense in theory). Do you know of any quantitative evidence that this is solid advice? I suppose my testing is not at all scientific, just my anecdotal experience using make -j(i+1)
That was before mainstream SMT.
This is probably more worthwhile when using spinning disks, as opposed to SSDs.
For those running `make`, add this to you shell init to set the concurrency level via env var:
> expert MAKEFLAGS=-j$(nproc)
> expert MAKEFLAGS=-j$(nproc)
Good idea.
make -j without a number should do that, instead of spawning processes like there’s no tomorrow.
make -j without a number should do that, instead of spawning processes like there’s no tomorrow.
To be fair, having some way to spawn processes like there's no tomorrow is a good way to make sure there are no concurrency issues in your makefile.
I had a makefile with a race condition that triggered only sporadically. However, giving it more threads significantly increased its rate of occurrence.
I had a makefile with a race condition that triggered only sporadically. However, giving it more threads significantly increased its rate of occurrence.
Any doc that suggest this behaviour?
From the man page:
> If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.
What this practically means is that a plain "make -j" will spawn all the processes that could run concurrently. So if you're building 1,000 files, it'll spawn 1,000 processes. This can either be fine, or completely terrible, depending on the number of CPUs you have an the amount of memory each process uses.
*export
As a software developer one might also choose to use the switch -O as to not mangle the output in a (too) nondeterministic order.
Or prefer ninja overall.
Or prefer ninja overall.
Interesting, but how does nproc get the right information ?
OMP_NUM_THREADS and OMP_THREAD_LIMIT are in principle extra limitations, not the main source of information, right ?
OMP_NUM_THREADS and OMP_THREAD_LIMIT are in principle extra limitations, not the main source of information, right ?
It's sched_getaffinity, which makes sense, since that's what taskset and friends modify.
$ strace nproc
sched_getaffinity(0, 128, [0, 1, 2, ..., 78, 79]) = 40
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0xf), ...}) = 0
write(1, "80\n", 324) = 3
exit_group(0) = ?
+++ exited with 0 +++If you have OPM_NUM_THREADS and OMP_THREAD_LIMIT set it will use those, though.
nproc isn't available on some of the BSDs
$ nproc
ksh: nproc: not found
$ uname -a
OpenBSD obsd64.example.xyz 6.8 GENERIC.MP#828 arm64
$ sysctl hw.ncpu
hw.ncpu=2Right. So on Mac or other BSDs, you want use a ‘sysctl’.
On Macs, this "total cores available" strategy might not be a valid approach, since you probably only want to be using the high-performance cores and not also the low-power ones — and the CPU might well overbook all your threads to high-perf to leave room for the OS and other programs, even if you try to do otherwise.
Not just Macs, plenty of Linux systems have a big.LITTLE design.
with a big.LITTLE system it'll really depend what you're doing. Just compiling something? It's probably fine (if all cores can be active at once). If you're wanting more consistent performance for some calculations? You might need to think a bit more.
with a big.LITTLE system it'll really depend what you're doing. Just compiling something? It's probably fine (if all cores can be active at once). If you're wanting more consistent performance for some calculations? You might need to think a bit more.
Another thought, I don’t know if Alpine Linux or other musl-libc based systems will have ‘nproc’ if, for instance, you’re running containers or other minimal systems.
cpuset -g on FreeBSD >= 7.1 will tell you what cpus are available to your shell, where a sysctl will only tell you what's on your system. (Lots of good info in the related manual pages)
Yeah, I just use $(nproc) in scripts when I need to set up threads number.
[1]: https://github.com/lxc/lxcfs