Caliper: Right-size your CI runners(attune.inc)
attune.inc
Caliper: Right-size your CI runners
https://www.attune.inc/blog/caliper
7 comments
The main problem is that builds require a variable amount of cores depending on what needs to be (re)built. The ideal thing to do is to have the build system itself orchestrate remote builds, since it actually knows how many things need building and how expensive they are.
This is what nixbuild.net does, it tracks historic CPU and memory usage of individual builds, and takes that into account when deciding what resources to allocate for new builds. You can configure limits on max/min CPUs on your account or individual builds. Also, if a build runs out of memory we simply restart it with more memory. The client will just see that the build log starts over.
That's precisely what I'm not describing; Nix doesn't even have access to the build DAG.
Please correct me if I'm wrong, but I assume you mean Nix doesn't have access to the build DAG that may exist inside the hermetic environment of individual Nix builds? If so, that's true, because Nix doesn't do that level of granularity unless you have a way to translate such DAGs into Nix derivations.
But Nix certainly tracks dependencies between Nix packages, and have knowledge about what packages need to be rebuilt if you make a change somewhere. Some of these packages might build config files, while other may build Chromium, ie wildly different CPU+mem needs.
But Nix certainly tracks dependencies between Nix packages, and have knowledge about what packages need to be rebuilt if you make a change somewhere. Some of these packages might build config files, while other may build Chromium, ie wildly different CPU+mem needs.
Right, I'm arguing this is the wrong abstraction level, and that only the build system can make correct container sizing decisions.
Great idea. Don’t have a personal need for it but imagine many will!
In my experience CI/CD tasks are more "single thread" bound that people expect. The Epyc CPU cores are _slow_ per core, so trading less cores for fewer faster ones actually works out well.
So if you are wanting fast CI/CD builds you are much better off using desktop CPU cores vs enterprise server cpus like this.
Wrote some thoughts up on this a while back https://martinalderson.com/posts/how-i-make-cicd-much-faster... - more focussed on the change from github to selfhosted runners, but I'd be interested to see a comparison on desktop class CPUs.