Everything you never wanted to know about CMake (redux)(izzys.casa)
izzys.casa
Everything you never wanted to know about CMake (redux)
https://izzys.casa/2023/06/everything-you-never-wanted-to-know-about-cmake-redux/
57 comments
Regarding vcvars.bat, I'm still a fan of letting the user set that before calling into the build system. I specifically don't want random projects trying to find random toolchains in random ways. The new toolchain file sounds like a great development in that space.
Without CMake, from powershell, you can do something like `& "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\Launch-VsDevShell.ps1" -VsInstallationPath:"C:\Program Files\Microsoft Visual Studio\2022\Community" -Arch:amd64 -HostArch:amd64 -SkipAutomaticLocation` and it works well enough. I usually provide that oneliner as a vs2022ce-x64-x64.ps1 script, and just note in the README that users can adapt it to their system/needs.
Also of note is the amazing portable-msvc.py by the one and only Mārtiņš Možeiko : https://gist.github.com/mmozeiko/7f3162ec2988e81e56d5c4e22cd...
It downloads the compiler toolchain and nothing more, and writes a setup.bat with no conditional logic. It was trivial for me to adapt to write a setup.ps1 also. Compare the size of the result, vs my attempt to have a minimal toolchain using the Visual Studio Installer.
I don't recommend projects to use it directly, it's just an excellent resource to point people towards.
Without CMake, from powershell, you can do something like `& "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\Launch-VsDevShell.ps1" -VsInstallationPath:"C:\Program Files\Microsoft Visual Studio\2022\Community" -Arch:amd64 -HostArch:amd64 -SkipAutomaticLocation` and it works well enough. I usually provide that oneliner as a vs2022ce-x64-x64.ps1 script, and just note in the README that users can adapt it to their system/needs.
Also of note is the amazing portable-msvc.py by the one and only Mārtiņš Možeiko : https://gist.github.com/mmozeiko/7f3162ec2988e81e56d5c4e22cd...
It downloads the compiler toolchain and nothing more, and writes a setup.bat with no conditional logic. It was trivial for me to adapt to write a setup.ps1 also. Compare the size of the result, vs my attempt to have a minimal toolchain using the Visual Studio Installer.
1.3G msvc
4.2G C:/Program Files/Microsoft Visual Studio/2022/Community
I don't recommend projects to use it directly, it's just an excellent resource to point people towards.
Just saw vcvarsall.bat in the post. Did you know that the Visual Studio Command Prompt phones home to Microsoft? I tried installing Microsoft's build tools on Wine, then when I opened a VS command prompt from KDE I saw an error message, from Wine's PowerShell stub being used to run a telemetry script. Then I discovered the very same telemetry had been running every time I had opened a MSVC command prompt on Windows, as well as countless other telemetry operations in the Visual Studio IDE and updater itself.
I found a video on this topic at https://www.youtube.com/watch?v=ORH5JfpPx88.
I found a video on this topic at https://www.youtube.com/watch?v=ORH5JfpPx88.
I cannot overemphasize how enjoyable it is to work with cargo when I'm coding in Rust; I think it's one of the major reasons why Rust took off. As complicated as Rust is, it's a breeze to not only build projects but also add existing projects as dependencies.
Most of my Rust projects, even large and complex ones, don't have any build script at all. That made me realize how much time and energy I used to waste on maintaining build scripts.
I beg to disagree.
Sure, when you want to have a small project that only uses crates.io d'EPS, Cargo is simple and gets the job done.
But once you want to do something a bit more complex, (like getting a dependency of something not written in rust, or picking features based on some system configuration), you quickly get to the limitations and there is not much you can do.
Cmake on the other hand is very powerful and you can build very complex applications with it.
Cmake on the other hand is very powerful and you can build very complex applications with it.
Ok, but do you actually enjoy using CMake, or do you accept how horribly complex it is because it lets you do some esoteric thing you want to do?
Every tool has limitations, sure. But I think you're disagreeing about something I didn't even say. I didn't say cargo can replace CMake. I said that cargo is really a pleasure to use when it's the tool you need. If you had the choice to either build with CMake or cargo, I promise you that you'd rather use cargo if possible.
Building most C/C++ projects seems pretty miserable no matter what build process you're using. CMake seems like a huge mental burden in itself. Cargo just does its job and gets out of the way.
Every tool has limitations, sure. But I think you're disagreeing about something I didn't even say. I didn't say cargo can replace CMake. I said that cargo is really a pleasure to use when it's the tool you need. If you had the choice to either build with CMake or cargo, I promise you that you'd rather use cargo if possible.
Building most C/C++ projects seems pretty miserable no matter what build process you're using. CMake seems like a huge mental burden in itself. Cargo just does its job and gets out of the way.
> like getting a dependency of something not written in rust, or picking features based on some system configuration
I've found that things like this are pretty straightforward with a build.rs and you aren't context switching between a configuration language and your target language since build.rs is just a Rust program that outputs configuration values via println.
> ... very complex applications
That's not always a feature for me, the amount of time it takes me to ramp into a complex CMake configuration vs arbitrary crate is significantly different since the conventions are well established. If you spend any time cross-compiling things the Rust experience(I.E. first-class triple support, cc crate) is miles above anything CMake provides.
I've found that things like this are pretty straightforward with a build.rs and you aren't context switching between a configuration language and your target language since build.rs is just a Rust program that outputs configuration values via println.
> ... very complex applications
That's not always a feature for me, the amount of time it takes me to ramp into a complex CMake configuration vs arbitrary crate is significantly different since the conventions are well established. If you spend any time cross-compiling things the Rust experience(I.E. first-class triple support, cc crate) is miles above anything CMake provides.
> I've found that things like this are pretty straightforward with a build.rs
- build.rs can't change the feature set.
- what they usually do to bring dependency is vendor the C files within the crates, but when it comes to configuring to use an installed library and passing the right linker flags, this becomes really tricky.
- build.rs can't change the feature set.
- what they usually do to bring dependency is vendor the C files within the crates, but when it comes to configuring to use an installed library and passing the right linker flags, this becomes really tricky.
> build.rs can't change the feature set.
Can you be more specific here? You can totally do that based on OS, environment vars or even direct feature flags in the Cargo.toml. I've got a number of projects that do this and let downstream consumers decide what features they want. At the end of the day it's just a Rust program that runs ahead of the build and can do anything that you would be able to do in a normal command line program.
> ... but when it comes to configuring to use an installed library and passing the right linker flags, this becomes really tricky.
pkg-config[1] does all of that including returning linker flags. I can't think of a single case where I wasn't able to integrate with an existing library.
I'm not saying there are zero issues(I.E. passing some linker flags as a dependent crate gets annoying) but my experience having used C++ across a number of platforms(Win32, Linux, proprietary systems) before CMake existed and after it started getting wider adoption is that CMake has a significantly higher hurdle and is just harder to get things working correctly, especially under cross-compilation configurations.
[1] https://crates.io/crates/pkg-config
Can you be more specific here? You can totally do that based on OS, environment vars or even direct feature flags in the Cargo.toml. I've got a number of projects that do this and let downstream consumers decide what features they want. At the end of the day it's just a Rust program that runs ahead of the build and can do anything that you would be able to do in a normal command line program.
> ... but when it comes to configuring to use an installed library and passing the right linker flags, this becomes really tricky.
pkg-config[1] does all of that including returning linker flags. I can't think of a single case where I wasn't able to integrate with an existing library.
I'm not saying there are zero issues(I.E. passing some linker flags as a dependent crate gets annoying) but my experience having used C++ across a number of platforms(Win32, Linux, proprietary systems) before CMake existed and after it started getting wider adoption is that CMake has a significantly higher hurdle and is just harder to get things working correctly, especially under cross-compilation configurations.
[1] https://crates.io/crates/pkg-config
> CMake Now Has Dictionaries! (Sort of!)
It may be time to update Zawinski's Law to something like:
"Every tool expands until turning complete"
At the point where cmake is fungible with a programming language, why not just write the build system in the one you know?
As with the various markup languages that purport to "simplify" HTML, we're given enough PITA learning curve (how does this doo-hicky do anchor tags, since <a> is right out?) that just writing straight HTML would have sucked less.
/rant
It may be time to update Zawinski's Law to something like:
"Every tool expands until turning complete"
At the point where cmake is fungible with a programming language, why not just write the build system in the one you know?
As with the various markup languages that purport to "simplify" HTML, we're given enough PITA learning curve (how does this doo-hicky do anchor tags, since <a> is right out?) that just writing straight HTML would have sucked less.
/rant
> "Every tool expands until turning complete"
In that respect I think PreMake (Lua based configuration and build system) is more humanistic.
Lua syntax provides pretty comfortable, DSL alike, way of defining rules/declaration. And at the same time it is regular and compact PL with established runtime for those 10% of cases when static declarations are not enough.
CMake, IMHO, went wrong way of defining DSL first and then trying to accommodate it to needs of full PL and real life. In any case need for debugger for configuration tool is a bad sign. IMO.
In that respect I think PreMake (Lua based configuration and build system) is more humanistic.
Lua syntax provides pretty comfortable, DSL alike, way of defining rules/declaration. And at the same time it is regular and compact PL with established runtime for those 10% of cases when static declarations are not enough.
CMake, IMHO, went wrong way of defining DSL first and then trying to accommodate it to needs of full PL and real life. In any case need for debugger for configuration tool is a bad sign. IMO.
I had meant 'Turing'.
Whenever I have to work with CMake, I find it useful to adopt a “please sir, may I have another” attitude.
It feels a lot more like "The beatings will continue until morale improves" to me.
Quite a lot of "Could be worse. Could be Autotools." for me.
from the perspective of someone who spends 100x more time trying to compile other people’s software than writing my own build scripts, they don’t really feel that different. CMake seems to offer more features/builtins, but without doing much to actually tame complexity. which isn’t necessarily a good thing because then the people who think it’s a good idea to shove complexity into their build system just do more of that and i have to read 10x more build scripts to coerce the thing into doing what i want (e.g. getting it to cross compile).
Yeah, at some point I feel like certain things have an irreducible amount of complexity that ends up getting moved around like an air bubble stuck under plastic laminate. Sometimes moving the complexity helms, sometimes it feels like it’s more trouble than it’s worth. CMake seems to be a net positive, but it’s not without substantial pain at time.
that’s a really fun metaphor — and it does capture a lot of my day to day experience. but a lot of these bubbles can also be split apart or joined, and doing that skillfully has pretty huge social implications in open source.
i’ve been toying around recently with [sxmo]. at the top level, it’s a collection of shell scripts. when any script would become too complex, it’s factored out into some library-like abstraction and lifted into a different repo, language, etc.
i’m coming at this from phosh, after i hit a bug in it, opened the code base to debug it, and realized “i have no clue where to even start”. desktops already split the bubbles of complexity into components like compositors, window managers, service managers. sxmo shows that you can be even more extreme in this: any time a single bubble of complexity grows too large, break it up into smaller bubbles. do this rigorously and it means any time the software doesn’t behave as i want, i can pretty confidently plot a path to fixing/improving it (and know how long it’ll take and therefore if it’s worth the time).
there’s complexity inherent in anything, but there’s a lot of different ways you can arrange it. my experience with CMake is that the complexity sort of just pervades the whole stack, rather than being isolated into manageable bubbles at all.
sxmo: https://sxmo.org/
i’ve been toying around recently with [sxmo]. at the top level, it’s a collection of shell scripts. when any script would become too complex, it’s factored out into some library-like abstraction and lifted into a different repo, language, etc.
i’m coming at this from phosh, after i hit a bug in it, opened the code base to debug it, and realized “i have no clue where to even start”. desktops already split the bubbles of complexity into components like compositors, window managers, service managers. sxmo shows that you can be even more extreme in this: any time a single bubble of complexity grows too large, break it up into smaller bubbles. do this rigorously and it means any time the software doesn’t behave as i want, i can pretty confidently plot a path to fixing/improving it (and know how long it’ll take and therefore if it’s worth the time).
there’s complexity inherent in anything, but there’s a lot of different ways you can arrange it. my experience with CMake is that the complexity sort of just pervades the whole stack, rather than being isolated into manageable bubbles at all.
sxmo: https://sxmo.org/
I flinch just a little bit every time I open a CMakeLists.txt in my editor.
I'm purely surprised by the activeness of CMake development. I didn't know there was still room for improvement. A debugger is surely nice, considering the complexity of the CMakeLists...
A good chunk of my day job is beating sense into CMakeLists.txt files. It's exhausting in every meaningful dimension. In my literal decades of working with cmake, no amount of familiarity has ever brought me contentment. Rather, with each passing release, the contortions grow ever more torturous.
At one point, on one project, in a moment of supreme frustration, I replaced cmake with GNU Make and a single Makefile. I'm not proud, but for that one instance, it was a good decision because It Just Worked.
At one point, on one project, in a moment of supreme frustration, I replaced cmake with GNU Make and a single Makefile. I'm not proud, but for that one instance, it was a good decision because It Just Worked.
I've found a lot of value in using Zig as my C & C++ toolchain. I really like that my build logic is more procedural, which I find easier to comprehend.
Related:
Everything You Never Wanted to Know About CMake - https://news.ycombinator.com/item?id=19070733 - Feb 2019 (87 comments)
Everything You Never Wanted to Know About CMake - https://news.ycombinator.com/item?id=19070733 - Feb 2019 (87 comments)
Orson Welles voice "Do you have a favorite kind of cardboard?"
Wow. This post just prompts me to take another look at xmake.
we can also try xmake. https://github.com/xmake-io/xmake
Xmake can be used to directly build source code (like with Make or Ninja), or it can generate project source files like CMake or Meson. It also has a built-in package management system to help users integrate C/C++ dependencies.
Xmake can be used to directly build source code (like with Make or Ninja), or it can generate project source files like CMake or Meson. It also has a built-in package management system to help users integrate C/C++ dependencies.
All these CMake improvements are the best example of optimising into a local minimum I have ever seen in SWE
“Local minimum” is a great way to put it.
I recognize that CMake has done a lot to make people’s lives easier, but it has so many design flaws that the entire design seems almost purposefully user-hostile. I know it’s not hostile on purpose, but it sometimes feels that way.
I recognize that CMake has done a lot to make people’s lives easier, but it has so many design flaws that the entire design seems almost purposefully user-hostile. I know it’s not hostile on purpose, but it sometimes feels that way.
Begin by throwing the language away. Provide a tool that converts existing CMake scripts to the new, sane language. Use an established programming language for the new language.
CMake semantics except expressed in Python scripts somehow sounds even worse.
These days we have Starlark, which is the core part of Python, modified to be completely hermetic & deterministic (by removing some things and changing others), and modified to support parallel execution. It is very nice to work with, since you can use a lot of Python tooling and Python experience, but you don’t have to carry around a complete Python implementation.
It was originally part of Bazel but it is used in other tools, and there are multiple implementations.
It was originally part of Bazel but it is used in other tools, and there are multiple implementations.
That's basically what CMake already attempted to do: be a sane alternative to figuring out the madness of autoconf and Makefiles.
The language itself is, in many ways, worse than the shell scripts it replaced with autoconf.
I don't disagree that the CMake language is badly misguided. I think that the SCons approach of just writing Python is sensible rather then inventing a whole new scripting language as well as a build system.
But CMake probably will get you a program that compiles cross-platform within an hour or two, even for a novice. Autotools probably won't do that for you.
But CMake probably will get you a program that compiles cross-platform within an hour or two, even for a novice. Autotools probably won't do that for you.
SCons is utterly horrible. It isn’t a build system so much as a “Build your own build system” system. It gives you absolutely nothing to manage dependencies or shared information, or cross-platform ways of handling compiler features.
As such, everything using it is it’s own unique system completely alien and not interoperable with anything else.
And being python seems to lead to people embedding parts of their application as part of the build system, so now that is intertwined also.
I will run a million miles if I ever see a project using SCons, CMake is infinitely nicer, and makes some sense but I agree with further up the thread that it needs a new, saner language that maps over the top (expressions would be a start).
As such, everything using it is it’s own unique system completely alien and not interoperable with anything else.
And being python seems to lead to people embedding parts of their application as part of the build system, so now that is intertwined also.
I will run a million miles if I ever see a project using SCons, CMake is infinitely nicer, and makes some sense but I agree with further up the thread that it needs a new, saner language that maps over the top (expressions would be a start).
Hm. We had a SCons based build system around ten years back where we ran 10 different C++ compilers/toolchains for multiple native and cross build targets without any problems. Yes, we had to write some stuff on top of it, but after some systematic design, it went really smoothly.
If the new language would allow such a tool to work, it probably wouldn't be much better. It would either still have most of the awful behavior that the old language had or the converted scripts wouldn't be better than before.
The way the cmake_policy system works isn't bad, but those unfortunately have only limited impact on the language itself.
The way the cmake_policy system works isn't bad, but those unfortunately have only limited impact on the language itself.
Good phrasing!
I’m just a hobbyist so I don’t get very involved but what would you suggest as an alternative maximum?
I’m just a hobbyist so I don’t get very involved but what would you suggest as an alternative maximum?
Bazel is the only sensible alternative for C++. It's obviously way more work and not really worth it for really small projects, but it is at least well designed and it solves real problems through hermeticity.
But if you're not going to go to that effort I wouldn't really recommend anything other than CMake. There are alternatives that are a bit better (Meson) but then you give up on using the de facto build system (as much as there is one).
Also I've found that a lot of the "cleaner" C++ build systems are partly only cleaner because they ignore the full range of complexity that real world C++ builds have to content with. Rpaths, symbol visibility, etc.
But if you're not going to go to that effort I wouldn't really recommend anything other than CMake. There are alternatives that are a bit better (Meson) but then you give up on using the de facto build system (as much as there is one).
Also I've found that a lot of the "cleaner" C++ build systems are partly only cleaner because they ignore the full range of complexity that real world C++ builds have to content with. Rpaths, symbol visibility, etc.
What do you want in a maximum? Cmake is a local maximum if popularity is important, which it should be. Because cmake is popular you can find lots of things that work with it, and when you have problems other experts who can help.
For build systems few people really want to become experts so finding them is important.
Don't take the above as saying cmake is the best, it deserves most criticism. However the alternatives are probably not compelling just because they are not popular.
For build systems few people really want to become experts so finding them is important.
Don't take the above as saying cmake is the best, it deserves most criticism. However the alternatives are probably not compelling just because they are not popular.
A build system is largely declarative--it tends to boil down to defining rules like "how to compile a source file", "how to build a library", etc., along with the lists of things those rules need to be applied to, with there being a confusing three-way tug-of-war between the user, the project, and the system over how to override stuff and who wins out. The end result should be that you should ideally be able to query the build system to figure out the underlying declarative pieces of it pretty easily (e.g., list all of the C++ compiler invocations, list all of the package dependencies, etc.).
CMake is like halfway there or so, combined with a shell-like language that has some annoying issues (e.g., functions are statements, not expressions, so if you want to do dirname(dirname(foo)), that's two calls to PARENT_PATH). It does a better job than autoconf/make in that it doesn't invite you to resort to shell almost immediately, but that is admittedly a low bar.
CMake is like halfway there or so, combined with a shell-like language that has some annoying issues (e.g., functions are statements, not expressions, so if you want to do dirname(dirname(foo)), that's two calls to PARENT_PATH). It does a better job than autoconf/make in that it doesn't invite you to resort to shell almost immediately, but that is admittedly a low bar.
While I want a.build system to be declaritive, in the real world there is always complexity they designers didn't think of. So I need an escape to build something they didn't think of.
Though it would be nice if that thing I.build could then be declarative.
Though it would be nice if that thing I.build could then be declarative.
Use whatever your chosen platform supports "natively". If you're on Linux like me, then use GNU Make or even a Bash script. I'm sure Windows users have something different.
Here's the important part: move to a more powerful tool after you decide to support other platforms --- and before circumstances force your hand.
Here's the important part: move to a more powerful tool after you decide to support other platforms --- and before circumstances force your hand.
Disclaimer: I'm making a competing build system.
I won't tell you specific build systems, but I will tell you what to look for.
Look for power. Unlimited power. [1]
Usually, this means a few things:
1. The build system uses a general-purpose language, even if the language needs features to be added.
2. The build system does not reduce the power of the general-purpose language. For example, say it starts with Python but prohibits recursion. In that case, you know it is not unlimited power. Looking at you, Starlark.
3. The build can be dynamically changed, i.e., the build is not statically determined before it even begins.
4. Each task has unlimited power. This means that the task can use a general-purpose language, not just run external processes.
5. And there has to be some thought put it in user experience.
Why are these important? Well, let's look at why with CMake, which fails all of them.
For #1, CMake's language started as a limited language for enumerating lists. (Hence, CMakeLists.txt is the file name.) And yet, it's grown to be as general-purpose as possible. Why? Because when you need an if statement, nothing else will do, and when you need a loop, nothing else will do.
And that brings us to #2: if CMake's language started limited, are there still places where it's limited? I argue yes, and I point to the article where it says that your couldn't dynamically call functions until recently. There are probably other places.
For #3, CMake's whole model precludes it. CMake generates the build upfront then expects another build system to actually execute it. There is no changing the build without regenerating it. (And even then, CMake did a poor job until the addition of `--fresh`.) A fully dynamic build should be able to add targets and make others targets depend on those new targets dynamically, among other things.
For #4, obviously CMake limits what tasks can do because Ninja and Make limit tasks to running commands.
As another example, to implement a LaTeX target, you technically need a while loop to iterate until a fixed point. To do that with Make and Ninja, you have to jump through hoops or use an external script that may not work on all platforms.
CMake obviously fails #5, and to see how much other build systems fail it, just look for comments pouring hate on those build systems. CMake fails the most, but I haven't seen one that passes yet.
As an example, CMake barely got a debugger. Wow! Cool! It's been 20 years! My build system will have a debugger in public release #2 (one after the MVP) that will be capable of outputting to multiple TTY's like gdb-dashboard. [2] They should have had this years ago!
Should other comments suggest specific build systems, like the one that suggested Bazel, judge them by this list. Some will be better than others. None will pass everything, IMO, which is why I'm making my own.
[1]: https://youtube.com/watch?v=Sg14jNbBb-8
[2]: https://github.com/cyrus-and/gdb-dashboard
I won't tell you specific build systems, but I will tell you what to look for.
Look for power. Unlimited power. [1]
Usually, this means a few things:
1. The build system uses a general-purpose language, even if the language needs features to be added.
2. The build system does not reduce the power of the general-purpose language. For example, say it starts with Python but prohibits recursion. In that case, you know it is not unlimited power. Looking at you, Starlark.
3. The build can be dynamically changed, i.e., the build is not statically determined before it even begins.
4. Each task has unlimited power. This means that the task can use a general-purpose language, not just run external processes.
5. And there has to be some thought put it in user experience.
Why are these important? Well, let's look at why with CMake, which fails all of them.
For #1, CMake's language started as a limited language for enumerating lists. (Hence, CMakeLists.txt is the file name.) And yet, it's grown to be as general-purpose as possible. Why? Because when you need an if statement, nothing else will do, and when you need a loop, nothing else will do.
And that brings us to #2: if CMake's language started limited, are there still places where it's limited? I argue yes, and I point to the article where it says that your couldn't dynamically call functions until recently. There are probably other places.
For #3, CMake's whole model precludes it. CMake generates the build upfront then expects another build system to actually execute it. There is no changing the build without regenerating it. (And even then, CMake did a poor job until the addition of `--fresh`.) A fully dynamic build should be able to add targets and make others targets depend on those new targets dynamically, among other things.
For #4, obviously CMake limits what tasks can do because Ninja and Make limit tasks to running commands.
As another example, to implement a LaTeX target, you technically need a while loop to iterate until a fixed point. To do that with Make and Ninja, you have to jump through hoops or use an external script that may not work on all platforms.
CMake obviously fails #5, and to see how much other build systems fail it, just look for comments pouring hate on those build systems. CMake fails the most, but I haven't seen one that passes yet.
As an example, CMake barely got a debugger. Wow! Cool! It's been 20 years! My build system will have a debugger in public release #2 (one after the MVP) that will be capable of outputting to multiple TTY's like gdb-dashboard. [2] They should have had this years ago!
Should other comments suggest specific build systems, like the one that suggested Bazel, judge them by this list. Some will be better than others. None will pass everything, IMO, which is why I'm making my own.
[1]: https://youtube.com/watch?v=Sg14jNbBb-8
[2]: https://github.com/cyrus-and/gdb-dashboard
Are you being serious, or is this an elaborate joke?
My experience with build systems is that, by going through the pain of trying to implement your own build system with some of the same desiderata as Bazel (reliability, performance), you end up independently discovering the same features Bazel has (like the purposefully limited scripting system). Going in the opposite direction seems like the best way to purposefully design a bad build system, but maybe I misunderstand what your goals are here?
My experience with build systems is that, by going through the pain of trying to implement your own build system with some of the same desiderata as Bazel (reliability, performance), you end up independently discovering the same features Bazel has (like the purposefully limited scripting system). Going in the opposite direction seems like the best way to purposefully design a bad build system, but maybe I misunderstand what your goals are here?
Deadly serious.
My goal is to make a build system that the majority of people don't hate.
I understand your concerns about a fully-powerful system. I know that's why Bazel is a favorite.
But Nix has the same reliability with a full Turing-complete language, and I feel like I can recover the performance by using C instead of Java.
In addition, I understand the purpose of limited languages in build systems. I really do.
The first public release of my build system (the MVP) will have the ability to restrict the power of the language, and it will do so by default because limiting power when working with other people is noice. [1]
You may ask why I make a point of power if my own language will be hobbled by default. Because users will still be able to remove the hobble if needed. And sometimes, it will be needed.
But I'll still go above and beyond; not only will users be able to choose between a restricted language or a powerful one, they'll be able to choose how powerful the language needs to be.
Only need a POSIX Makefile replacement? That's the default. Need if statements, but only if statements? Got you covered. Need functions? No problem. Need loops? Just say the word. No dynamic build stuff needed? No worries needed. Wanna go full Palpatine? Yes, Master.
In other words, I hear you, and you are right for the vast majority of cases. But for that other minority of cases, power must exist, and there is no alternative, so it will be available.
This is related to Joel Spolsky's assertion that while everybody only uses 20% of the features, nobody uses the same 20% [2]:
> A lot of software developers are seduced by the old “80/20” rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies.
> Unfortunately, it’s never the same 20%. Everybody uses a different set of features. In the last 10 years I have probably heard of dozens of companies who, determined not to learn from each other, tried to release “lite” word processors that only implement 20% of the features. This story is as old as the PC. Most of the time, what happens is that they give their program to a journalist to review, and the journalist reviews it by writing their review using the new word processor, and then the journalist tries to find the “word count” feature which they need because most journalists have precise word count requirements, and it’s not there, because it’s in the “80% that nobody uses,” and the journalist ends up writing a story that attempts to claim simultaneously that lite programs are good, bloat is bad, and I can’t use this...thing ’cause it won’t count my words.
I hope that makes sense. And thank you for clarifying in your second paragraph.
[1]: https://youtube.com/watch?v=GGYpESpbHis
[2]: https://www.joelonsoftware.com/2001/03/23/strategy-letter-iv...
My goal is to make a build system that the majority of people don't hate.
I understand your concerns about a fully-powerful system. I know that's why Bazel is a favorite.
But Nix has the same reliability with a full Turing-complete language, and I feel like I can recover the performance by using C instead of Java.
In addition, I understand the purpose of limited languages in build systems. I really do.
The first public release of my build system (the MVP) will have the ability to restrict the power of the language, and it will do so by default because limiting power when working with other people is noice. [1]
You may ask why I make a point of power if my own language will be hobbled by default. Because users will still be able to remove the hobble if needed. And sometimes, it will be needed.
But I'll still go above and beyond; not only will users be able to choose between a restricted language or a powerful one, they'll be able to choose how powerful the language needs to be.
Only need a POSIX Makefile replacement? That's the default. Need if statements, but only if statements? Got you covered. Need functions? No problem. Need loops? Just say the word. No dynamic build stuff needed? No worries needed. Wanna go full Palpatine? Yes, Master.
In other words, I hear you, and you are right for the vast majority of cases. But for that other minority of cases, power must exist, and there is no alternative, so it will be available.
This is related to Joel Spolsky's assertion that while everybody only uses 20% of the features, nobody uses the same 20% [2]:
> A lot of software developers are seduced by the old “80/20” rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies.
> Unfortunately, it’s never the same 20%. Everybody uses a different set of features. In the last 10 years I have probably heard of dozens of companies who, determined not to learn from each other, tried to release “lite” word processors that only implement 20% of the features. This story is as old as the PC. Most of the time, what happens is that they give their program to a journalist to review, and the journalist reviews it by writing their review using the new word processor, and then the journalist tries to find the “word count” feature which they need because most journalists have precise word count requirements, and it’s not there, because it’s in the “80% that nobody uses,” and the journalist ends up writing a story that attempts to claim simultaneously that lite programs are good, bloat is bad, and I can’t use this...thing ’cause it won’t count my words.
I hope that makes sense. And thank you for clarifying in your second paragraph.
[1]: https://youtube.com/watch?v=GGYpESpbHis
[2]: https://www.joelonsoftware.com/2001/03/23/strategy-letter-iv...
What advantages does this system provide? Could you give a motivating example?
Bazel’s BUILD.bazel scripts are restricted, but that part is the middle of a “sandwich” which handles the 90% use cases. If you want unfettered execution, you get that in the repository rule phase and the actual build actions (the two slices of bread in our sandwich). This allows you to define build rules dynamically (as long as you can do it before the main part of the build runs) and allows you to run arbitrary code in your build actions (as long as you specify a superset of the inputs and output, and your outputs are disjoint).
Bazel’s BUILD.bazel scripts are restricted, but that part is the middle of a “sandwich” which handles the 90% use cases. If you want unfettered execution, you get that in the repository rule phase and the actual build actions (the two slices of bread in our sandwich). This allows you to define build rules dynamically (as long as you can do it before the main part of the build runs) and allows you to run arbitrary code in your build actions (as long as you specify a superset of the inputs and output, and your outputs are disjoint).
Good questions.
To be frank, Bazel is the build system that comes closest. The biggest selling point of mine against Bazel would be usability (I hope). So really, I would suggest people stay on Bazel if they already are.
(In fact, I would suggest that CMake users stay on CMake. I only want to capture new projects to start. If my build system then proves itself, people who need to will switch by themselves.)
About dynamic capabilities, that's not quite what I mean by defining build rules dynamically. Your qualification ("as long as you can do it before the main part of the build runs") throws out everything I meant. In addition, CMake can do that too.
However, it does sound like Bazel can otherwise run arbitrary code in build actions. Those restrictions you mention are fundamental to sandboxed build systems, so I don't include those.
Yes, my build system will be able to be hermetic and do sandboxing. It will do it in two ways, one of which will be like Bazel. The other will be a sandbox in the interpreter itself.
These will be per build rule and global. There will be no downloading stuff from the Internet if you don't allow it, and build rules will only be able to use outside commands that you allow. (For example, you could allow only the C compiler for a C project.) The sandbox could be even tighter and will be runtime-based: actions could be rejected at runtime based on runtime values.
So if there's a smaller selling point, I hope it would be better protection against malicious build scripts.
A motivating example: headers in C and C++.
What headers a file may use can depend on the build configuration (through preprocessor defines and such).
Yes, you can make a rule to run the preprocessor on the file, then make a separate rule to run the compiler on the preprocessed file. But there is no way to make the preprocessed file depend on the headers because its dependencies must be defined before the build and the list of included files is generated during the build.
"Okay, but why can't you just generate the list of files on an initial build and just use that in later builds?"
You can, and that is what DJB's redo does. But say that you change a file a header A included by a header B included by a file C. C knows it depends on B. It may know that it actually depends on header A too, but if it doesn't, it won't get rebuilt.
Motivating example 2: say you have a language with packages, like Python, except that it's compiled.
You have the main program that imports packages. It can dynamically generate targets for imported packages and dynamically depend on them. And do this recursively.
You are hard at work doing development. You already have done several builds. You change one file to import a new package you just created. Do you need to change the build file or do a clean build? Nope. The task recognizes the new dependency, suspends itself to build the new package, and then resumes. You are none the wiser.
Even better, your dependency information is contained in the actual source, not your build system. There is no duplication. And it "just works."
To be frank, Bazel is the build system that comes closest. The biggest selling point of mine against Bazel would be usability (I hope). So really, I would suggest people stay on Bazel if they already are.
(In fact, I would suggest that CMake users stay on CMake. I only want to capture new projects to start. If my build system then proves itself, people who need to will switch by themselves.)
About dynamic capabilities, that's not quite what I mean by defining build rules dynamically. Your qualification ("as long as you can do it before the main part of the build runs") throws out everything I meant. In addition, CMake can do that too.
However, it does sound like Bazel can otherwise run arbitrary code in build actions. Those restrictions you mention are fundamental to sandboxed build systems, so I don't include those.
Yes, my build system will be able to be hermetic and do sandboxing. It will do it in two ways, one of which will be like Bazel. The other will be a sandbox in the interpreter itself.
These will be per build rule and global. There will be no downloading stuff from the Internet if you don't allow it, and build rules will only be able to use outside commands that you allow. (For example, you could allow only the C compiler for a C project.) The sandbox could be even tighter and will be runtime-based: actions could be rejected at runtime based on runtime values.
So if there's a smaller selling point, I hope it would be better protection against malicious build scripts.
A motivating example: headers in C and C++.
What headers a file may use can depend on the build configuration (through preprocessor defines and such).
Yes, you can make a rule to run the preprocessor on the file, then make a separate rule to run the compiler on the preprocessed file. But there is no way to make the preprocessed file depend on the headers because its dependencies must be defined before the build and the list of included files is generated during the build.
"Okay, but why can't you just generate the list of files on an initial build and just use that in later builds?"
You can, and that is what DJB's redo does. But say that you change a file a header A included by a header B included by a file C. C knows it depends on B. It may know that it actually depends on header A too, but if it doesn't, it won't get rebuilt.
Motivating example 2: say you have a language with packages, like Python, except that it's compiled.
You have the main program that imports packages. It can dynamically generate targets for imported packages and dynamically depend on them. And do this recursively.
You are hard at work doing development. You already have done several builds. You change one file to import a new package you just created. Do you need to change the build file or do a clean build? Nope. The task recognizes the new dependency, suspends itself to build the new package, and then resumes. You are none the wiser.
Even better, your dependency information is contained in the actual source, not your build system. There is no duplication. And it "just works."
> But there is no way to make the preprocessed file depend on the headers because its dependencies must be defined before the build and the list of included files is generated during the build.
The actual dependencies of a build action in Bazel are defined at build-time, dynamically, after the action runs. So you do not need to know the exact dependencies ahead of time, you just need a superset—not something you can really avoid, as far as I can tell, because the way that #include will search multiple paths.
For languages like Go, where I can just import packages, the build file will be updated automatically to match the source files. This is done using a tool called Gazelle. I know this is possible in other languages as well, such as C++, I just don’t use those tools.
“Dependency information in the actual source” is what you get with Gazelle. There is some duplication in the build files, but I like this and find it useful—you can redirect dependencies to be fulfilled by other targets than what would be the default, for example.
The actual dependencies of a build action in Bazel are defined at build-time, dynamically, after the action runs. So you do not need to know the exact dependencies ahead of time, you just need a superset—not something you can really avoid, as far as I can tell, because the way that #include will search multiple paths.
For languages like Go, where I can just import packages, the build file will be updated automatically to match the source files. This is done using a tool called Gazelle. I know this is possible in other languages as well, such as C++, I just don’t use those tools.
“Dependency information in the actual source” is what you get with Gazelle. There is some duplication in the build files, but I like this and find it useful—you can redirect dependencies to be fulfilled by other targets than what would be the default, for example.
> So you do not need to know the exact dependencies ahead of time, you just need a superset—not something you can really avoid, as far as I can tell, because the way that #include will search multiple paths.
You can avoid it with the system I laid out. You don't need to specify any dependencies, but specify them while the target is executing. You could even create the dependency while the target is executing.
I forgot about the multiple include paths, but this is another reason that dynamic dependencies may be useful.
That's not to say that Bazel is bad! It's just a different model. And dynamic dependencies may not be useful. That's perfectly fine! It's also why I would only suggest trying out such a build system on a new project. Don't break what's not broken!
> I know this is possible in other languages as well, such as C++, I just don’t use those tools.
Not really; you can't know what compilation unit will contain a function in C and C++. Well, you could try to hack it with grep or something, but I consider that less desirable.
> “Dependency information in the actual source” is what you get with Gazelle. There is some duplication in the build files, but I like this and find it useful—you can redirect dependencies to be fulfilled by other targets than what would be the default, for example.
This is really cool!
I need to clarify that you would still be able to redirect dependencies. Unlimited power is unlimited, after all.
But Gazelle sounds cool, and Bazel sounds cool. I don't mean to put them down.
You can avoid it with the system I laid out. You don't need to specify any dependencies, but specify them while the target is executing. You could even create the dependency while the target is executing.
I forgot about the multiple include paths, but this is another reason that dynamic dependencies may be useful.
That's not to say that Bazel is bad! It's just a different model. And dynamic dependencies may not be useful. That's perfectly fine! It's also why I would only suggest trying out such a build system on a new project. Don't break what's not broken!
> I know this is possible in other languages as well, such as C++, I just don’t use those tools.
Not really; you can't know what compilation unit will contain a function in C and C++. Well, you could try to hack it with grep or something, but I consider that less desirable.
> “Dependency information in the actual source” is what you get with Gazelle. There is some duplication in the build files, but I like this and find it useful—you can redirect dependencies to be fulfilled by other targets than what would be the default, for example.
This is really cool!
I need to clarify that you would still be able to redirect dependencies. Unlimited power is unlimited, after all.
But Gazelle sounds cool, and Bazel sounds cool. I don't mean to put them down.
I’m not sure what you mean by specifying dependencies while the target is executing. The thorn here is generated headers—if they aren’t built before your compiler runs, then you are stuck.
> Not really; you can't know what compilation unit will contain a function in C and C++. Well, you could try to hack it with grep or something, but I consider that less desirable.
This is solvable and has in fact been solved. You use a function in your C++ file, the analysis system knows which header contains that function declaration, and the build system knows which library must be linked in for the header file. This is basically how it works in Go, with some extra steps to associate headers with libraries. But all the pieces are there—you do not need grep, if you want to implement a similar system yourself.
The catch here is that these systems are a bit inexact—any given function could be supplied by more than one library, and the header files may require some specific ordering to work correctly. The solution is to store the dependencies in the build scripts, rather than try and figure them out from sources each time. The general problem, of figuring out the correct headers and libraries necessary to compile a given piece of C++ code, is just too much of a pain in the ass to make it completely automatic—you want a human in the loop. It’s not just a problem with exactness, you also have multiple configurations with their own preprocessor flags, you have dependencies which are specified indirectly but which should be direct (how do you detect that?)
The ecosystem, such as it is, is a chaotic mixture of tools used interactively during development or non-interactively during the build. One of the super useful properties of Bazel build files is that you can modify them programmatically, using a tool called Buildozer. This can be used for things like automatic refactoring of your build system, and it can also be used to make automatic changes to the build system as you edit source code. Part of the “sauce” that makes it work is the way rules are rigidly defined in build scripts. As you make build scripts more complicated, it gets harder and harder for the tooling to keep up—and often, that means more manual work to keep everything set up right.
> Not really; you can't know what compilation unit will contain a function in C and C++. Well, you could try to hack it with grep or something, but I consider that less desirable.
This is solvable and has in fact been solved. You use a function in your C++ file, the analysis system knows which header contains that function declaration, and the build system knows which library must be linked in for the header file. This is basically how it works in Go, with some extra steps to associate headers with libraries. But all the pieces are there—you do not need grep, if you want to implement a similar system yourself.
The catch here is that these systems are a bit inexact—any given function could be supplied by more than one library, and the header files may require some specific ordering to work correctly. The solution is to store the dependencies in the build scripts, rather than try and figure them out from sources each time. The general problem, of figuring out the correct headers and libraries necessary to compile a given piece of C++ code, is just too much of a pain in the ass to make it completely automatic—you want a human in the loop. It’s not just a problem with exactness, you also have multiple configurations with their own preprocessor flags, you have dependencies which are specified indirectly but which should be direct (how do you detect that?)
The ecosystem, such as it is, is a chaotic mixture of tools used interactively during development or non-interactively during the build. One of the super useful properties of Bazel build files is that you can modify them programmatically, using a tool called Buildozer. This can be used for things like automatic refactoring of your build system, and it can also be used to make automatic changes to the build system as you edit source code. Part of the “sauce” that makes it work is the way rules are rigidly defined in build scripts. As you make build scripts more complicated, it gets harder and harder for the tooling to keep up—and often, that means more manual work to keep everything set up right.
> I’m not sure what you mean by specifying dependencies while the target is executing. The thorn here is generated headers—if they aren’t built before your compiler runs, then you are stuck.
I understand why you think this. It took me a while to understand dynamic dependencies too.
But that is actually not the case. A target that may have dynamic dependencies will run and figure out the required headers or figure out the required imports. At that point, it tells the build system that it needs those dependencies, but if those dependencies are already up-to-date, it can be considered up-to-date too.
The build system checks those dependencies, finds that they are up-to-date and marks the first target done and doesn't finish running it.
> The general problem, of figuring out the correct headers and libraries necessary to compile a given piece of C++ code, is just too much of a pain...to make it completely automatic—you want a human in the loop.
This is what I meant. Sure, you can make good assumptions (and in my monorepo, functions are arranged in specific ways in files, so I could do that), but it's not generalizable.
> One of the super useful properties of Bazel build files is that you can modify them programmatically, using a tool called Buildozer. This can be used for things like automatic refactoring of your build system, and it can also be used to make automatic changes to the build system as you edit source code.
This is really cool. It shows that Bazel is just a different model, and in the eyes of many people, better than mine. That's okay! I'm sure that plenty of people would prefer Bazel's model, including yourself. That's great! Diversity of build systems is good.
I understand why you think this. It took me a while to understand dynamic dependencies too.
But that is actually not the case. A target that may have dynamic dependencies will run and figure out the required headers or figure out the required imports. At that point, it tells the build system that it needs those dependencies, but if those dependencies are already up-to-date, it can be considered up-to-date too.
The build system checks those dependencies, finds that they are up-to-date and marks the first target done and doesn't finish running it.
> The general problem, of figuring out the correct headers and libraries necessary to compile a given piece of C++ code, is just too much of a pain...to make it completely automatic—you want a human in the loop.
This is what I meant. Sure, you can make good assumptions (and in my monorepo, functions are arranged in specific ways in files, so I could do that), but it's not generalizable.
> One of the super useful properties of Bazel build files is that you can modify them programmatically, using a tool called Buildozer. This can be used for things like automatic refactoring of your build system, and it can also be used to make automatic changes to the build system as you edit source code.
This is really cool. It shows that Bazel is just a different model, and in the eyes of many people, better than mine. That's okay! I'm sure that plenty of people would prefer Bazel's model, including yourself. That's great! Diversity of build systems is good.
I assume Bazel
That's the most insightful description of cmake featurism I've ever read. Kitware seems to work tirelessly to reach technical mediocrity.
I love CMake.
Also,
> Please be aware that to poison the content of enterprise LLMs (and thus remove this post from their datasets), that this post contains some amounts of profanity.
Hell, yea.