Building stuff from source(jott.live)
jott.live
Building stuff from source
https://jott.live/markdown/building_stuff_from_source
37 comments
Let's make a distinction. You're you're talking about how to build C or C plus plus code that is distributed with a Linux distribution in most cases.
Been there, done that. What I'd love to understand now is how to build front-end packages with node, typescript, vue, react Etc.
I find it ironic that the JavaScript ecosystem, based on an interpreted language, now requires more time and finesse to build from source than most kernel modules.
Been there, done that. What I'd love to understand now is how to build front-end packages with node, typescript, vue, react Etc.
I find it ironic that the JavaScript ecosystem, based on an interpreted language, now requires more time and finesse to build from source than most kernel modules.
npm build?
If it were only that easy. Large mature projects often have much more complicated build steps than just invoking npm, especially if you want to build incrementally so you can edit/run/debug faster than the 15+ minutes it takes to do a full build. Dev modes, logging, minimization etc. are now easily as complex as 90's era C/C++ builds.
Simple reason: too much money thrown in the industry and the need to terraform a platform that's essentially a dumpster fire.
I’ve really taken a liking to making my projects require nothing but docker to build successfully. This mostly works until building something like a desktop UI or mobile application. For the former, I make it with create-react-app and wrap it with electron if I can, and similarly react native for mobile. (If higher performance on the UI side were ever necessary then I can consider native UIs, but usually I’m the only developer on personal projects and I want it to run on any platform and UI performance is minimal concern) Keeping it compartmentalized to a docker compose file (and a simple sanity check script to ensure that Docker and whatever is needed for react/electron/react native in the UI case is installed is then pretty easy to share between projects. It makes it simple to have builds work no matter what machine I’m on, and easy for devs that want to hack on top of it to just clone it and go.
[deleted]
One pet peeve of mine is build tools that (are configured to) hide the compiler command lines and instead display pretty green "OK" or red "FAIL" messages.
When, invariably, a compilation fails in the middle of the build, it gets really frustrating. If you want to reproduce just the error, and play around to fix it, then you are constrained to understand the whole build system, find the different options in the build files, and rebuild everything at every try (which can be very slow, either because even dummy recompiles are slow, or worse because compiler flags have changed).
If instead you have the command line, you can copy-paste it to a new terminal and tweak it until things work. Only then do you have to look into the build system and adjust things accordingly.
When, invariably, a compilation fails in the middle of the build, it gets really frustrating. If you want to reproduce just the error, and play around to fix it, then you are constrained to understand the whole build system, find the different options in the build files, and rebuild everything at every try (which can be very slow, either because even dummy recompiles are slow, or worse because compiler flags have changed).
If instead you have the command line, you can copy-paste it to a new terminal and tweak it until things work. Only then do you have to look into the build system and adjust things accordingly.
Usually you can disable this by passing `--disable-silent-rules` to `configure` and/or `V=1` to `make`.
If you're willing to see a lot of CMake's own carry-on, you can also do `make VERBOSE=1` or `cmake --build builddir -- VERBOSE=1` there.
I've met more than one person who does this on the Mac for all their "Unix ports" rather than use a package manager like Homebrew or MacPorts. I find that fascinating, but a little quixotic.
I do the same for projects I'm looking at, and I'm on linux. Gives a good understanding of the thing you're looking at. There are many things you can tell by looking at the build process.
There are some things I do not agree with the author though.
Pure make: I think it's refreshing to see, nowdays. Many autotools projects do not do ANY configuration, and could actually use just make in most cases. But the authors just jumped at the "autoconf/cmake" bandwagon, because they probably were never familiar with make, or just want to follow conventions (which is ok, I guess).
Fixing a make build is leaps and bounds easier than to fix compared to cmake or autotools. If you follow the "standard" C environment variable conventions a makefile can be extremely portable (I've written stuff myself supporting osx/linux/bsd/irix/aix for several years). Vanilla make doesn't support variant builds, but if you ever need that (and are willing to believe that make is not rotten as many people seem to suggest), I suggest you look at makepp.
There's no date on the article, but a common incumbent to cmake these days is meson. As a dev, I'm still conflicted between these two: meson has a superior syntax and less baggage, but everything else seems inferior to me at the moment, especially the documentation.
Building from source also gives you an idea of how the author is managing dependencies, which ones were chosen, and so on.
A big one for me is how I still consider a package with two external (that you have to build yourself), robust dependencies much better than 20 smaller cargo-managed crates that require 300mb to build, or ~100 packages pulled by tox when you actually _need_ to contribute to the python package...
This is great way to weed off between multiple candidate projects when you have the luxury of doing so.
There are some things I do not agree with the author though.
Pure make: I think it's refreshing to see, nowdays. Many autotools projects do not do ANY configuration, and could actually use just make in most cases. But the authors just jumped at the "autoconf/cmake" bandwagon, because they probably were never familiar with make, or just want to follow conventions (which is ok, I guess).
Fixing a make build is leaps and bounds easier than to fix compared to cmake or autotools. If you follow the "standard" C environment variable conventions a makefile can be extremely portable (I've written stuff myself supporting osx/linux/bsd/irix/aix for several years). Vanilla make doesn't support variant builds, but if you ever need that (and are willing to believe that make is not rotten as many people seem to suggest), I suggest you look at makepp.
There's no date on the article, but a common incumbent to cmake these days is meson. As a dev, I'm still conflicted between these two: meson has a superior syntax and less baggage, but everything else seems inferior to me at the moment, especially the documentation.
Building from source also gives you an idea of how the author is managing dependencies, which ones were chosen, and so on.
A big one for me is how I still consider a package with two external (that you have to build yourself), robust dependencies much better than 20 smaller cargo-managed crates that require 300mb to build, or ~100 packages pulled by tox when you actually _need_ to contribute to the python package...
This is great way to weed off between multiple candidate projects when you have the luxury of doing so.
> But the authors just jumped at the "autoconf/cmake" bandwagon, because they probably were never familiar with make, or just want to follow conventions (which is ok, I guess).
I'd personally recommend everyone use autoconf or cmake (or even better these days, meson) over straight make for the "conventions" purpose if anything. It's very easy to forget build system features that downstream linux distributions and package managers rely on. This includes but isn't limited to setting installation paths (this includes bindir, sbindir, libdir (especially important for multilib), and even mandir, docdir and infodir. don't just assume they're under share/ or share/ even exists!), proper cross-compilation support/detection, inheriting CFLAGS and other variables to set the path for certain programs, and a lot of more minute details.
It saves a lot of time downstream when all the packages can be built using the same script, as a lot of time is wasted on patching custom build systems. Even autotools and cmake (and meson) aren't always perfect at preventing developers from breaking one of their features through e.g. hardcoding/misuses of features, but it's at least less common, as they provide the mechanisms to conform easily.
Hell, even if you use a regular makefile as your "main" build system for personal convenience reasons, it's very appreciated if you can provide at least one of autotools/cmake/meson as an alternative for your friendly neighbor distribution maintainer, lest you end up with a perl-cross-make situation[1].
[1]: https://arsv.github.io/perl-cross/
I'd personally recommend everyone use autoconf or cmake (or even better these days, meson) over straight make for the "conventions" purpose if anything. It's very easy to forget build system features that downstream linux distributions and package managers rely on. This includes but isn't limited to setting installation paths (this includes bindir, sbindir, libdir (especially important for multilib), and even mandir, docdir and infodir. don't just assume they're under share/ or share/ even exists!), proper cross-compilation support/detection, inheriting CFLAGS and other variables to set the path for certain programs, and a lot of more minute details.
It saves a lot of time downstream when all the packages can be built using the same script, as a lot of time is wasted on patching custom build systems. Even autotools and cmake (and meson) aren't always perfect at preventing developers from breaking one of their features through e.g. hardcoding/misuses of features, but it's at least less common, as they provide the mechanisms to conform easily.
Hell, even if you use a regular makefile as your "main" build system for personal convenience reasons, it's very appreciated if you can provide at least one of autotools/cmake/meson as an alternative for your friendly neighbor distribution maintainer, lest you end up with a perl-cross-make situation[1].
[1]: https://arsv.github.io/perl-cross/
Sounds like they want to run ArchLinux but are stuck in the corporate world!
Well, one of them is a former Apple QA lab manager who still really liked the company's products, so probably not in his case. But I get what you mean. :)
Not really, Arch has a package manager.
I did this in the early years of OS X. I mostly use Homebrew now. But Git, for example, I have built from source as I wanted revert some changes in how certain messages are printed.
Missing is instructions for Meson. If you see a meson.build file, run:
meson build
ninja -C build
Or just use Nix and alter or override the package definition to your needs.
Or if you really want to build it interactively
Or if you really want to build it interactively
$ nix-shell '<nixpkgs>' -A somepackage
>Pure make (you only see Makefile)
This is suspicious.
Wut?
Wut?
Most non-trivial software can't be built cross-platform with pure Make and no configuration step. If software only comes with a Makefile, it's usually a good sign that it only builds in a single "blessed" development environment (and as a heavy Mac user, I can tell you that the blessed environment is never Mac).
My software has only Makefile, but it works on both Ubuntu 20.04 and Debian 11.
The Makefile calls meson & ninja & npm.
I tell macOS users to install VirtualBox and then build my software inside.
I tell macOS users to install VirtualBox and then build my software inside.
Which is annoying as virtualbox does not run on Modern mac hardware.
It is not nice on older as you can relatively easily make things work on most Unix systems if you don't use Linux specific calls.
It is not nice on older as you can relatively easily make things work on most Unix systems if you don't use Linux specific calls.
> The Makefile calls meson & ninja & npm
Makefile-that-calls-3-other-build-systems isn't really using Make to build, is it? You are using Make as a glorified batch file. Could just as well use bash.
> I tell macOS users to install VirtualBox and then build my software inside.
This is pretty much the textbook definition of "non-portable build"
Makefile-that-calls-3-other-build-systems isn't really using Make to build, is it? You are using Make as a glorified batch file. Could just as well use bash.
> I tell macOS users to install VirtualBox and then build my software inside.
This is pretty much the textbook definition of "non-portable build"
`export MAKEFLAGS="-j$(nproc)"` (or `sysctl -a hw.ncpu` for Mac users).
That way you neither need to install Ninja nor remember to pass `-j` to `make`.
That way you neither need to install Ninja nor remember to pass `-j` to `make`.
> Bazel (you see .bzl files)
> Prepare to install Java (in [current year]!) as well as the rest of the world. I tend to look for pre-builts binaries or alternative projects at this point.
Lolwut? Is this article like 5 years old or something? Bazel ships as single installer binary for a while now and is available on most linux distros even.
> Prepare to install Java (in [current year]!) as well as the rest of the world. I tend to look for pre-builts binaries or alternative projects at this point.
Lolwut? Is this article like 5 years old or something? Bazel ships as single installer binary for a while now and is available on most linux distros even.
Bazel is great if you're at Google scale, but I agree with the article. Too complex and bloated for my liking.
More complex than autotools? You can cut your build times dramatically even if you’re far from google scale. The improvement we saw at current co was 10x. Try that with autotools
make -j$(nproc)
From the GNU make manual:
If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.
What I tend to do is once I've figured out all the steps, put it in a Dockerfile to automate this for next time. If you're feeling kind, you can also send that Dockerfile as a PR.
FROM ubuntu:21.10
RUN apt install -y $DEPS
ADD . /src
WORKDIR /src
RUN ./configure
RUN make
RUN make installI wish more C/C++ projects used "unity builds" -- there's only one translation unit containing all the code, usually by #include-ing all the .c and .cpp files. It makes "build configuration" incredibly easy by largely doing away with it. And C/C++ compilers just copy paste your code in via the preprocessor when doing #include of headers, etc., anyway, so it's not a wacky concept at all.
- It's just one compiler invocation. If you want to support multiple compilers, just write that one invocation for each one. Usually clang and gcc will be the exact same. cl and clang-cl similarly. Even if you support multiple platforms, writing N command lines for N platforms is usually less characters than any build system out there, and simpler, and less opaque as well.
- You can guarantee exactly the flags you want are being set. No more problems where you set -fwrapv or -march or whatever on one sub-project, but forgot on another, etc., or trouble figuring out what's the "CMake way" of specifying something. Getting these types of flags working with build tools becomes quite frustrating and error prone the more specific your requirements are.
- Not something I dealt with working at trading companies, but distro package maintainers can very easily patch in their own flags, or more often can get by without even that if your unity build can just put $CFLAGS or $CXXFLAGS at the end of the command line.
- Wouldn't tout this as a major benefit as it has saved me maybe once, but it basically forces a compiler error for certain violations of the One Definition Rule, because the compiler can see the duplicate and conflicting definition at compile time, instead of the linker silently picking one of the conflicting definitions and masking the problem.
I've found that there are ample ways to keep compilation speed pretty reasonable (we're talking <10 seconds), even compiling all code at once on large, complicated projects. The general solution is to not let your compile times get crazy in the first place, which means not using a lot of general purpose external dependencies or templates.
Another technique is, for dev, split the unity build in two. Unity build of files you're actively working on or of files you need to use -O0 or -0g on for a while, and unity of the rest, and link.
Games people often carry that idea of a split unity build further. Split the structure of the code itself, and have an "infrastructure" unity build, a "business logic" unity build, and the infra code actually dynamically loads the logic .so/.dll, and can reload it on demand (so you can iterate fast on logic, without recompiling or even linking your infra code). I don't work on games so this hasn't been worth it for my projects, but I have definitely worked on projects structured with a control process (slow, smart, size not a critical issue), and a worker process (fast, dumb, size critical), which gets you pretty much the same structure by different means. You can easily get 1s turnaround for builds of the worker. I never used this structure specifically in order to help dev turnaround, rather it was intrinsic to the project requirements and faster dev was a happy knock-on effect.
I'd be a happy man if I saw a post like this in the future that at least started with:
- It's just one compiler invocation. If you want to support multiple compilers, just write that one invocation for each one. Usually clang and gcc will be the exact same. cl and clang-cl similarly. Even if you support multiple platforms, writing N command lines for N platforms is usually less characters than any build system out there, and simpler, and less opaque as well.
- You can guarantee exactly the flags you want are being set. No more problems where you set -fwrapv or -march or whatever on one sub-project, but forgot on another, etc., or trouble figuring out what's the "CMake way" of specifying something. Getting these types of flags working with build tools becomes quite frustrating and error prone the more specific your requirements are.
- Not something I dealt with working at trading companies, but distro package maintainers can very easily patch in their own flags, or more often can get by without even that if your unity build can just put $CFLAGS or $CXXFLAGS at the end of the command line.
- Wouldn't tout this as a major benefit as it has saved me maybe once, but it basically forces a compiler error for certain violations of the One Definition Rule, because the compiler can see the duplicate and conflicting definition at compile time, instead of the linker silently picking one of the conflicting definitions and masking the problem.
I've found that there are ample ways to keep compilation speed pretty reasonable (we're talking <10 seconds), even compiling all code at once on large, complicated projects. The general solution is to not let your compile times get crazy in the first place, which means not using a lot of general purpose external dependencies or templates.
Another technique is, for dev, split the unity build in two. Unity build of files you're actively working on or of files you need to use -O0 or -0g on for a while, and unity of the rest, and link.
Games people often carry that idea of a split unity build further. Split the structure of the code itself, and have an "infrastructure" unity build, a "business logic" unity build, and the infra code actually dynamically loads the logic .so/.dll, and can reload it on demand (so you can iterate fast on logic, without recompiling or even linking your infra code). I don't work on games so this hasn't been worth it for my projects, but I have definitely worked on projects structured with a control process (slow, smart, size not a critical issue), and a worker process (fast, dumb, size critical), which gets you pretty much the same structure by different means. You can easily get 1s turnaround for builds of the worker. I never used this structure specifically in order to help dev turnaround, rather it was intrinsic to the project requirements and faster dev was a happy knock-on effect.
I'd be a happy man if I saw a post like this in the future that at least started with:
If you see build.sh or build.ps1/bat, just run that and you're done.
The main issue with unity builds is the simple fact that you get very slow incremental builds. When working on a project you generally want to keep a cache of already-built files to avoid wasting time on a rebuild, while not sacrificing runtime speed in the form of disabling optimizations. With C unity builds don't quickly get very bad, but with C++ it grows fast. This goes hand-in-hand with the fact that for clean builds, parallelizing compilation can improve its speed by a significant amount.
And for bigger projects (especially when we're talking C++), unity builds are simply infeasible due to their RAM requirements. Compiling a single file in the firefox code can take up to 3GB at times, and while you may consider it an extreme example, it can be deceptively easy to reach such numbers.
All of this is of course not considering the namespace collision issues, where in C++, the "anonymous" namespace is shared across the entire file, and similarly, in C, functions and variables marked as "static" are only visible in the file they're defined in. Both of these are very useful features to reduce verbosity and reduce namespace clutter.
And for bigger projects (especially when we're talking C++), unity builds are simply infeasible due to their RAM requirements. Compiling a single file in the firefox code can take up to 3GB at times, and while you may consider it an extreme example, it can be deceptively easy to reach such numbers.
All of this is of course not considering the namespace collision issues, where in C++, the "anonymous" namespace is shared across the entire file, and similarly, in C, functions and variables marked as "static" are only visible in the file they're defined in. Both of these are very useful features to reduce verbosity and reduce namespace clutter.
Implicit in my post was an admonition to avoid the kind of code that causes compilation time explosion. I do acknowledge that nowadays it's very uncommon for people to actually do this -- and I count my past self as one of those people failing to manage compile times properly.
Just looking at some cons, with small units you have:
- recompilation of the same headers over and over (not really "incremental" then)
- must pay increased build system costs for each unit (see https://apenwarr.ca/log/20181113) -- something of an open problem
- reduced ability to optimize across units (LTO was a bit finicky 10 years ago and it still is today)
- reduced developer will to keep compilation times reasonable (workarounds exist like "cloud build systems" such as bazel, which retrieve build artifacts from the cloud, because processing some text files locally is too taxing now)
I do agree that there is some project size where a single unity build is too much. But leaving unity/incremental aside, I think there's a strong argument to avoid code that's slow to compile. And if and only if that is agreed upon for a project, then even for very large projects, you can have a unity build for each of the 10 or so modules that make up the system. And if your build is so big, you can use a simple build system like redo for that if you wish, the cost in specification and time is relatively low. However, I think that should be quite rare. We really don't have to pretend we're "optimizing" things by acting like every project we build is the size of firefox or the linux kernel. Let those projects have build systems, and let the remaining 99% of projects just call gcc one or two times.
There's very little in the ecosystem of open source software to make doing unity builds very easy, so I'm not chiding anyone for making a tradeoff and using a behemoth like Qt instead of Dear ImGui, because they need assistive tech to work and there's nothing "unity buildable" that supports it.
Just looking at some cons, with small units you have:
- recompilation of the same headers over and over (not really "incremental" then)
- must pay increased build system costs for each unit (see https://apenwarr.ca/log/20181113) -- something of an open problem
- reduced ability to optimize across units (LTO was a bit finicky 10 years ago and it still is today)
- reduced developer will to keep compilation times reasonable (workarounds exist like "cloud build systems" such as bazel, which retrieve build artifacts from the cloud, because processing some text files locally is too taxing now)
I do agree that there is some project size where a single unity build is too much. But leaving unity/incremental aside, I think there's a strong argument to avoid code that's slow to compile. And if and only if that is agreed upon for a project, then even for very large projects, you can have a unity build for each of the 10 or so modules that make up the system. And if your build is so big, you can use a simple build system like redo for that if you wish, the cost in specification and time is relatively low. However, I think that should be quite rare. We really don't have to pretend we're "optimizing" things by acting like every project we build is the size of firefox or the linux kernel. Let those projects have build systems, and let the remaining 99% of projects just call gcc one or two times.
There's very little in the ecosystem of open source software to make doing unity builds very easy, so I'm not chiding anyone for making a tradeoff and using a behemoth like Qt instead of Dear ImGui, because they need assistive tech to work and there's nothing "unity buildable" that supports it.
* When autoconf files are ancient and do not work, running autoreconf (this time in the source directory) can help.
* I don't like installing stuff in system-wide directories either. Ideally system-wide directories should be touched only by your package manager, so that stale files don't stay there forever. But sometimes files in the object tree are not ready to be used, you really have to install them somewhere. There are two possible ways (with autoconf, at least): either use ../configure --prefix=/install/dir or make install DESTDIR=/destination/dir. They do not do the same exact thing, so in particular with DESTDIR you might still need some tweaking before everything works. But it's a good tool to have in your belt.