SCons is not slow(github.com)
github.com
SCons is not slow
https://github.com/SCons/scons/wiki/WhySconsIsNotSlow
73 comments
If forking is the issue, why didn't they just do multiprocessing instead with a bunch of worker processes and work going into queues?
Why is Waf slower? It has separate config / build steps.
I don't recall (from time-served at Bloomberg) that Waf was especially slow, but I found that to make it do anything useful, you had to write a lot of obscure, custom Python code nobody could understand.
And you think this custom Python code wouldn't have been necessary with SCons?
Don't really know, tbh.
Things I used SCons on were different from the things I used Waf on. SCons might need as much custom Python for those other uses as Waf. But that does not make Waf any better.
Things I used SCons on were different from the things I used Waf on. SCons might need as much custom Python for those other uses as Waf. But that does not make Waf any better.
I'm still wondering what makes Waf worse in your opinion, if it wasn't performance or custom code required.
Unpleasant memories, which might be mixed up in the Bloomberg experience.
To ne the overriding criterion for a build system is that it must not demand attention. Waf did. SCons seemed not to.
To ne the overriding criterion for a build system is that it must not demand attention. Waf did. SCons seemed not to.
> I don't know if SCons is still slow, but in 2018 it was definitely very, very slow. I doubt it has improved much, as its maintainers were in similar denial then.
And, at this point, its time has passed.
The big projects that were using scons all moved on to other systems for various reasons.
Better to move on to something that actually might generate a critical mass and community.
And, at this point, its time has passed.
The big projects that were using scons all moved on to other systems for various reasons.
Better to move on to something that actually might generate a critical mass and community.
Actually not true. Many large projects continue to use SCons and new projects as well.
I don't think in 2018 we had our heads in the sand about SCons' performance.
As project maintainer I've (along with other community members) have been working on performance since at least 2016.
If you had a hand in getting it to generate ninja scripts: Congratulations. That made a big improvement in the daily experience of people maintaining projects built under SCons.
To be clear: SCons is not notably worse at describing what must be done to build a big project than other systems of its time. Lots of code has been compiled under it. Its deep failure has been in trying to direct the activities of compilations in real time, at which it is bad, gmake is good, and ninja positively excels.
Still, slowness in resolving dependencies is hard to defend. It is a problem in operations long well-understood, with familiar algorithms. A data structure well tuned to the algorithm would make even Python able to solve it quickly enough.
Nowadays the difficult part of a build system is the complexities of managing foreign dependencies, at which CMake is only just adequate.
To be clear: SCons is not notably worse at describing what must be done to build a big project than other systems of its time. Lots of code has been compiled under it. Its deep failure has been in trying to direct the activities of compilations in real time, at which it is bad, gmake is good, and ninja positively excels.
Still, slowness in resolving dependencies is hard to defend. It is a problem in operations long well-understood, with familiar algorithms. A data structure well tuned to the algorithm would make even Python able to solve it quickly enough.
Nowadays the difficult part of a build system is the complexities of managing foreign dependencies, at which CMake is only just adequate.
I feel like if you close with "only 20% slower than make" you can't justify the title "is not slow."
20% is a pretty huge margin for large builds, which I wouldn't consider using make for in the first place. CMake + Ninja has shaved enormous amounts of time off my builds from make, so why should I use something that's even slower?
20% is a pretty huge margin for large builds, which I wouldn't consider using make for in the first place. CMake + Ninja has shaved enormous amounts of time off my builds from make, so why should I use something that's even slower?
As a counterpoint, my builds are mostly too fast to care, but modifying the CMake files that describe them is excruciating. It's a miracle of software engineering that CMake functions, but wow is the language awful.
Is there something specific you dislike about it besides the syntax? I frequently hear complaints about cmake not supporting one thing or another but a lot of those have been addressed in recent releases. It's not clear to me why you say it's a miracle either, AFAIK it does most of the same things any other makefile generator does.
(Disclaimer: I'm not involved in the cmake project but I often see people struggling with it because of misuse, possibly because of expectations that it will work exactly like autotools)
(Disclaimer: I'm not involved in the cmake project but I often see people struggling with it because of misuse, possibly because of expectations that it will work exactly like autotools)
Honestly, I switched to bazel and did not look back.
Yeah we switched from Scons to CMake and I have to admit I loved being able to use python for Scons. CMake's language is unintuitive and clunky. I wish CMake had been designed like LLVM with a front and back end so that there could be a python-based front end (for example).
Yeah, I've been unhappy with any automake system for a long time. Haven't tried ninja though. Scons was fine-ish.
Now that because work forces me to develop on a locked down windows box, make is the last thing I want.
Now that because work forces me to develop on a locked down windows box, make is the last thing I want.
Ninja is not a build system, it is an interpreter for the output of a build system.
A very, very fast interpreter, with truly admirable process-management skills.
A very, very fast interpreter, with truly admirable process-management skills.
there is llbuild...
https://github.com/apple/swift-llbuild
llbuild is a set of libraries for building build systems. Unlike most build system projects which focus on the syntax for describing the build, llbuild is designed around a reusable, flexible, and scalable general purpose build engine capable of solving many "build system"-like problems. The project also includes additional libraries on top of that engine which provide support for constructing bespoke build systems (like swift build) or for building from Ninja manifests.
https://github.com/apple/swift-llbuild
llbuild is a set of libraries for building build systems. Unlike most build system projects which focus on the syntax for describing the build, llbuild is designed around a reusable, flexible, and scalable general purpose build engine capable of solving many "build system"-like problems. The project also includes additional libraries on top of that engine which provide support for constructing bespoke build systems (like swift build) or for building from Ninja manifests.
> but modifying the CMake files that describe them is excruciating.
I've been using CMake for over a decade and the only time editing CMake files was not straight-forward was when whoever wrote them had no idea of what they were doing and were trying to crowbar a round peg into a square hole.
With "modern cmake", which is modern in name only, everything is declarative, modularized, and straight-forward. You just specify targets, target properties, and that's it. I have absolutely no idea of where all the drama comes from.
I've been using CMake for over a decade and the only time editing CMake files was not straight-forward was when whoever wrote them had no idea of what they were doing and were trying to crowbar a round peg into a square hole.
With "modern cmake", which is modern in name only, everything is declarative, modularized, and straight-forward. You just specify targets, target properties, and that's it. I have absolutely no idea of where all the drama comes from.
I guess it’s just a steep learning curve. I’ve been doing some cross platform JNI and python development with cmake and as someone who’s not intrinsically familiar with it, it’s very difficult to “grasp” how everything relates, how I can properly make cmake detect some arcane libraries on computers installed by customers, even when they put them in weird paths, etc etc.
I guess these are problems with all builds systems, but cmake feels like it’s hiding everything behind macros and it’s really difficult to understand what they are actually doing under the hood sometimes.
I guess these are problems with all builds systems, but cmake feels like it’s hiding everything behind macros and it’s really difficult to understand what they are actually doing under the hood sometimes.
> I feel like if you close with "only 20% slower than make" you can't justify the title "is not slow."
IIUC, the 20% slowdown happens with a build that does nothing, it's just the scons overhead. (Not sure if they replaced the compiler invocations with echo, or if they touched the result files right from Python, for this number. They mentioned both, but I don't remember which one of them resulted in the 20% figure.)
The 20% figure should go down for real world builds.
IIUC, the 20% slowdown happens with a build that does nothing, it's just the scons overhead. (Not sure if they replaced the compiler invocations with echo, or if they touched the result files right from Python, for this number. They mentioned both, but I don't remember which one of them resulted in the 20% figure.)
The 20% figure should go down for real world builds.
Unless I'm misreading your description though, the problem is that "a build that does nothing", or "a build that does very little" is absolutely a real world build, because it's almost exactly what happens when you do incremental recompilation. It's one of the most common cases you would encounter.
Sure, but re-linking was the big factor, or god forbid you touched a header file that was used in 400 code units -- or your ccache corrupted itself, or you changed your compiler flags (which would force ccache to recompile everything anyway).
As much as I don't like programming golang, they did get compilation right. And Scons/ninja/automake/cmake issues just went away.
As much as I don't like programming golang, they did get compilation right. And Scons/ninja/automake/cmake issues just went away.
> or god forbid you touched a header file that was used in 400 code units
In C/C++ that means you decided to change 400 translation units.
That's not a problem with the build system at all. That's a problem with how the developers working on the project failed to follow basic best-practices in software development. No build system saves you from the mess you create yourself.
In C/C++ that means you decided to change 400 translation units.
That's not a problem with the build system at all. That's a problem with how the developers working on the project failed to follow basic best-practices in software development. No build system saves you from the mess you create yourself.
> IIUC, the 20% slowdown happens with a build that does nothing, it's just the scons overhead.
That's the whole point of performance measurement for build systems. The overhead is precisely what counts here.
If you have many source files to recompile, the fraction of time spent in the build system itself is going to quickly drop to almost zero : build-system stuff is generally several orders of magnitude faster than compiler stuff.
So it would make little sense to compare build-system performance on full builds.
That's the whole point of performance measurement for build systems. The overhead is precisely what counts here.
If you have many source files to recompile, the fraction of time spent in the build system itself is going to quickly drop to almost zero : build-system stuff is generally several orders of magnitude faster than compiler stuff.
So it would make little sense to compare build-system performance on full builds.
I'm just kind of amazed that there is this kind of lack of understanding of performance & measurement amongst people working on optimizing low-level technical tools like build systems.
Ironically, hitting the front page of HN with a title of "SCons Is Not Slow" is, for me, excellent circumstantial evidence that SCons is, in fact, slow.
Scons is definitely slow in some scenarios. It has O(m * n) performance if a rule has m inputs and n outputs. I tdiscovered this when working with a code base that had a single rule with all .ts files as inputs, and all .js files as outputs. As the number of files we had got bigger, SCons got painfully slow. Of course we can rewrite our rules, but that shouldn’t be necessary, especially since it was quite hard to figure out why our builds got so slow.
Slow or not, it becomes cryptic to write Sconsfiles with any complexity, and it's impossible to debug.
It's python, so you can write your own abstractions as you see fit. Like any general purpose programming language, you can make something maintainable or you can make a mess.
Sometimes having a tool prevent you from shooting yourself in the foot is kinder.
But it's more than that, a good build system does not need to be Turing complete, and having Turing completeness imposes certain requirements on it that may make it inherently slower.
But it's more than that, a good build system does not need to be Turing complete, and having Turing completeness imposes certain requirements on it that may make it inherently slower.
Going along with the firearm analogy, it's only fully useful and powerful tool once you've removed it from its case and removed the trigger lock. Any given firearm may have additional safety features such as interlocks, but they're only effective if they stay out of the way when you intend to shoot. You avoid accidentally shooting yourself and others through training, practice and discipline: treat any firearm as if it were loaded, keep your finger off the trigger until you intend to shoot, etc.
When setting up a build system for multiple people to use, it's important to agree on a general approach and some rules to follow in order to keep the system from turning into a mess. It's code like anything else, and benefits from training, practice and discipline. Using a language that most folk already know is almost always better than using an obscure one out of a sense that the obscurity makes it safer.
When building up a build system, I suggest focusing on designing abstractions to make 90% of use cases trivial for other developers to use. Spitting out libraries and linking them into binaries, existing autocoding scripts, etc. The other 10% of cases that other folk run into, loudly volunteer to make yourself available to help them work through it. Most of the time, there's a way to do what they need that isn't horrendous, or you can take the opportunity to improve your abstractions.
If your problem is that other developers you work with just go in with a hatchet and make a mess without informing anyone, perhaps it's time to adobt some processes to address that, such as better version control hygiene and code reviews.
When setting up a build system for multiple people to use, it's important to agree on a general approach and some rules to follow in order to keep the system from turning into a mess. It's code like anything else, and benefits from training, practice and discipline. Using a language that most folk already know is almost always better than using an obscure one out of a sense that the obscurity makes it safer.
When building up a build system, I suggest focusing on designing abstractions to make 90% of use cases trivial for other developers to use. Spitting out libraries and linking them into binaries, existing autocoding scripts, etc. The other 10% of cases that other folk run into, loudly volunteer to make yourself available to help them work through it. Most of the time, there's a way to do what they need that isn't horrendous, or you can take the opportunity to improve your abstractions.
If your problem is that other developers you work with just go in with a hatchet and make a mess without informing anyone, perhaps it's time to adobt some processes to address that, such as better version control hygiene and code reviews.
> It's python, so you can write your own abstractions as you see fit.
And that's the likely reason why scons projects become cryptic to write, impossible to debug, and unmanageable in general.
And that's the likely reason why scons projects become cryptic to write, impossible to debug, and unmanageable in general.
As the maintainer of SCons I can assure you that we are aware of SCons's performance issue.
The page in question is actually quite old at this point.
Since the content was written Ninja, Cmake, Bazel, and other build systems have launched and become mature.
We are slowly working through the easiest of the list of known performance issues.
In addition MongoDB has contributed their Ninja file output logic. We expect to integrate that in the next major release.
Projects like MongoDB, Godot, VMware, PlatformIO, and many others use SCons. One of it's strengths is the ability to correctly build complex builds.
Please engage with the community our users mailing list, IRC Channel, or discord server if you're a user.
One frustration for the project is that users run into issues using SCons and never reach out and ask for help. Many times a small change in usage will resolve whatever has a user stuck.
The page in question is actually quite old at this point.
Since the content was written Ninja, Cmake, Bazel, and other build systems have launched and become mature.
We are slowly working through the easiest of the list of known performance issues.
In addition MongoDB has contributed their Ninja file output logic. We expect to integrate that in the next major release.
Projects like MongoDB, Godot, VMware, PlatformIO, and many others use SCons. One of it's strengths is the ability to correctly build complex builds.
Please engage with the community our users mailing list, IRC Channel, or discord server if you're a user.
One frustration for the project is that users run into issues using SCons and never reach out and ask for help. Many times a small change in usage will resolve whatever has a user stuck.
I hope that FOSS can converge to a single build system as much as possible, because it's not very convenient to learn them all.
I don't think a single build system is necessary or desirable. meson (as well as waf and scons) depends on python, which is not so nice for bootstrapping. Thus desktop applications and low level userspace plumbing could benefit from different build systems.
That said, I hope waf and scons die off soon (only a handful of packages even use them). CMake and Meson are much better choices.
That said, I hope waf and scons die off soon (only a handful of packages even use them). CMake and Meson are much better choices.
What’s wrong with waf? I’ve been using it lately and think it’s great. Writing build complex build tasks is easy, and the performance is excellent.
I haven’t tried Meson, but CMake cant really do much besides compile the programming languages it explicitly supports. Automation tasks are pretty much impossible.
I haven’t tried Meson, but CMake cant really do much besides compile the programming languages it explicitly supports. Automation tasks are pretty much impossible.
Do you have a link to these wscripts, or a pastebin? I am curious on what types of automations you are referring to.
I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. Only having to know a half dozen build systems is way easier than having to know waf, Scons, meson, autotools, make, setuptools, cargo, qmake, cmake is a pain.
Only 25 packages in the distro I contribute to use waf. Most are audio recording tools that do not get much maintenance. 20 packages use scons. Meanwhile almost 400 use meson, ~500 use qmake, ~1000 use cmake, and over 2000 use autotools.
I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects. Only having to know a half dozen build systems is way easier than having to know waf, Scons, meson, autotools, make, setuptools, cargo, qmake, cmake is a pain.
Only 25 packages in the distro I contribute to use waf. Most are audio recording tools that do not get much maintenance. 20 packages use scons. Meanwhile almost 400 use meson, ~500 use qmake, ~1000 use cmake, and over 2000 use autotools.
> Do you have a link to these wscripts, or a pastebin? I am curious on what types of automations you are referring to.
Not currently, but I do have a half-written article I plan to publish eventually once I complete this project I'm working on, to showcase my use of waf and how it's helped me. That's how much I like it!
But to give you a quick example: my project (a cross-platform game) has a bunch of assets that I am embedding in the executable. To do this, I wrote a custom waf task that automatically generates a char array in a .cpp/.h bundle, and adds it to my build and include paths.
Using that same tool, I am able to embed other assets that need to be transformed in various ways before being embedded. One of those is a bundle of binary files and meta-data combined together inside a single flatbuffer data blob. This involves compiling the schema, generating a small command-line utility to write the blob using the sources generated from the schema compiler, using that utility to generate the blob, and finally embedding the blob using the asset embedding tool. (Plus some of the binary files themselves need to be processed before all of this happens)
I could do that with shell scripts, but waf's design makes it easy to do, in a small amount of reusable code (everything I just described I implemented in <400 lines of python), and in a general-purpose and easy to read programming language. Plus the tasks automatically execute in parallel wherever possible.
> I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects.
I've heard that complaint before, and I can see why it would be annoying for outside maintainers. Waf's customizability means two Waf projects might look nothing alike. So even if one becomes an expert at Waf (which is not very easy considering how bad the docs are), they might be completely lost in someone else's project. I failed a game jam once because I was using Amazon Lumberyard (which uses waf), and even though I completed the game, I couldn't figure out how to compile the damn thing before the deadline.
But if it's any consolation, I don't plan on distributing my project through any linux distros :P
Not currently, but I do have a half-written article I plan to publish eventually once I complete this project I'm working on, to showcase my use of waf and how it's helped me. That's how much I like it!
But to give you a quick example: my project (a cross-platform game) has a bunch of assets that I am embedding in the executable. To do this, I wrote a custom waf task that automatically generates a char array in a .cpp/.h bundle, and adds it to my build and include paths.
Using that same tool, I am able to embed other assets that need to be transformed in various ways before being embedded. One of those is a bundle of binary files and meta-data combined together inside a single flatbuffer data blob. This involves compiling the schema, generating a small command-line utility to write the blob using the sources generated from the schema compiler, using that utility to generate the blob, and finally embedding the blob using the asset embedding tool. (Plus some of the binary files themselves need to be processed before all of this happens)
I could do that with shell scripts, but waf's design makes it easy to do, in a small amount of reusable code (everything I just described I implemented in <400 lines of python), and in a general-purpose and easy to read programming language. Plus the tasks automatically execute in parallel wherever possible.
> I'm mainly coming at it from the perspective of a distro maintainer. Few pieces of software use waf, and digging into the wscripts can be a pain for large projects.
I've heard that complaint before, and I can see why it would be annoying for outside maintainers. Waf's customizability means two Waf projects might look nothing alike. So even if one becomes an expert at Waf (which is not very easy considering how bad the docs are), they might be completely lost in someone else's project. I failed a game jam once because I was using Amazon Lumberyard (which uses waf), and even though I completed the game, I couldn't figure out how to compile the damn thing before the deadline.
But if it's any consolation, I don't plan on distributing my project through any linux distros :P
Having used WAF extensively in large legacy projects, one of it’s large problems is that it isn’t particularly batteries included, and relies on a lot of customizations, and doesn’t have a lot of rails so it’s really easy to break it’s reliability guarantees. And having a unreliable build where one is never sure if a problem will go away by deleting intermediates is super frustrating.
> I don't think a single build system is necessary or desirable.
Why not? In the old days, it was "./configure; make". Today, it's much less certain what build system(s) you'll have to deal with when building a FOSS tool. That's not a problem when everything works, but as experience shows, more often than not you'll run into issues and then I would say it's nice if you are familiar with the system.
And just like with other tools it's nice if everybody uses the same thing. Github wouldn't exist if everybody rolled their own version control system, for example.
Why not? In the old days, it was "./configure; make". Today, it's much less certain what build system(s) you'll have to deal with when building a FOSS tool. That's not a problem when everything works, but as experience shows, more often than not you'll run into issues and then I would say it's nice if you are familiar with the system.
And just like with other tools it's nice if everybody uses the same thing. Github wouldn't exist if everybody rolled their own version control system, for example.
> Github wouldn't exist if everybody rolled their own version control system, for example.
It also wouldn't exist if no one did. It's all about compromises. Having too many tools for the same job isn't good, but monoculture isn't either.
It also wouldn't exist if no one did. It's all about compromises. Having too many tools for the same job isn't good, but monoculture isn't either.
If Cargo didn't exist I would believe you. But a group of people learning from past mistakes and doing most of the things right from the start has convinced me otherwise.
Sadly when it comes to C++ I don't think we'll get a do-over. Too many build systems exist already, and important libraries from a decade or two ago will use those systems until the end of time, and so will the projects that depend on them indirectly.
Sadly when it comes to C++ I don't think we'll get a do-over. Too many build systems exist already, and important libraries from a decade or two ago will use those systems until the end of time, and so will the projects that depend on them indirectly.
I’m going to sit in my little corner of the world and wish hg and qmake had won... while learning all of the other ways to generate Makefiles
FOSS build systems are https://xkcd.com/927/ on repeat.
Unifying around a single thing is not something FOSS likes to do.
This is not FOSS specific. In companies you can observe the same tension between the efficiency of unity and the flexibility of diversity.
There's an xkcd for that. In seriousness, I wish the same thing, but it's disruptive and thankless work to ask of volunteers
I don't know, a lot of FOSS seems to have unified around Linux for whatever reason.
Linux is really 5 million different systems though. Debian or redhat (or etc)? Systemd or init? Bash or zsh? Gnome or KDE? X11 or Wayland? GCC or clang? Almost every single component has multiple choices.
I've been tossing around thoughts for this recently. I think the solution is something that, like scons, is built on top of a proper programming language. Unlike scons, however, it should be imperative, rather than declarative.
Core primitives, like 'a file is out of date iff its modification time or hash has changed', should be expressible 'in userspace', as it were—the core of the build system shouldn't know about them. (Though it should have some concept of a target being up-to-date and not being rebuilt, it shouldn't need to know about files.) This, combined with the ability to create proper abstractions, means you can emulate the primitives of all existing build systems and give them a common substrate.
(Bonus: assuming the core primitives are done right, you can still get performance as good as ninja. Which is good, because you won't be able to compile to a build system like ninja.)
Core primitives, like 'a file is out of date iff its modification time or hash has changed', should be expressible 'in userspace', as it were—the core of the build system shouldn't know about them. (Though it should have some concept of a target being up-to-date and not being rebuilt, it shouldn't need to know about files.) This, combined with the ability to create proper abstractions, means you can emulate the primitives of all existing build systems and give them a common substrate.
(Bonus: assuming the core primitives are done right, you can still get performance as good as ninja. Which is good, because you won't be able to compile to a build system like ninja.)
The answer to this is "use Shake", which is similar to what you describe. The secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files — so you can depend on environment variables, file contents, results of dynamic commands, etc — you need to be able to express "dynamic" rules where you can compute new dependencies for a rule "on the fly" while it's executing, and you need elbow grease to write it as an API. Shake also has had a lot of research done about it comparing it to other systems, is used very successfully, and has an array of nice features other systems don't, including .ninja file compatibility and build profile reports: https://shakebuild.com/
Shake is written in Haskell as a library (so your build rules have to be written in Haskell too, as programs) but you could either wire up a language via the FFI to make some classes of build rules easily expressible in JavaScript/Lua, or in theory rewrite the core of Shake in another language too. The theory is all there, though.
My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity (multi language builds, requires parsing or other "advanced" features, etc). As a bonus it's also very fast, within a few % of Ninja when parsing Ninja files, and sometimes faster than that. I was surprised when it came in as 2x as fast as Ninja on a large-ish C codebase at work, due to having picked a different "build schedule" that distributed compilations across cores better.
Shake is written in Haskell as a library (so your build rules have to be written in Haskell too, as programs) but you could either wire up a language via the FFI to make some classes of build rules easily expressible in JavaScript/Lua, or in theory rewrite the core of Shake in another language too. The theory is all there, though.
My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity (multi language builds, requires parsing or other "advanced" features, etc). As a bonus it's also very fast, within a few % of Ninja when parsing Ninja files, and sometimes faster than that. I was surprised when it came in as 2x as fast as Ninja on a large-ish C codebase at work, due to having picked a different "build schedule" that distributed compilations across cores better.
> My personal rule of thumb is: use a "specific" build system if possible, but use Shake the second any build system requires even a tiny bit of complexity
Systems evolve. You don't want to rewrite your build scripts the moment you cross the complexity line.
Your mention of Shake is nice though. However, I'm not sure if it could gain widespread adoption, since it's based on Haskell which is not very popular unfortunately (and has a bad name when it comes to learning curve).
Systems evolve. You don't want to rewrite your build scripts the moment you cross the complexity line.
Your mention of Shake is nice though. However, I'm not sure if it could gain widespread adoption, since it's based on Haskell which is not very popular unfortunately (and has a bad name when it comes to learning curve).
Shake is interesting. I've been using bazel with a degree of success. It seems to incorporate these same concerns and scales out well to large builds. I'm not sure if shake solves enough of a new set of problems to really take hold.
Interesting. That looks like—mostly—what I was aiming for. (Though I would prefer lisp to haskell for this application, mainly because of metaprogramming.) However, I'm somewhat suspicious of this:
> A target is a file we want the build system to produce
This is somewhat arbitrary, and shouldn't be mandated by the build system. You say 'the secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files', but that doesn't seem to be corroborated?
They do show examples of 'phony' targets, but why not simply eliminate the conflation of build targets and files entirely? Phony targets are just a holdover from make.
> A target is a file we want the build system to produce
This is somewhat arbitrary, and shouldn't be mandated by the build system. You say 'the secrets are that you need a more abstract concept of "targets" or "dependencies" that aren't just files', but that doesn't seem to be corroborated?
They do show examples of 'phony' targets, but why not simply eliminate the conflation of build targets and files entirely? Phony targets are just a holdover from make.
______________________________________________________
A single-build-system world would need to see a build system that recognizes all languages and use cases as equally important.
There's very few build systems that actually support this in a reproducible and hermetic way.
There's very few build systems that actually support this in a reproducible and hermetic way.
Indeed, I had the misfortune of using SCons on one project back in 2017-2019. It was ridiculous how slow it was. Even for the case when "Nothing to be done" was the result it took 20-40 seconds to output the message.
Thankfully we switched to ninja.
Thankfully we switched to ninja.
SCons is the worst experience I've had with a build system. No one, even its advocates, ever really understood what was going on. Things became so much simpler and orders of magnitude faster when we switched away
You must not have used cmake. It’s fast but man is it cryptic and has a steep learning curve.
Maybe add (2014) to the title? It looks like the text may have been revised a few times since, but the original appears to be at least that old.
I have not tried SCons, and I have not heard about it in a while.
Is it good? who does it compare to CMkae or Bazel?
For context, I don't like autotools.
For context, I don't like autotools.
It's good in that it's mature, cross platform, and it's python so you can build whatever you want on top of it. We use it at work for a relatively large codebase that targets multiple languages and architectures.
I like python a lot better than cmake's syntax. It's nice to be able to define functions and importable modules to abstract away the ugly details of the build so that other developers can safely use them without being build system experts.
Bazel is cool, but it's relatively immature. Last time I looked into it, you couldn't just install it easily. It's easy enough to make scons look like bazel. I suspect bazel handles very large codebases more efficiently.
I like python a lot better than cmake's syntax. It's nice to be able to define functions and importable modules to abstract away the ugly details of the build so that other developers can safely use them without being build system experts.
Bazel is cool, but it's relatively immature. Last time I looked into it, you couldn't just install it easily. It's easy enough to make scons look like bazel. I suspect bazel handles very large codebases more efficiently.
There is literally nobody who will admit to liking autotools. But they used to solve many problems we don't have anymore, so we used them.
SCons is bad, in that it is slow, slow, slow, slow. If your time is worth nothing, it can in fact be made to do things. But other alternatives can, too, and faster.
SCons is bad, in that it is slow, slow, slow, slow. If your time is worth nothing, it can in fact be made to do things. But other alternatives can, too, and faster.
Do people use tup?
I used to use tup for small projects but then I switched to CMake/ninja.
I can't say anything against tup other than it is not widely used. In terms of performance, I did not notice any difference between tup and ninja -- but these were small projects.
I can't say anything against tup other than it is not widely used. In terms of performance, I did not notice any difference between tup and ninja -- but these were small projects.
Tup requires FUSE, so is a deal breaking dependency for many corporate setups due to security.
If you’re into Tup, you might also like https://github.com/droundy/fac
Fun fact: ninja was inspired by Tup, but is a bit more geared toward pragmatic usages.
If you’re into Tup, you might also like https://github.com/droundy/fac
Fun fact: ninja was inspired by Tup, but is a bit more geared toward pragmatic usages.
I think tup used to use LD_PRELOAD and seems to still support it:
https://github.com/gittup/tup/issues/394#issuecomment-636378...
https://github.com/gittup/tup/issues/394#issuecomment-636378...
We need a deep learning based build system.
When we used it at MongoDB, somebody (thanks Mathias!) put together apparatus to make it generate a ninja config instead of building by itself. That was fast! Anyway, after you had the ninja config.
But the edit-build-test cycle did not include a slow SCons run anymore, and that was huge.
The MongoDB internal wiki had a page titled "SCons Isn't That Slow", with advice for how to live with how slow it really was, later including ("unofficial") ninja and ccache setup. (Which I added.)
But at least it wasn't WAF.