Benchmarks of various C++ build tools(retropaganda.info)
retropaganda.info
Benchmarks of various C++ build tools
http://retropaganda.info/~bohan/devel/wonderbuild/benchmarks/time.xml
14 comments
The cmake benchmarks used the make backend on linux and cygwin, and the vcbuild, msbuild and mscv-ide backend on windows/msvc.
The ninja backend is indeed faster than the make one. An update to these benchmarks is needed to take it into account.
PS: I fixed the broken links in that benchmark page.
PS: I fixed the broken links in that benchmark page.
After maybe three years of experience with cmake I must admit its scripting language is just disgusting. Makefile provided a nice, well thought out, simple DSL, with just three concepts: targets, dependencies and actions, and while it didn't provide enough support for handling multiple compiler toolchains, etc. I think replacing it with the cmake shell like 'scripting language', sporting one data structure (string; lists are really just semicolon separated strings...), is an example of very, very bad engineering.
Yes. If they had used something like Python, it would probably be heaps better. (Though to be honest I'm not really sure I trust the cmake developers' collective judgment. I'm sure they could manage to make it awful.)
But, even though it's horrid, I do find it CMake easier to work with than make on a day to day basis. CMake's scripting language is at least straightforwardly imperative; CMake has at least some support for working around certain platform/toolchain differences (though this support is rather half-hearted in some key places); it has a couple of handy features that make working with C/C++ libraries quite easy; it has builtin support for test suites. All more convenient than doing it all by hand in GNU make, and you've got support for multiple backends too. Oh, and it supports Windows, too.
Though I admit it only through gritted teeth, it provides value.
But, even though it's horrid, I do find it CMake easier to work with than make on a day to day basis. CMake's scripting language is at least straightforwardly imperative; CMake has at least some support for working around certain platform/toolchain differences (though this support is rather half-hearted in some key places); it has a couple of handy features that make working with C/C++ libraries quite easy; it has builtin support for test suites. All more convenient than doing it all by hand in GNU make, and you've got support for multiple backends too. Oh, and it supports Windows, too.
Though I admit it only through gritted teeth, it provides value.
> My notes don't say anything about builds where only one or two files were changed, but my recollection is that that case was noticeably improved as well
I don't have timings, but this has been my experience as well.
I don't have timings, but this has been my experience as well.
This is pretty old and the title should reflect that. The versions of the software used are ancient by many people's standards: Linux 2.6.24 (released six years ago, in January 2008), ext2 file system (ext3 was introduced in the 2.4 series, seven years before that kernel release), gcc 4.2.4 (also 2008, 4.2 series is from 2007), cmake 2.4.7 (2007). Overall it seems like these undated measurements were performed in early 2008, nearly six years ago. A lot has changed since then and I don't see how this article in its current form should still be relevant today.
I take note that I should put a date on the page!
I actually ran again some of these benchmarks again over the years (last time about one years ago), on the same hardware and updated software, and results didn't change significantly.
FWIW, at the time these results were first published, I had my working dir on ext2 because it had higher performance than ext3.
Hi, I'm the author of these benchmarks, and yes, they are getting old. One of Scons' maintainers kindly asked me a few days ago to try again with current Scons version, so I've started to update and run these benchmarks again with current software and hardware (SSD).
It takes some time to do it right (blame API breakage!) and collect all the results, so I won't be able to publish them before at least a month or so.
I will include the meson+ninja and cmake+ninja combinations, and the new CBS tool (an experimental build system from Qt).
Nice to see that WAF, being python is competitive. It goes to show that smart can beat brawn.
Agreed, but it seems those benchmarks were made to promote wonderbuild. I'm not sure if it's still updated (it doesn't seem so), and it uses old versions for cmake and waf.
it would also be nice to compare other build system like ninja+meson.
it would also be nice to compare other build system like ninja+meson.
It's only really SCons which gives Python as a build system a bad name in terms of speed in my experience.
And judging by the "number of system calls" section, SCons's performance problems don't appear to be due to the Python runtime.
I'd say Python's runtime only has a small fixed startup cost that prevent it from beating natively-compiled tools in no-op rebuilds (when no source file has changed). Some tools like tup or ninja reach insanely low times that are below the startup time of Python itself. But at such low times, it doesn't matter anymore whether it's 10ms or 100ms.
I mention this because I'm using cmake for a current project, and backend choice seems to be somewhat important. I found a noticeable improvement when I switched from using a make-based backend (NMake on Windows, GNU Make on OS X and Linux) to the Ninja one (see https://github.com/ninja-build/ninja).
Based on my notes from the time, full build times were basically unaffected on a per-core basisi - but no-op builds were markedly quicker. My notes don't say anything about builds where only one or two files were changed, but my recollection is that that case was noticeably improved as well.
For whatever it's worth, figures from my notepad from a few months ago. Apologies for the scrappiness; these were just brief notes I took at the time, all rather random, just to determine whether the improvement was ~0, or >0... no more than that :/
All times in seconds. All tests performed on my laptop - 3.1GHz dual-core i7.
Windows in a 3-core VM:
OS X native/Linux 4-core VM (unsure which - probably Linux VM):
(Other notes: cmake works on Windows, and has a reasonably-sized ecosystem, but... OMG. The scripting language is fucking insane. Still worth a look though.)