Mold/macOS is 11 times faster than the Apple's default linker to link Chrome(twitter.com)
twitter.com
Mold/macOS is 11 times faster than the Apple's default linker to link Chrome
https://twitter.com/rui314/status/1537279524341432320
119 comments
Author here. If you have any questions about mold/macOS, feel free to ask me as a reply.
Does mold support jobserver or some other mechanism to throttle threads?
We use lld currently but had to disable threading as sometimes in CI several of our test binaries would get linked at the same time. Whenever this happened the lld instances appear to have spawned enough threads to overload our Jenkins slave to the point that the master wasn't able to reach it and failed the build.
We use lld currently but had to disable threading as sometimes in CI several of our test binaries would get linked at the same time. Whenever this happened the lld instances appear to have spawned enough threads to overload our Jenkins slave to the point that the master wasn't able to reach it and failed the build.
It's being discussed (https://github.com/rui314/mold/issues/117) but haven't reached any conclusion. The problem is that the jobserver protocol assumes that one process is one job, and its model doesn't fit very well to programs such as mold.
Thank you rui314. There is a lot of good information collected in that issue.
If you use Ninja, you can create a job pool for linking, separate from compiling, and restrict how many simultaneous linker jobs are run. You can even create and set job pools for Ninja through CMake, if you use that in your tooling.
Unfortunately GNU Make offers no such mechanism. And for Ninja build generation I don't think Meson does it either.
Unfortunately GNU Make offers no such mechanism. And for Ninja build generation I don't think Meson does it either.
I was not aware of this feature and we do use CMake. Thank you for letting me know about it!
Can mold also serve as a replacement for dsymutil (i.e. linking DWARF info)?
If not, any plans to support this in the future? :)
If not, any plans to support this in the future? :)
Why is it so much faster?
The biggest reason is because it is multi-threaded. When building a program, the compilation step is parallelized (the build system invokes a compiler for each source file), but the final link step is not. So it is important to make the linker itself multi-threaded.
But even without multi-threading, mold is still faster than other linkers. I can think of various reasons why, but I don't know which attributes how much. I believe the biggest contributor is its efficient data structure -- it is hard to make program faster by writing fast code, but it can naturally be achieved by designing efficient data structures. That said, it is hard to compare two or more programs to find out why one program is faster than the others unless their designs are similar.
But even without multi-threading, mold is still faster than other linkers. I can think of various reasons why, but I don't know which attributes how much. I believe the biggest contributor is its efficient data structure -- it is hard to make program faster by writing fast code, but it can naturally be achieved by designing efficient data structures. That said, it is hard to compare two or more programs to find out why one program is faster than the others unless their designs are similar.
Note that recent lld is also multithreaded, and with tweaks, can be faster than mold at low core counts: https://bugzilla.mozilla.org/show_bug.cgi?id=1746462#c2
I wouldn't be surprised. In mold, if we have more than one choices to implement a feature, I always take the one that scales well for more cores even if it doesn't perform the best on low-core count machines.
My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.
My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.
> My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.
From the manpage of mold-1.3.0, I get the impression mold is designed to scale up to 32 cores, but not more.
From the manpage of mold-1.3.0, I get the impression mold is designed to scale up to 32 cores, but not more.
man mold | rg -C2 32
--threads
--no-threads
Use multiple threads. By default, mold uses as many threads as the number of cores or 32, whichever is the smallest. The reason why it is capped to 32 is because mold doesn't scale well beyond that point. To use only one thread, pass --no-threads or --thread-count=1.
Could you comment on which data structures are most critical to mold’s performance, and what makes them so fast?
Ah, I overlooked this explanation at first: https://github.com/rui314/mold/blob/main/docs/design.md
Thanks for the great write up as well as mold itself!
Thanks for the great write up as well as mold itself!
Looking further into the future and the advent of io_uring in Linux, would you consider special-casing Linux IO ops to use io_uring or do you not expect any speedup there?
I'm not sure if io_uring can improve mold's performance, as it has to access random locations while copying file contents to apply relocations. Currently, we mmap all input files and an output file and use memcpy to copy file contents.
I see mold does not yet support LTO [1]. But if it were to, could we still expect performance gain over lld when doing LTO?
[1] https://github.com/rui314/mold/issues/181
[1] https://github.com/rui314/mold/issues/181
mold does support LTO. It's not faster or slower than lld both in terms of link speed and the output binary's speed.
It might be asking for too much, but this project would be a nice addition to http://aosabook.org
Thanks for your good work! I still remember the O(n^2) complexity of ld.bfd when linking C++ code.
Thanks for your good work! I still remember the O(n^2) complexity of ld.bfd when linking C++ code.
I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!
I'm one data point but I'd buy it in an instant.
yep yep I have it :)))
> I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!
I know this is a type, but it is interesting to see knowledge as a score or commodity you can "earn".
I know this is a type, but it is interesting to see knowledge as a score or commodity you can "earn".
Your tweet shows quicker linking on a Mac yet you say down-thread that Mac executable linking is still not fully supported. So can we or can we not use it today on a Mac? (My main use-case is Rust.)
It can create Mac executables, but mold/macOS is still in pre-alpha and no one should expect it to work for their programs. Once it becomes out of beta, I'll release it as mold 2.0, so please wait for it.
Thanks. I will wait for an official announcement then. Just subscribed to release notifications on GitHub, too.
When are you going to start working on a 10x faster clang++? :-)
That's an even crazier goal which is probably 100x harder than writing a 10x faster linker. But I believe it's technically doable. At least, the world needs more crazy people who believe it is technically doable and take it as a challenge. If I get $$$ by selling the mold project to a big tech, I might be able to create a team with that money to tackle that crazy goal...
More power (and moneys) to you sir - I just used mold 1.3 to link qtwebkit right before this article appeared.
do you have a plan to supersede official linkers in gcc/ llvm?
I don't have a plan, and that's not what I can plan. They can plan in theory, but I believe that's very unlikely to happen.
if mold has multiple advantages over the official ones and no drawback, can't you send them an RFC once mold reaches 1.0?
None of gold, lld or mold can replace GNU ld entirely because they don't cover all features that GNU ld has.
the whole embedded world uses linker scripts. for sure not mold goal. also, they don't have linker performance problems, but the big C++ apps have.
Read the readme. Full compatibility with gold/lld is a non-goal and will significantly slow down the linker IIUC.
I can infer from context that mold is a linker. It took searching for https://github.com/rui314/mold to learn what it actually was, though:
"mold is a faster drop-in replacement for existing Unix linkers. It is several times faster than the LLVM lld linker, the second-fastest open-source linker which I originally created a few years ago. mold is designed to increase developer productivity by reducing build time, especially in rapid debug-edit-rebuild cycles."
I don't understand how things like this still manage to front-page without a cursory explanation.
"mold is a faster drop-in replacement for existing Unix linkers. It is several times faster than the LLVM lld linker, the second-fastest open-source linker which I originally created a few years ago. mold is designed to increase developer productivity by reducing build time, especially in rapid debug-edit-rebuild cycles."
I don't understand how things like this still manage to front-page without a cursory explanation.
I think yn is a mature enough site that stories are placed by the site curators and twitter users with connections to the site moderators are part of the curation process. There have been a lot more twitter posts as new submissions this past year or two.
Also, thank you for finding out what the topic discussion was about and sharing. I couldn't infer the meaning.
Also, thank you for finding out what the topic discussion was about and sharing. I couldn't infer the meaning.
Some posts get picked for a second-chance by moderators - see https://news.ycombinator.com/item?id=26998308 - but it doesn't have any connection to twitterers, nor to 'connections to the site moderators', except in the sense that we recognize some usernames as good submitters.
The intention is just to comb through the stacks looking for good stories that didn't get attention the first time around—especially on topics that aren't correlated with anything else. HN has a wealth of these and I'm pretty sure we still miss half of them.
The intention is just to comb through the stacks looking for good stories that didn't get attention the first time around—especially on topics that aren't correlated with anything else. HN has a wealth of these and I'm pretty sure we still miss half of them.
20 seconds down to 2 seconds is a huge achievement. I'll still be telling my boss I'm waiting for a build while I go for my 15 minute coffee, though
If it takes 20 seconds, you go for a coffee or switch the window to start web browsing, but if it takes only 2 seconds, you can wait without being distracted. So it's more than just saving 18 seconds for each build. It wouldn't interrupt your flow.
I think the point the OP was making is that they still want their break :P
our builds usually take 1-4 hours on windows with crazy conan. even cached binaries don't help much.
a simple rebase could last 2 days.
You'll have to switch to instant coffee!
Apple and Microsoft should really throw some money at Rui; Mold is a massive improvement to developer experience, and the sooner it's ported to their platforms the better.
I find myself defaulting to Linux for Rust development these days, mostly because Mold makes the inner loop of development so much faster.
I find myself defaulting to Linux for Rust development these days, mostly because Mold makes the inner loop of development so much faster.
Rui has made it explicit that a commercial license is available. Hey Rui, you should replace "please call" with a dollar amount in order to increase your conversion rate ;)
I dislike "contact us for pricing" too, but pricing is hard! I don't know how much I should ask for. This is also a bit unusual business model because mold can be used for free. I'm happy if users pay me 10%-20% of the total cost they could save by using mold though.
I sort of wish Apple, Amazon, Google, Microsoft and Facebook all contribute $200K each and say hey can you now make Mold MIT or BSD.
$200K is nowadays an annual total compensation of a junior dev. I believe mold values much more than a one year outcome of a typical 5-person junior dev team.
> $200K is nowadays an annual total compensation of a junior dev
*in the US.
In my country, those salaries are completely unheard of.
*in the US.
In my country, those salaries are completely unheard of.
Not even in most of the U.S. I’ve never broken that salary in the Midwest and I’ve gone from Junior to SWE to Senior to Lead to Manager to Director. Even in total compensation it took me until Director (managing managers who in turn manage individual contributors) to reach that level.
In my country, $200k is what a senior dev makes in 10 years. The issue is, I very much doubt people like Rui could be expected to live in places like mine. So comparing against US salaries is probably what you should do anyway.
I'm curious - is that salary difference enough to cause a significant brain drain? Can you hire really good developers for that price in your country?
I live in the UK and dev salaries are maybe half of the US, but people don't leave in droves because there are other factors - family, friends, quality of life, effort of moving your life across continents, etc.
I live in the UK and dev salaries are maybe half of the US, but people don't leave in droves because there are other factors - family, friends, quality of life, effort of moving your life across continents, etc.
Sure. Significant brain drain might be an understatement — I don't think there are any excellent developers left here (if there were any at all). Anyone half worth their salt either moves to Russia where there was a decent IT sector up until the recent events, or wins a green card/marriage lottery and leaves for the West. FWIW I'm not leaving because I didn't want to live in Russia even before the war started (and even less so now), and I'm not good enough to emigrate to the West.
Very interesting thanks!
As I know, Rui was in the US (when he was at Google), so it is a fair statement.
I am sorry I was not trying to undermine the value of Mold. I was told the average / median of Junior Dev are't even $200K in FAANG. But then I dont work in the US so I dont know.
( I hope more US companies are opened to truly remote )
( I hope more US companies are opened to truly remote )
I didn't take it as an offense, so don't say sorry! I just wanted to say that if a senior dev wants to earn $1M in a few years, they can just find a job and do whatever assigned to them there. They didn't have to take a risk of starting a completely new ambitious open-source project to earn that much.
Easiest way to confirm what value it has is to simply charge for it :)
Look at Sidekiq for inspiration on how to charge for open source.
Look at Sidekiq for inspiration on how to charge for open source.
Please give a list of companies (Europe/Japan) where I can apply for such a salary. I started working recently and I'm at more than 10 times less than this...
I don’t know much about the European job market, but you can see lots of examples of US job positions and their salaries at websites such as https://www.levels.fyi.
I'm not interested in migrating to US for ideological and culinary reasons, but that a very intresting website, thanks!
What's wrong with the current agpl license, especially for a linker?
Corporations don't understand the GPL and how isolated, unmodified tools aren't viral.
Build times are a huge issue impacting how fast I can develop at my job (and I'm fairly certain that's the norm for my team), and I suspect that using a faster linker would mitigate that by a significant chunk. Unfortunately I'm just an IC and don't have the authority to look into something like paying for this, and I'd likely need to actually try it out and show numbers for how much things would improve to get management interested in actually looking into this. Because the project I work on requires interfacing with some cloud services that I can't connect to locally, I would either need to actually build on one of our cloud servers or compile the entire thing locally and copy it over, but I imagine the performance gains from multithreading would not be nearly as great on the 8 cores on my laptop compared to the 64 on my cloud server. I'm a bit hesitant to take initiative and just try this out due to the AGPL license though, since IANAL and I imagine if I tried to get permission for this internally, I'd run into the circular issue that without data to show it's worth it to look into, I'd probably just be told not to do it.
Basically, I think your best chance of generating interest from this is the bottom-up (since the individual engineers are the ones who would be feeling the pain that this could help solve), but I'm not really sure ICs at the companies large enough to be potential customers for this have any likely path forward with the way you've structured things right now. I'm not sure if you're flexible on this at all, and obviously I can't guarantee we'd get anywhere, but if you're interested in hearing more details about the potential use my team (and probably a number of other teams at my company would be able to make use if we were able to work something out), feel free to email me! Any prefix @<my username>.com will forward to my gmail.
Basically, I think your best chance of generating interest from this is the bottom-up (since the individual engineers are the ones who would be feeling the pain that this could help solve), but I'm not really sure ICs at the companies large enough to be potential customers for this have any likely path forward with the way you've structured things right now. I'm not sure if you're flexible on this at all, and obviously I can't guarantee we'd get anywhere, but if you're interested in hearing more details about the potential use my team (and probably a number of other teams at my company would be able to make use if we were able to work something out), feel free to email me! Any prefix @<my username>.com will forward to my gmail.
I was joking and I don't think you need a price on there. But if you're serious, you can put out an asking price and everyone involved should just assume it's negotiable. Just use something like a couple of million USD as a starting point.
> This is also a bit unusual business model because mold can be used for free.
You're selling the copyright, right? That's ok - businesses pay for stuff like this. Although I wonder if instead you'd be better off if you created a corporation and instead offered to sell the company's IP + your commitment to support the work over ~1-2 years. This might be a more common scenario for M&A teams to work with. Especially if you are keen on supporting more target architectures/OSs. Someone like ARM or SiFive or FAANG would easily shell out that kind of money to get mold.
> This is also a bit unusual business model because mold can be used for free.
You're selling the copyright, right? That's ok - businesses pay for stuff like this. Although I wonder if instead you'd be better off if you created a corporation and instead offered to sell the company's IP + your commitment to support the work over ~1-2 years. This might be a more common scenario for M&A teams to work with. Especially if you are keen on supporting more target architectures/OSs. Someone like ARM or SiFive or FAANG would easily shell out that kind of money to get mold.
I wish basic toolchain changes happened faster though. Regular ld is still the default on most Linux dists.
Can you please share the link about how it helps in inner loop? Sorry, I never heard of these tools and google is not returning any useful results
It's not really something you can "link" to tho? But the basic loop of development in most cases (in compiled languages) is making a code change, then building the software (or tests), and then running the software (or tests) to check that your change did what it was supposed to do.
Building the software is divided into two steps; compiling your source files into object files, then linking those object files together into an executable. As a project grows, the time taken by the compile step in this development loop stays roughly constant; if you only change a single source file, only that source file has to be recompiled. However, the link step has to link together all your object files, from scratch, every time, so the time taken by the link step grows roughly linearly as the project size grows.
Since mainstream linkers are fairly slows, an incremental build of huge projects is generally dominated by the link step. I've personally experienced the pain of making a small change to Chromium, waiting a second or so for the source file I changed to be recompiled, then waiting a minute or two for the linker to go through every single object file and produce the final executable. Mold would have reduced the build time in the development loop from minutes to seconds.
I hope that helped.
Building the software is divided into two steps; compiling your source files into object files, then linking those object files together into an executable. As a project grows, the time taken by the compile step in this development loop stays roughly constant; if you only change a single source file, only that source file has to be recompiled. However, the link step has to link together all your object files, from scratch, every time, so the time taken by the link step grows roughly linearly as the project size grows.
Since mainstream linkers are fairly slows, an incremental build of huge projects is generally dominated by the link step. I've personally experienced the pain of making a small change to Chromium, waiting a second or so for the source file I changed to be recompiled, then waiting a minute or two for the linker to go through every single object file and produce the final executable. Mold would have reduced the build time in the development loop from minutes to seconds.
I hope that helped.
[deleted]
The Apple linker also seems to have a habit of randomly changing it's mind about certain niggly details about MachO binaries which is extremely irritating as a compiler dev
What’s an example?
I can't remember the details (I'm stuck on my phone but I'll try and dig it up later if I remember) but we had some issues with certain sections not being to it's liking so code that was fine on one xcode version then broke on upgrade.
Similarly I had to replace how we implement C runtime destructors because Apple deprecated the way to write a destructor in a binary. This wasn't too hard to do, just call atexit from the itanium C++ ABI, but we had basically no warning as far as I'm aware.
Similarly I had to replace how we implement C runtime destructors because Apple deprecated the way to write a destructor in a binary. This wasn't too hard to do, just call atexit from the itanium C++ ABI, but we had basically no warning as far as I'm aware.
I’ve always been more attracted to the higher level languages and followed llvm from a compiler interest. So I know the principles, but I would like to understand more. Especially related to cross compilation.
Does someone have a source explaining the state of the art? With all the different compilers, gcc, clang and the different flavours of linkers: ld, lld, mold, gold, ?
Does someone have a source explaining the state of the art? With all the different compilers, gcc, clang and the different flavours of linkers: ld, lld, mold, gold, ?
A linker seems like it ought to be a simple piece of software. What is a linker doing that takes so long?
Well at a minimum it has to load into RAM all the built objects, which probably include debugging symbols. For something like Chrome, thats probably 20 Gigs...
Huh? Isn't the output of the linker roughly the same size as the sum of the inputs?
> Huh? Isn't the output of the linker roughly the same size as the sum of the inputs?
No. Linkers today even do link-time code generation. Debug info is gargantuan and can be kept as separate files too.
No. Linkers today even do link-time code generation. Debug info is gargantuan and can be kept as separate files too.
I don't know what code linkers are generating on their own but doesn't it sort of imply that the output is at least as large as the input?
Linkers can also do something called "Identical code folding", or ICF, whereby the linker notices that two pieces of code are exactly the same and can merge them. lld's sources include a little overview of how this is done, see https://github.com/llvm-mirror/lld/blob/master/ELF/ICF.cpp
ICF exists but no linker is going to silently do it behind your back without an explicit directive, because it breaks debugging in certain ways. Folded identical functions can't be disambiguated in the file/line tables, so symbolized backtraces may contain impossible calls.
> CF exists but no linker is going to silently do it behind your back without an explicit directive,
Those explicit directives might be more implicit than you think. A linker will likely fold functions declared as inline. Template functions and template classes are implicitly inline, so for example, all uses of std::vector<std::string> will (likely) be implicitly folded together.
Those explicit directives might be more implicit than you think. A linker will likely fold functions declared as inline. Template functions and template classes are implicitly inline, so for example, all uses of std::vector<std::string> will (likely) be implicitly folded together.
Aren't these folded by the compiler already?
Not if they're compiled by separate invocations of the compiler. If a.cpp and b.cpp both have a vector<string> and are compiled independently, the first tool that actually gets to see them both is likely a linker (or archiver but that's a glorified zip tool)
Ok, but if these functions are inlined, would a linker then be able to fold these definitions?
(Or are you assuming that a debug flag is used so that all inlined functions are not really inlined by the compiler?)
(Or are you assuming that a debug flag is used so that all inlined functions are not really inlined by the compiler?)
OK but removing functions that are identical and that have the same name is not "identical code folding". ICF is removing functions that have identical bodies and different names.
No, outputs are generally smaller than inputs, and for large programs they are much smaller than the inputs. The .o files contain all the data necessary to put the program together and after linking most of that information is no longer needed. I just built a small program I happen to have locally and the constituent .o files add up to 580KiB but the linked program is 208KiB. Another small program has 192KiB of linker inputs and 124KiB of output. This effect is larger for large programs.
>> Why is time spent on System is so much higher than the other one even though that one takes longer?
> Good question but I don't know the answer.
Don’t know if that’s the final answer here (didn’t actually investigate), but system is the time spent in syscalls, and locks are usually provided by the OS, aka syscalls (though there might be userland components to avoid the syscall e.g. futex).
Thus the need for synchronisation of threaded program generally leads to higher system time. Though here mold is clearly a lot more efficient as well (lower user time).
> Good question but I don't know the answer.
Don’t know if that’s the final answer here (didn’t actually investigate), but system is the time spent in syscalls, and locks are usually provided by the OS, aka syscalls (though there might be userland components to avoid the syscall e.g. futex).
Thus the need for synchronisation of threaded program generally leads to higher system time. Though here mold is clearly a lot more efficient as well (lower user time).
How do I learn about what’s being discussed here? Why do I need a linker?
John R. Levine’s book Linkers and Loaders remains the only comprehensive coverage on the topic. You can get the manuscript proofs for free on the author’s website:
https://www.iecc.com/linker/
The Solaris Linkers and Libraries docs are very good and mostly relevant to Linux, with fewer distractions about Windows:
https://docs.oracle.com/cd/E37838_01/html/E36783/index.html
https://www.iecc.com/linker/
The Solaris Linkers and Libraries docs are very good and mostly relevant to Linux, with fewer distractions about Windows:
https://docs.oracle.com/cd/E37838_01/html/E36783/index.html
Here is a video presentation from Apple about the topic: https://developer.apple.com/videos/play/wwdc2022/110362
It has an introduction section on what is a linker and why you need one. The gist is, linker is the piece of software that puts together different source file, be it at compile time or runtime. So, the faster is your linker the quicker the app compiles and opens.
It has an introduction section on what is a linker and why you need one. The gist is, linker is the piece of software that puts together different source file, be it at compile time or runtime. So, the faster is your linker the quicker the app compiles and opens.
Individual compilation units (ex: .o files built from .c files) hold references (ex: from .h files) to other compilation units. In order to assemble a full executable, a linker takes all these individual units and links them together.
You may want to read this section of the README.
https://github.com/rui314/mold#why-does-the-speed-of-linking...
https://github.com/rui314/mold#why-does-the-speed-of-linking...
To assemble executable code from separately compiled files. Or to tell your executable code from where other executable code it needs can be loaded at runtime.
[deleted]
Mold is absolutely invaluable for building projects on Linux with Rust, so I'm super excited that work is being done to make it work with Mac.
Great achievement, does it support all deployment targets, build configurations for Swift and Objective-C?
How does it compare to zld?
zld is probably no longer relevant as of Xcode 14 as ld64 now performs similarly to it. Mold is much faster than both.