FYI, Zig is in the process of dropping dependence on lld on Linux and Windows; macOS uses Zig's own linker since roughly 2021 as far as I recall. Tracking issue for reference: https://github.com/ziglang/zig/issues/8726
Hi, kubkon here, the author of Zig's MachO linker. I just wanted to explain our reasoning here a little bit. As you have hopefully noticed in the release notes, there are 5 core team members that are paid either full- or part-time for their work on Zig. That's not an awful lot, and as you can see in our latest roadmap for 0.12 release, we have quite a few challenging components to develop including functioning linkers for the most common platforms, and not only macOS/Apple. This makes it really tricky to justify spending a considerable amount of time trying to support an OS that was officially dropped by a vendor such as macOS/x86, or watchOS/arm32 (FYI, as a fun fact, arm32 is considered private as opposed to public by Apple according to comments and conditional includes in Apple's libc headers). That said, after 1.0/stable release of Zig, I would love to spend more time into adding backwards support for old Apple platforms in the MachO linker and Zig as a whole.
Great question. With the presented approach in Zig, since you work on the binary directly, after you finish your hot-code reloading session, you can still run the generated binary from disk (and debug it or whatnot) as all writes to memory were also committed to the underlying file on disk. There is therefore no need to recompile your program to an executable from a dynamic library as I guess would be the case for approach taken by Nim/V.
The presented approach might also be more resource efficient as it is writing directly to program's memory rather than unloading and reloading a dynamic library, but this is very much a guess and I would need to do some benchmarking to get a better feel for it.
In general though, this approach is possible in Zig since first of all, we have our own linker for each target file format (ELF, MachO, COFF-coming-soon-tm), secondly, the compiler generates the executable file directly, and thirdly, incremental updates are super granular in order to minimise writes to the file as much as possible.
Exactly, and the way I see it, this is the only valid use case for hot-code reloading in the first place: app in-development. I'll try augmenting my linker to be able to bake in the entitlements into the Zig compiler and this hopefully will remove the requirement for elevating privs via "sudo". Thanks for your comments though - it's been very enlightening :-)
Wait, but what about debuggers then? Plus hot-code reloading should only ever be used for quick development cycles when prototyping your app in debug mode, so very much what a debugger is used for, right? Additionally, I actually based the implementation of this PoC on lldb's debugserver for macOS.
Yep, you are absolutely right, and hence why this is a proof-of-concept after all and a start really. With the proper tweaks to the linker though I am sure it would be possible orchestrate updates in such a way as to not clobber any existing global state or cause weird UB (by overwriting the code a thread is currently executing, etc.). Either way, thanks so much for the feedback!
Hey, author of the article here. Thanks for the suggestion! I actually didn't know about the MAP_JIT flag to mmap before and will defo consider it. As to elevating your privs - this is just a temp solution until I work out how to add this entitlement https://developer.apple.com/documentation/bundleresources/en... to the Zig compiler. I wrote the default Zig's MachO linker from scratch and it can embed the adhoc code signatures no probs, but haven't worked out baking the entitlements in yet.
Hey smeenai! I am the author of the MachO linker in Zig. First of, I am closely following your work in LLVM and I am really delighted that someone of your aptitude took over the backend - I definitely intend to take it for a spin soon!
As far as cross-linking is concerned, we do the same thing we do for macOS libc headers - we ship the definitions. You can think of it as effectively shipping a preprocessed version of libSystem.tbd with Zig. This is still early days though, so currently as a workaround for not having a functional yaml parser (so that `zig ld` can link against tbds), every unresolved proxy symbol is simply assumed to come from libSystem. This is of course not ideal since then unresolved symbols are flagged only at runtime rather than at link time.
Anyhow, shipping preprocessed libSystem.tbd makes it possible to cross-compile valid PIE binaries (which is a strict requirement on Apple Silicon anyway).
If you wanna discuss it more, feel free to DM me in Zig's Discord - I'm always game to discuss linkers!