As the author of the article, I'll say that my intent was to share the good time I had writing a Lua interpreter in Go. Other folks in the thread are right: more than one thing in this world can rock. :)
Yup, as others have mentioned, it's for zb. I originally outlined some background on design goals for this blog post, but the outline for that section was as big as the outline for the rest of the post, so I cut it. I'll probably write that one up as another blog post in the near future. :)
Thanks! And go for it, it's a good learning experience! It's a really interesting problem domain and there's a lot of different directions you can take it.
Heh, I think I need to add something to the README. I've been pronouncing it as "zeeb" in my head as in the first syllable Zebesian Space Pirate from Metroid, but TIL that that's canonically "Zay-bay-zee-uhn" so idk.
According to Build systems à la carte, "it is not possible
to express dynamic dependencies in [Bazel's] user-defined build rules; however some of the pre-defined build rules require dynamic dependencies and the internal build engine can cope with them by using a restarting task scheduler, which is similar to that of Excel but does not use the calc chain." (p6)
IME import-from-derivation and similar in Nix is usually used for importing build configurations from remote repositories. Bazel has a repository rule system that is similar: https://bazel.build/extending/repo
So to answer your question: yes from the strictest possible definition, but in practice, I believe the tradeoffs are acceptable.
That's accurate (unless the config file attempts to read something from the build process, that will trigger a build).
It's a good point about debugging build problems. This is an issue I've experienced in Nix and Bazel as well. I'm not convinced that I have a great solution yet, but at least for my own debugging while using the system, I've included a `zb derivation env` command which spits out a .env file that matches the environment the builder runs under. I'd like to extend that to pop open a shell.
MinGW is used to build Lua using cgo. I'd like to remove that part, see https://github.com/256lights/zb/issues/28 I haven't started the userspace for Windows yet (https://github.com/256lights/zb/issues/6), but I suspect that it will be more "download the Visual C++ compiler binary from this URL" than the Linux source bootstrap.
Yeah, I'm happy with live-bootstrap, too! I tried emulating Guix's bootstrap, but it depended a little too much on Scheme for me to use as-is. live-bootstrap has mostly worked out-of-the-box, which was a great validation test for this approach.
- Nixpkgs is not content-addressed (yet). I made a conscious decision to only support content-addressed derivations in zb to simplify the build model and provide easier-to-understand guarantees to users. As a result, the store paths are different (/zb/store instead of /nix/store). Which leads to...
- Nix store objects have no notion of cross-store references. I am not sure how many assumptions are made on this in the codebases, but it seems gnarly in general. (e.g. how would GC work, how do you download the closure of a cross-store object, etc.)
- In order to obtain Nixpkgs derivations, you need to run a Nix evaluator, which means you still need Nix installed. I'm not sure of a way around this, and seems like it would be a hassle for users.
I have experienced the same friction in build infra for Nix. My hope is that by reusing the binary cache layer and introducing a JSON-RPC-based public API (already checked in, but needs to be documented and cleaned up) for running builds that the infrastructure ecosystem will be easier.
The last commit using that approach was https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9... if you want to try it out. And actually, I haven't touched the Lua frontend much since I swapped out the backend: the .drv files it writes are the same.
The motivation behind replacing the backend was content-addressibility and Windows support, which have been slow to be adopted in Nix core.
(Hi! I recognize your name from Bazel mailing lists but I forget whether we've talked before.)
I'm mostly contrasting from Nix, which has difficulty with poisoning cache when faced with non-deterministic build steps when using input-addressing (the default mode). If zb encounters a build target with multiple cached outputs for the same inputs, it rebuilds and then relies on content-addressing to obtain build outputs for subsequent steps if possible. (I have an open issue for marking a target as intentionally non-deterministic and always triggering this re-run behavior: https://github.com/256lights/zb/issues/33)
I'll admit I haven't done my research into how Bazel handles non-determinism, especially nowadays, so I can't remark there. I know from my Google days that even writing genrules you had to be careful about introducing non-determinism, but I forget how that failure mode plays out. If you have a good link (or don't mind giving a quick summary), I'd love to read up.
I have considered Starlark, and still might end up using it. The critical feature I wanted to bolt in from Nix was having strings carrying dependency information (see https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb... for a description of the feature). In my prototyping, this was pretty simple to bolt on to Lua, but I'm not sure how disruptive that would be to Starlark. Nix configurations tend to be a bit more complex than Bazel ones, so having a more full-featured language felt more appropriate. Still exploring the design space!
Yup, remote building and caching is on my radar. I expect it will work much in the same way Nix does now, although I'm being a bit more deliberate in creating an RPC layer so build coordinators and other such tools are more straightforward to build.
The string tweak is transparent to users broadly speaking. IME with Nix this thing works the way people expect (i.e if you use a dependency variable in your build target, it adds a dependency).