Zig Is Self-Hosted Now, What's Next?(kristoff.it)
kristoff.it
Zig Is Self-Hosted Now, What's Next?
https://kristoff.it/blog/zig-self-hosted-now-what/
158 comments
I started trying to learn Zig as my first non-GC language but switched over to C because I had trouble finding learning resources for Zig, while there are plenty of books and C code written to ISO standards to learn from. It felt like a language for people already proficient in C/C++ and systems programming.. but props to you for proving that wrong!
Yeah that's a good point. I was a little surprised at the spartan quality of documentation. But I'm also comfortable grepping around the compiler/standard library source code to see patterns. And most importantly, I ask a ton of dumb questions on the Zig Programming Language's #zig-help channel. :)
https://discord.gg/9Cp4uhDu
https://discord.gg/9Cp4uhDu
the source code was very well laid put from what i remember, and tests were particularly helpful for understanding how to use library code. i just didn’t feel like i quite had the knowledge to architect low level code in general, you must have picked up on that naturally
Of all the reasons I might imagine someone feeling like C is easier to pick up than Zig, "hardware-software knowledge to know how to architect low level code in general" does not make a ton of sense to me. In what way do you feel this was easier in C?
I think their point is that you need to learn that stuff and with C you can learn it "accidentally" through C learning materials. Zig doesn't have yet enough learning materials to "accidentally" teach you systems programming.
The solution would be to have learning materials that teach you systems programming in a way that is not tied to a specific language, but I don't know of any example of that, and I guess the same applies to all the people that I had this discussion with in the past.
The solution would be to have learning materials that teach you systems programming in a way that is not tied to a specific language, but I don't know of any example of that, and I guess the same applies to all the people that I had this discussion with in the past.
when i have an idea for a program i want to build, i struggle to express it in terms of structs, pointers, and C/zig’s language features. but with C, i can often find similar programs and find good examples of how to do what i want.
for example right now i’m working on an SDL/opengl app with an immediate mode gui, and there’s enough code in the wild for me to pour over to understand how and why they structure their programs
but when i started with zig i would often get stuck, unable to really understand the c code i was interfacing with and struggling with the language basics a bit
for example right now i’m working on an SDL/opengl app with an immediate mode gui, and there’s enough code in the wild for me to pour over to understand how and why they structure their programs
but when i started with zig i would often get stuck, unable to really understand the c code i was interfacing with and struggling with the language basics a bit
On knowing how to structure low level code, I highly recommend reading Casey Muratori's blog post on Semantic Compression: https://caseymuratori.com/blog_0015
And, more generally, watching some of his (free) handmade hero video series: https://handmadehero.org/
And, more generally, watching some of his (free) handmade hero video series: https://handmadehero.org/
Wow, that was incredibly helpful. I could read writeups of straightforward code all day. I also love how his ethos extends to the design of the website as well. Thanks so much for sharing.
Yeah, that makes sense. We'll try to improve the situation, we just need to get a bit further ahead with the language first.
Zig is not stable enough (in terms of language design) to be in the same spot as C wrt books and sample code. Even the linked blogpost has some examples of breaking syntax changes around for-loops, which is pretty basic stuff.
If anything, I'd say it's extremely usable despite being very much a work-in-progress, largely due to their focus on tooling (and not just the language in and of itself) - it's also why I think it'll be an amazing ecosystem to work in once it matures.
If anything, I'd say it's extremely usable despite being very much a work-in-progress, largely due to their focus on tooling (and not just the language in and of itself) - it's also why I think it'll be an amazing ecosystem to work in once it matures.
Being a better C is not hard to achieve as goal, JOVIAL, NEWP and PL/I did it one decade before C was born, and Modula-2 did it while C was mostly a Bell Labs toy, back in 1978.
The problem is getting the wider market to pick better C languages.
The problem is getting the wider market to pick better C languages.
I agree! I think the Zig community is doing a better job marketing than most better-c languages (which saddens me in some respects because I like D too, but sometimes you go with what meets most criteria and seems like it will succeed with adoption!).
> I haven't noticed miscompilation yet
I found it shockingly easy to get a miscompilation that it really soured my interest in it. It’s good to hear that in practice that appears to be less of a concern.
I found it shockingly easy to get a miscompilation that it really soured my interest in it. It’s good to hear that in practice that appears to be less of a concern.
Anecdotally again, but I've been coding in Zig since 2020 and have hit I think 2-3 compiler bugs in all that time?
The first was fixed within 24 hours, in fact just before I reported it. The others had clear TODO error messages in the panic, and there were easy enough workarounds.
The first was fixed within 24 hours, in fact just before I reported it. The others had clear TODO error messages in the panic, and there were easy enough workarounds.
Correctness please.
Zig is a way too simple language to justify all the really easy to hit bugs.
I'd give everything for a zig compiler that's 10 times as slow but actually compiles my code.
Also testing infrastructure. Zig really needs fuzzing and property testing tooling build-in. Hitting stage 2 with arbitrary autogenerated zig code and IR, would have really helped detect regressions for example.
I'd give everything for a zig compiler that's 10 times as slow but actually compiles my code.
Also testing infrastructure. Zig really needs fuzzing and property testing tooling build-in. Hitting stage 2 with arbitrary autogenerated zig code and IR, would have really helped detect regressions for example.
If bugs are your concern, then this upcoming release is good news for you. Total number of open bug reports is 1,124. However if we remove bugs labeled "stage1" - that is - bugs that are no longer applicable because this release changes the default compiler to the self-hosted one - then the number drops to 514. It would be a fair characterization to say that this release cuts the number of known bugs in half. We have those stage1 bug reports still open because I mandated that every one must have test coverage before being closed, even though they are already fixed in master branch.
That said, I would humbly ask for you to trust my leadership on finding the balance on how much effort to spend on bug fixing versus how much to spend on progress. I think you underestimate how much is going to change before Zig tags 1.0. There are still a few experimental and exploratory changes planned, and when software specifications change, bugs are inevitably introduced. Spending the time to iron out 100% of the bugs found from fuzz testing, on a feature that ends up replaced with a different one that also needs to get debugged, is a waste of time, and a disservice to the wider user base of the project. We're better off finding the "sweet spot" for a swift journey, arriving at our destination early, and then busting out the hardcore QA.
I see https://github.com/ziglang/zig/issues/13211 mentioned elsewhere in this thread. I want to point out that getting the C ABI correct when using LLVM is a notoriously perilous process, as any other compiler dev reading HN today can attest, and this regression which lasted for a total of 6 days in master branch without ever gracing a tagged release, is a symptom of the fact that we actually addressed this C ABI problem at its core, added more test coverage, added an abstraction in the LLVM code generation, and fixed the C ABI for several targets.
And finally, I agree that fuzz testing should be integrated into Zig's unit testing system. That would be a nice enhancement.
That said, I would humbly ask for you to trust my leadership on finding the balance on how much effort to spend on bug fixing versus how much to spend on progress. I think you underestimate how much is going to change before Zig tags 1.0. There are still a few experimental and exploratory changes planned, and when software specifications change, bugs are inevitably introduced. Spending the time to iron out 100% of the bugs found from fuzz testing, on a feature that ends up replaced with a different one that also needs to get debugged, is a waste of time, and a disservice to the wider user base of the project. We're better off finding the "sweet spot" for a swift journey, arriving at our destination early, and then busting out the hardcore QA.
I see https://github.com/ziglang/zig/issues/13211 mentioned elsewhere in this thread. I want to point out that getting the C ABI correct when using LLVM is a notoriously perilous process, as any other compiler dev reading HN today can attest, and this regression which lasted for a total of 6 days in master branch without ever gracing a tagged release, is a symptom of the fact that we actually addressed this C ABI problem at its core, added more test coverage, added an abstraction in the LLVM code generation, and fixed the C ABI for several targets.
And finally, I agree that fuzz testing should be integrated into Zig's unit testing system. That would be a nice enhancement.
Does/will Zig support the Actually Portable Executable format?
https://justine.lol/ape.html
https://justine.lol/ape.html
This.
Also have this very same question / use “Cosmopolitan” LibC with Zig.
- Zig Team?
Thanks in advance.
- Zig Team?
Thanks in advance.
I believe where this stands is that if someone were to contribute and maintain this, then it could be part of the Zig compiler, however it is not something the Zig team themselves are planning on doing. Maybe give it a shot? I personally find the project interesting and fun but I'm not convinced that it's a good solution.
The bug that's currently blocking me from any work using master is https://github.com/ziglang/zig/issues/12325
That's not really a bug related to unfinished features that might get replaced anyways, but simply one involving recursive structs with pointers between, i.e. linked lists.
Not detecting issues like these feels less like a "these parts are unfinished hence here be dragons" thing and more like a fundamental culture problem.
I'm fine with cutting myself every now and then with a sharp tool and cutting edge technology, but that can also be done while having a ever growing and mature core that promises at least some stability. The only alternative seems to be to pull a Hickey and work on it for two years without releasing anything.
That's not really a bug related to unfinished features that might get replaced anyways, but simply one involving recursive structs with pointers between, i.e. linked lists.
Not detecting issues like these feels less like a "these parts are unfinished hence here be dragons" thing and more like a fundamental culture problem.
I'm fine with cutting myself every now and then with a sharp tool and cutting edge technology, but that can also be done while having a ever growing and mature core that promises at least some stability. The only alternative seems to be to pull a Hickey and work on it for two years without releasing anything.
You're using the master branch of a pre-1.0 tool, and you expect it to be stable without regressions? I feel like your expectations are too high.
he just want to force zig devs to tackle his problems first.
Using phrases like "fundamental culture problem" or "at least some stability"
Using phrases like "fundamental culture problem" or "at least some stability"
Frankly I don't care if the problem gets fixed or not.
I've already started a rewrite in Rust, and I don't plan on revisiting the Zig version until Zig has hit 1.0.
Coming from a Clojure background I generally agree with Hoare's "One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors.", and in that sense Zig should easily be more correct than Rust which is much much much more complex.
The fact that that hasn't happened, with Rust being much more production ready for most of it's life, despite some major backwards compatibility breaking changes, leaves two possible answers for me:
1. The Rust developers are magically better programmers, which I believe to be false.
2. The rust community has a general culture for correctness and QA which the Zig community lacks.
Coming from a Clojure background I generally agree with Hoare's "One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors.", and in that sense Zig should easily be more correct than Rust which is much much much more complex.
The fact that that hasn't happened, with Rust being much more production ready for most of it's life, despite some major backwards compatibility breaking changes, leaves two possible answers for me:
1. The Rust developers are magically better programmers, which I believe to be false.
2. The rust community has a general culture for correctness and QA which the Zig community lacks.
I'd be interested in some constructive feedback on your Zig experience. There's nothing I can do with the claim that Zig users have a culture problem, especially since the "the zig community" does not work on the compiler - it's just me and a handful of other people - and I'm not really able to evaluate the claim about Rust being in a more "production ready" state at a comparable point in its life. To be honest, these two points feel like nothing more than personal insults.
So, putting those aside, if you have any actionable experiences to share, such as other problems you ran into, what went well, what went poorly, what workarounds you tried, other experiences you had that could help inform the direction of the project, then these things would help make Zig shine when you revisit 1.0.
So, putting those aside, if you have any actionable experiences to share, such as other problems you ran into, what went well, what went poorly, what workarounds you tried, other experiences you had that could help inform the direction of the project, then these things would help make Zig shine when you revisit 1.0.
puffoflogic(1)
Am I missing something here?
You are comparing a language which hit 1.0 and had seven years of additional refinement, to a language which is still at least 2-3 years away from 1.0 and spent most if not all its time in the past 12 months on self hosting compiler?
You are comparing a language which hit 1.0 and had seven years of additional refinement, to a language which is still at least 2-3 years away from 1.0 and spent most if not all its time in the past 12 months on self hosting compiler?
> Am I missing something here?
Yes. I'm comparing the stability of very early Rust e.g. 0.x, to the stability of Zig now. Not Rust 1.6 with Zig 0.x.
Yes. I'm comparing the stability of very early Rust e.g. 0.x, to the stability of Zig now. Not Rust 1.6 with Zig 0.x.
> and then busting out the hardcore QA
This thinking seems... backward (or at least overall counterproductive/like more work) to me. Do you not have a test suite? At least, unit tests for core language features? It seems (with all due respect) a bit irresponsible not to have this, it would at least prevent most regressions in tested code for example (such as breaking "for" loops, as someone else mentioned)
This thinking seems... backward (or at least overall counterproductive/like more work) to me. Do you not have a test suite? At least, unit tests for core language features? It seems (with all due respect) a bit irresponsible not to have this, it would at least prevent most regressions in tested code for example (such as breaking "for" loops, as someone else mentioned)
Zig has thousands of tests, but the problem space of a programming language is pretty big, making fuzzing and other similar techniques worth the effort in the long run. The discussion wasn't about the existence of basic unit tests.
https://github.com/ziglang/zig/tree/master/test
https://github.com/ziglang/zig/tree/master/test
I also possibly misinterpreted someone else's comment about "for" breaking, assuming it was a regression instead of an intended syntax change which might have been covered by units. My bad
Yep, I've also run into a myriad of gotchas while building things or poking around in written code. Just last week I was trying to build someone's year old repo that blew up on nightly, yet compiled on 0.9 just fine.
But - it's plastered all over the entire project that it is a work in progress. This post itself is talking about how for loops are getting broken. It's an ambitious project with a talented and ambitious team, and their vision and execution so far has been awesome. I'm not using Zig for anything resembling production code at the moment - because it's going to break. But when it stabilizes - there is a lot of potential.
But - it's plastered all over the entire project that it is a work in progress. This post itself is talking about how for loops are getting broken. It's an ambitious project with a talented and ambitious team, and their vision and execution so far has been awesome. I'm not using Zig for anything resembling production code at the moment - because it's going to break. But when it stabilizes - there is a lot of potential.
> Just last week I was trying to build someone's year old repo that blew up on nightly, yet compiled on 0.9 just fine.
Isn’t that the point of nightly
Isn’t that the point of nightly
Looks like your complaint was a known tradeoff.
> The self-hosted compiler brings many advantages, but it did cost us a significant amount of effort and time. While Zig is still going up in popularity and starting to make a tiny dent in the industry, people that have been following along for long enough will know that this work has reduced our momentum in the last two years.
> Bug fixes in the compiler have been often put on hold since fixing the bootstrap compiler was ultimately useless, and accepted feature proposals have been piling up because it would have required implementing everything twice.
> The self-hosted compiler brings many advantages, but it did cost us a significant amount of effort and time. While Zig is still going up in popularity and starting to make a tiny dent in the industry, people that have been following along for long enough will know that this work has reduced our momentum in the last two years.
> Bug fixes in the compiler have been often put on hold since fixing the bootstrap compiler was ultimately useless, and accepted feature proposals have been piling up because it would have required implementing everything twice.
Your quoted sentence talks about bugs in the old C++ compiler, I'm talking about bugs in both compilers.
Pretty much all of the talks and blog posts on the `in-Zig` compiler are about better performance, lower memory footprint, e.t.c.
I would much rather have heard a talk about how they improved correctness through automatic testing, formal methods, simple language design e.t.c.
A language that (does or doesn't) compile your code in 0.1 seconds to a broken binary is worse than piping your code into dev/null.
Pretty much all of the talks and blog posts on the `in-Zig` compiler are about better performance, lower memory footprint, e.t.c.
I would much rather have heard a talk about how they improved correctness through automatic testing, formal methods, simple language design e.t.c.
A language that (does or doesn't) compile your code in 0.1 seconds to a broken binary is worse than piping your code into dev/null.
I see the acknowledgements of the bugs in the old compiler, but what bugs in the new compiler are you talking about? Also, a recurring theme in zig has been the advantages of their simple language design.
See my reply in another comment tree https://news.ycombinator.com/item?id=33333827
[deleted]
I'm not so familiar with Zig, are you saying that the compiler fails to compile correct (i.e. according to spec) code?
We're using the C++ compiler rather than the self-hosted compiler because the self-hosted compiler does not yet support async. Yes, it sometimes fails to compile correct code (by crashing) or compiles correct code to incorrect code.
puffoflogic(1)
Yeah, theres a surprising amount of stuff that's buggy, think `if` or `recursive structs with pointers` level stuff.
Can you substantiate this further? I don't see how basic stuff like that could be subtly buggy, the Zig devs are very qualified IMO.
We do have a good number of known miscompilations, I can substantiate for the parent poster with a simple link: https://github.com/ziglang/zig/labels/miscompilation
It's also true that the self-hosted compiler has caused new bugs to pop up, which in turn made us write new behavior tests. To me that's to be expected with every project aimed at rewriting an existing system.
It's also true that proper testing infrastructure is critical, and we do have it, although the requirement of more than 8gb of ram has not played in our favor in the past. We plan to revamp our CI systems once we are done with this release.
That said, it's a v0 language. If you choose to use it, you know you're coming with us on a ride. The choice of how much to prioritize correctness vs progress is influenced by many things that are going on in the project and there is no one clear strategy that trumps all others.
Quoting j-pb from another comment:
> A language that (does or doesn't) compile your code in 0.1 seconds to a broken binary is worse than piping your code into dev/null.
Eh, yes, but also no. Obviously a compiler needs to work correctly to be useful, but our priority right now is to move the project forward using our best judgment, not ensure that it has maximal utility to early adopters. I do agree with the sentiment that the compiler needs to be reliable, it's just that Zig is v0 for a reason.
In concrete terms I think that once we're done with an initial adjustment phase, you will find that the compiler will have a steady increase in stability over time. That's how it was with the bootstrap compiler before we started the self-hosted work.
Also, quoting dist1 from up the comment tree:
> I'm not so familiar with Zig, are you saying that the compiler fails to compile correct (i.e. according to spec) code?
Keeping in mind everything that I just wrote, note that Zig doesn't have a spec yet. Sometimes, some issues in the compiler are the result of discovering that a design choice needs to be amended.
Recent example of that: https://github.com/ziglang/zig/issues/12251
It's also true that the self-hosted compiler has caused new bugs to pop up, which in turn made us write new behavior tests. To me that's to be expected with every project aimed at rewriting an existing system.
It's also true that proper testing infrastructure is critical, and we do have it, although the requirement of more than 8gb of ram has not played in our favor in the past. We plan to revamp our CI systems once we are done with this release.
That said, it's a v0 language. If you choose to use it, you know you're coming with us on a ride. The choice of how much to prioritize correctness vs progress is influenced by many things that are going on in the project and there is no one clear strategy that trumps all others.
Quoting j-pb from another comment:
> A language that (does or doesn't) compile your code in 0.1 seconds to a broken binary is worse than piping your code into dev/null.
Eh, yes, but also no. Obviously a compiler needs to work correctly to be useful, but our priority right now is to move the project forward using our best judgment, not ensure that it has maximal utility to early adopters. I do agree with the sentiment that the compiler needs to be reliable, it's just that Zig is v0 for a reason.
In concrete terms I think that once we're done with an initial adjustment phase, you will find that the compiler will have a steady increase in stability over time. That's how it was with the bootstrap compiler before we started the self-hosted work.
Also, quoting dist1 from up the comment tree:
> I'm not so familiar with Zig, are you saying that the compiler fails to compile correct (i.e. according to spec) code?
Keeping in mind everything that I just wrote, note that Zig doesn't have a spec yet. Sometimes, some issues in the compiler are the result of discovering that a design choice needs to be amended.
Recent example of that: https://github.com/ziglang/zig/issues/12251
> Eh, yes, but also no. Obviously a compiler needs to work correctly to be useful, but our priority right now is to move the project forward using our best judgment, not ensure that it has maximal utility to early adopters.
I feel like I’m a potential early adopter and the amount of miscompilation I ran into playing around with it really made me lose interest. It seems like the priorities are set somewhere else entirely.
I feel like I’m a potential early adopter and the amount of miscompilation I ran into playing around with it really made me lose interest. It seems like the priorities are set somewhere else entirely.
Going by your own comment, you seem to not be an "early enough" adopter. That's totally fair, maybe Zig will be more interesting for you once it stabilizes more in the future.
I don’t believe this to be true.
Zig is 6 years old now going by Wikipedia. In comparison I use Rust in an more extensive way than Zig for almost 20 years now. I for sure wrote more Rust in 2012 when it was ~2 years? old than Zig last year, yet I don’t remember miscompilations on that level. The same is true for my recollection of using very early D which however at this point was in an earlier life of mine and very fuzzy.
Zig is 6 years old now going by Wikipedia. In comparison I use Rust in an more extensive way than Zig for almost 20 years now. I for sure wrote more Rust in 2012 when it was ~2 years? old than Zig last year, yet I don’t remember miscompilations on that level. The same is true for my recollection of using very early D which however at this point was in an earlier life of mine and very fuzzy.
I'm not sure I understand your point. Zig is clearly too unstable for your liking, by your own admission.
If your point is that Rust was more stable (and why not, even an overall better project) at the same age, then I'm happy to immediately concede the point. But that doesn't really have any influence on the main question, does it?
If your point is that Rust was more stable (and why not, even an overall better project) at the same age, then I'm happy to immediately concede the point. But that doesn't really have any influence on the main question, does it?
The argument that you were making (unless I misunderstood you) is that there are more important issues for the language than correctness or appealing to early adopters to which I made the remark that this attitude is steering me away from the language.
Zig is in a weird spot because it has a similar momentum behind it than some other languages I was exited about, it has even more hype (and is very hype driven with streams and everything) yet it has from my experience with it much more stability issues. And I assume some of this comes from a relay complex compile time system that is harder to implement rigorously than some other things.
I also have to say that I find these responses unnecessarily combative.
Zig is in a weird spot because it has a similar momentum behind it than some other languages I was exited about, it has even more hype (and is very hype driven with streams and everything) yet it has from my experience with it much more stability issues. And I assume some of this comes from a relay complex compile time system that is harder to implement rigorously than some other things.
I also have to say that I find these responses unnecessarily combative.
> The argument that you were making (unless I misunderstood you) is that there are more important issues for the language than correctness or appealing to early adopters to which I made the remark that this attitude is steering me away from the language.
Yes, and I still stand by my assertion. Next to you in the "early adopter spectrum" are people who can put up with more breakage on one side, and people who cannot put up with the current level on the other side. At any point in time the project will be somewhere along that spectrum and people will be on either side of that line.
Similarly, different things going on in the project will move the line in either direction. Switching to a new compiler implementation is unsurprisingly a source of bugs, if you want to use Zig now, you will have to accept the current state of things.
> it has even more hype (and is very hype driven with streams and everything)
Hype driven... you mean Andrew's 100 viewer streams? Or my 30 viewer streams? What hype-driving streams are you referring to. My YouTube channel with 4k subs? (hit that milestone today btw)
> I also have to say that I find these responses unnecessarily combative.
I really don't know what to tell you. If Zig is too broken for your liking, don't use it. If other projects at the same level of age/popularity/whatever had a level of breakage that you found tolerable, then great, and if Zig doesn't meet that standard, then it means, by definition, that it's too early for you to use it.
If you want to make a different point, then state it clearly. From the early adoption perspective, I don't see any argument to be discussed here.
Yes, and I still stand by my assertion. Next to you in the "early adopter spectrum" are people who can put up with more breakage on one side, and people who cannot put up with the current level on the other side. At any point in time the project will be somewhere along that spectrum and people will be on either side of that line.
Similarly, different things going on in the project will move the line in either direction. Switching to a new compiler implementation is unsurprisingly a source of bugs, if you want to use Zig now, you will have to accept the current state of things.
> it has even more hype (and is very hype driven with streams and everything)
Hype driven... you mean Andrew's 100 viewer streams? Or my 30 viewer streams? What hype-driving streams are you referring to. My YouTube channel with 4k subs? (hit that milestone today btw)
> I also have to say that I find these responses unnecessarily combative.
I really don't know what to tell you. If Zig is too broken for your liking, don't use it. If other projects at the same level of age/popularity/whatever had a level of breakage that you found tolerable, then great, and if Zig doesn't meet that standard, then it means, by definition, that it's too early for you to use it.
If you want to make a different point, then state it clearly. From the early adoption perspective, I don't see any argument to be discussed here.
> Hype driven... you mean Andrew's 100 viewer streams? Or my 30 viewer streams?
More like release notes with 1,000 upvotes on Reddit:
https://www.reddit.com/r/programming/comments/rl87pr/zig_pro...
More like release notes with 1,000 upvotes on Reddit:
https://www.reddit.com/r/programming/comments/rl87pr/zig_pro...
[deleted]
2022 - 2010 ==> 12 years. :)
Here's a pretty substantial bug from just a week ago: https://github.com/ziglang/zig/issues/13211
"stage1 is expected to be broken in many ways and does not handle non-x86_64 C ABI at all."
so its self-hosted for x86_64 only so far, which is fine if communicated
so its self-hosted for x86_64 only so far, which is fine if communicated
stage1 is the non-self-hosted compiler (the one written in C++)
eh. if it isn't fast, then there's no point using it either. the c++ i have to compile at work takes 16 minutes to test some crap. that is prohibitively slow.
I watched the linked talk by Andrew Kelly - super interesting stuff. I've played a bit with data oriented design in the context of implementing an ECS, but some of the topics he discusses take it to another level entirely, and it's very cool to see it applied, with measurable results, in a large serious project like a compiler.
I have a lot of admiration for Zig, as I think it takes a very interesting approach to being a "better C", and the no-nonsense approach, ruthless simplicity, and thin abstraction over the hardware fall very much in line with my values as a programmer.
As a product manager, a few thoughts ran through my head when when I was watching this: with where Zig is as a language, is this kind of focus on extreme memory optimization actually the best use of his time?
There's an argument that it might be: maybe this is an exercise in dogfooding Zig as the kind of language which is suitable for data oriented design. Or it could be Kelly has fallen victim to the temptation many of us do, in following the most fascinating path rather than the one that's creating the most long lasting value.
Either way I guess that's a luxury he has the right to indulge as an independently funded auteur PL designer.
I have a lot of admiration for Zig, as I think it takes a very interesting approach to being a "better C", and the no-nonsense approach, ruthless simplicity, and thin abstraction over the hardware fall very much in line with my values as a programmer.
As a product manager, a few thoughts ran through my head when when I was watching this: with where Zig is as a language, is this kind of focus on extreme memory optimization actually the best use of his time?
There's an argument that it might be: maybe this is an exercise in dogfooding Zig as the kind of language which is suitable for data oriented design. Or it could be Kelly has fallen victim to the temptation many of us do, in following the most fascinating path rather than the one that's creating the most long lasting value.
Either way I guess that's a luxury he has the right to indulge as an independently funded auteur PL designer.
The choice of improving the performance of the compiler before working on the package manager was deliberate. With the new compiler you will be able to have plenty of dependencies and compilation times / memory usage will remain reasonable. The same could not be said of the bootstrap compiler.
For us compilation speed is part of the main value offering of Zig.
For us compilation speed is part of the main value offering of Zig.
Is this working? stage2 seems to take similar and large amounts of time to compile a non-async-using program from our project with no caches and with caches after a whitespace change.
The linked post mentioned that the compiler itself does compile faster, but not by much. Memory usage is the greatest impact of stage2.
The speed will come when the native backends are finished. Right now the compile time is mostly llvm. The idea is a native and fast backend for debug (development) builds, and llvm for optimized releases.
The speed will come when the native backends are finished. Right now the compile time is mostly llvm. The idea is a native and fast backend for debug (development) builds, and llvm for optimized releases.
Well, while you are developing a language, in particular a system programming language like zig, having to write your compiler in zig is a great way to learn first hand what your users are complaining about. Normally if you dogfood your own creations, you experience them maybe an hour a day, but the compiler will be the number one tool when working on zig.
As a compiler engineer myself, I dislike dogfooding in the compiler itself. It biases us to design the language for the compiler use case, and that is in fact a very niche use case. I much prefer having a side project that uses the language I'm implementing, instead of using it to implement the compiler itself.
On that note Andrew started a side project recently using Zig, and has already written a bunch of useful notes from the user's POV. I'm certain he'll make that public soon enough.
While a good decision, it also triggers the usual discussion of "your language is not good for X because it was written in Y".
Congrats, fast compile speed is a requirement nowadays, any language that is slow to compile puts you at a risk of not being able to fix important prod issues quick enough, hence why i use Go for my backend, i can't imagine using anything else
Tests, deployment, up to restarting the service in prod, the language should empower me to not loose time
I'm not talking about memory bugs, i'm talking about typo issues, API issues, comment/uncomment, renaming, missing a semicolon (that one should go imo), refactoring etc, a fast compiler allows me to be more productive
Having to recompile due to typos is the main contributor of wasting my time, if it takes 10+ seconds, then the language sucks, it should be instant up to 1 second
Glad that you understand that and you made sure you built the foundation that enables it, that's game changer
Tests, deployment, up to restarting the service in prod, the language should empower me to not loose time
I'm not talking about memory bugs, i'm talking about typo issues, API issues, comment/uncomment, renaming, missing a semicolon (that one should go imo), refactoring etc, a fast compiler allows me to be more productive
Having to recompile due to typos is the main contributor of wasting my time, if it takes 10+ seconds, then the language sucks, it should be instant up to 1 second
Glad that you understand that and you made sure you built the foundation that enables it, that's game changer
Single biggest reason I don't start new rust projects
> We are also working on a special backend, one that produces C source code. Progress on the C backend has recently shot forward (87% and counting) thanks to an amazing recent contribution.
I love this. It's great story for people who want to develop in a newer language while having seamless interoperability into older toolchains and ecosystems (eg. TS/JS). If it reaches production quality, that will be a large selling point for Zig IMHO.
I love this. It's great story for people who want to develop in a newer language while having seamless interoperability into older toolchains and ecosystems (eg. TS/JS). If it reaches production quality, that will be a large selling point for Zig IMHO.
I always enjoy reading about Zig advancements. I haven't developed anything substantial in Zig yet, but I'm very optimistic about the future of the language.
If you're curious to see a large Zig codebase, two significant projects are Bun [1] and TigerBeetle [2].
[1] https://github.com/oven-sh/bun
[2] https://github.com/tigerbeetledb/tigerbeetle
If you're curious to see a large Zig codebase, two significant projects are Bun [1] and TigerBeetle [2].
[1] https://github.com/oven-sh/bun
[2] https://github.com/tigerbeetledb/tigerbeetle
Joran from TigerBeetle here!
Awesome to hear that you're excited about Zig.
Thanks for sharing the link to our repo also—would love to take you on a 1-on-1 tour of the codebase sometime if you'd be up for that!
Awesome to hear that you're excited about Zig.
Thanks for sharing the link to our repo also—would love to take you on a 1-on-1 tour of the codebase sometime if you'd be up for that!
Nice achievement.
Curious about this though:
> building the compiler itself used to require 9.6GB of RAM, while now it takes 2.8GB.
Why is the RAM usage so high? It was only a few years back and we were building C++ compilers on machines with barely 16MB.
Curious about this though:
> building the compiler itself used to require 9.6GB of RAM, while now it takes 2.8GB.
Why is the RAM usage so high? It was only a few years back and we were building C++ compilers on machines with barely 16MB.
> It was only a few years back and we were building C++ compilers on machines with barely 16MB.
"A few years" seems like a serious understatement, but beyond that... compilers in the time period you're talking about weren't really doing optimizations, they were just shoveling assembly out the door. Doing very little work takes very few resources, but the resulting binaries were way slower than they could have been. Any compiler that uses LLVM tends to be slow, in my experience, but it does offer excellent optimizations.
"A few years" seems like a serious understatement, but beyond that... compilers in the time period you're talking about weren't really doing optimizations, they were just shoveling assembly out the door. Doing very little work takes very few resources, but the resulting binaries were way slower than they could have been. Any compiler that uses LLVM tends to be slow, in my experience, but it does offer excellent optimizations.
Just wait until you get old... a "few" years ago, starts becoming a larger number...
This is not the case, optimizing compilers were quite far along by the 16MB era. RISC relied on them, GCC was widely adopted due to optimizations, etc.
The level of optimization available back then is a far cry from what we have today, as far as I've ever heard or seen. Based on looking at some histories of GCC, the EGCS fork in 1997 was responsible for starting to introduce meaningful optimizations into GCC, such as global CSE (common subexpression elimination), and that wasn’t merged back into mainline GCC until years later. LLVM didn’t really hit its stride until late 2000s, and I believe it was used as a research platform to develop a lot of the modern optimizations that exist today.
Do you want to be more specific about the kinds of optimizations that existed in GCC in ~1995? My understanding could be incomplete or wrong, but today's compilers aren’t just twiddling their thumbs and using tons of RAM. They’re doing useful work that simply didn’t happen back then. As with all software, there is surely room for improvement even today, both with the implementations of the compilers and with the optimizations that are being done.
Peephole optimizations (and efficient register allocation) would probably have been very helpful for RISC, and I’m guessing that’s what you’re referring to, but that’s a comparatively simple level of optimization that requires little RAM or heuristic analysis.
Do you want to be more specific about the kinds of optimizations that existed in GCC in ~1995? My understanding could be incomplete or wrong, but today's compilers aren’t just twiddling their thumbs and using tons of RAM. They’re doing useful work that simply didn’t happen back then. As with all software, there is surely room for improvement even today, both with the implementations of the compilers and with the optimizations that are being done.
Peephole optimizations (and efficient register allocation) would probably have been very helpful for RISC, and I’m guessing that’s what you’re referring to, but that’s a comparatively simple level of optimization that requires little RAM or heuristic analysis.
https://ftp.gnu.org/old-gnu/gcc/ has old GCC versions. Eg checking the manpage gcc.1 from the 2.7 tarball there's inlining, loop unrolling, strength reduction, cse, peephole, instruction scheduling, jump threading, and others, probably better described in thebtexinfo doc.
There are opts not covered in the command line options. Eg see this discussion of tail call optimizations already in GCC 1.x: https://groups.google.com/g/gnu.gcc.bug/c/Zzbfyvi2uAM/m/GVDI...
Sure, later compilers did more, but the above-mentioned opts made GCC known as an optimizing compiler. And there were other optimizing compilers too, for C but also other languages.
Fortran compiler histories are interesting, they were doing their own thing on numerical code, SIMD (called vector processors back then), etc. Eg https://www.deepdyve.com/lp/wiley/evaluation-of-fortran-vect...
There are opts not covered in the command line options. Eg see this discussion of tail call optimizations already in GCC 1.x: https://groups.google.com/g/gnu.gcc.bug/c/Zzbfyvi2uAM/m/GVDI...
Sure, later compilers did more, but the above-mentioned opts made GCC known as an optimizing compiler. And there were other optimizing compilers too, for C but also other languages.
Fortran compiler histories are interesting, they were doing their own thing on numerical code, SIMD (called vector processors back then), etc. Eg https://www.deepdyve.com/lp/wiley/evaluation-of-fortran-vect...
Happened to continue my browsing for compiler history and found also this interesting rationale for a new version of the Dhrystone benchmark from 1988 where they say that the previous version was too badly broken by optimizing compilers: https://dl.acm.org/doi/10.1145/47907.47911
The languages are interesting as well (versions for C, Pascal and Ada).
So this places the wide use of optimizing compilers earlier than the 16MB era, 1988 was sub-1 MB era for personal computers, VAX class minis might have had 16 MB (but in multiuser context, compilers probably wouldn't use nearly that much memory)
But of course the later Dhrystone was also later broken by improved compiler optimizations, supporting the notion that compilers did keep improving as well.
Also as another tangent, according to https://en.wikipedia.org/wiki/CMU_Common_Lisp the CMUCL Python compiler (famous for its optimizations) was started in 1985 and had to be fairly good at optimizing already by the mid-90s 16MB era. I found some release notes from 1993 that talk about optimizations: https://trac.common-lisp.net/cmucl/wiki/Release17c (summarized as "Improvements in compiler source-level optimization, inline expansion and instruction scheduling")
The languages are interesting as well (versions for C, Pascal and Ada).
So this places the wide use of optimizing compilers earlier than the 16MB era, 1988 was sub-1 MB era for personal computers, VAX class minis might have had 16 MB (but in multiuser context, compilers probably wouldn't use nearly that much memory)
But of course the later Dhrystone was also later broken by improved compiler optimizations, supporting the notion that compilers did keep improving as well.
Also as another tangent, according to https://en.wikipedia.org/wiki/CMU_Common_Lisp the CMUCL Python compiler (famous for its optimizations) was started in 1985 and had to be fairly good at optimizing already by the mid-90s 16MB era. I found some release notes from 1993 that talk about optimizations: https://trac.common-lisp.net/cmucl/wiki/Release17c (summarized as "Improvements in compiler source-level optimization, inline expansion and instruction scheduling")
I wanted to say "probably because of LLVM", but that wouldn't explain why replacing the frontend would make such a difference (assuming that the 2.8GB number includes the LLVM backend).
Yeah that is interesting ... I guess it's because Zig compilation is single-process, parallel, and threaded, not like the C++ compilation model of separate processes?
I never tried, but I highly doubt either gcc or Clang could be built with 16 MB of RAM, even serially, at any point in the last 10 years. Probably not even 20 years, though I could be wrong
I never tried, but I highly doubt either gcc or Clang could be built with 16 MB of RAM, even serially, at any point in the last 10 years. Probably not even 20 years, though I could be wrong
> I never tried, but I highly doubt either gcc or Clang could be built with 16 MB of RAM, even serially, at any point in the last 10 years. Probably not even 20 years, though I could be wrong
Last week I tried to compile GCC on a machine with 8GB of ram (no swap). Compilation failed, I add to upgrade to 16GB.
I think the last time gcc could build with 16MB was last century.
Last week I tried to compile GCC on a machine with 8GB of ram (no swap). Compilation failed, I add to upgrade to 16GB.
I think the last time gcc could build with 16MB was last century.
Hey, back in the day, I used a Pascal compiler on CP/M that worked with 64K.
Compilers like that would go directly from text -> machine instructions, step by step (or to very close approximation) because that is all you had the time and memory for.
Modern compilers lex then parse into tree structures and then do lots of mutations and traversals of that tree to optimize in different ways for different architectures, and more code is included at once to make more optimizations possible and on and on and on.
Modern compilers lex then parse into tree structures and then do lots of mutations and traversals of that tree to optimize in different ways for different architectures, and more code is included at once to make more optimizations possible and on and on and on.
Modern compilers like in 1970s?
https://en.wikipedia.org/wiki/PL/8
Or like the 1980s?
https://en.wikipedia.org/wiki/Amsterdam_Compiler_Kit
https://en.wikipedia.org/wiki/PL/8
Or like the 1980s?
https://en.wikipedia.org/wiki/Amsterdam_Compiler_Kit
I don't know about Zig specifically, but things using LLVM tend to be absurd memory hogs. I remember having to disable parallel compilation for clang to prevent OOM issues.
Yeah, 9.6GB of RAM seems like a really small number when you're talking about LLVM.
I'd guess at least part of it has to do with increased core counts. Each instance of the compiler might use a few hundred MB, which by itself is probably fine, but multiply that times 8, 16, or 32 and it adds up. And if you're building even a medium-sized c++ codebase you're really going to want to use all available cores.
And by “a few years” you mean a generation
Congrats Zig Team!
Allow me to comment from my professional perspective:
I have been working deep inside the spaghetti-stuffed soft underbelly of the computational infrastructures of multitudes of international “Big Science” projects and consortia. Zig seems like a very, very good fit. Are there any static code analysis tools that support it? Such functionality would be a killer app…
Wild idea: what about building one inside Zig? (or repurpose some other external tool to eventually integrate / be a “companion” tool and perform static analysis seamlessly)
PS: Of course with C support from the get go - maybe additionally other languages.
Allow me to comment from my professional perspective:
I have been working deep inside the spaghetti-stuffed soft underbelly of the computational infrastructures of multitudes of international “Big Science” projects and consortia. Zig seems like a very, very good fit. Are there any static code analysis tools that support it? Such functionality would be a killer app…
Wild idea: what about building one inside Zig? (or repurpose some other external tool to eventually integrate / be a “companion” tool and perform static analysis seamlessly)
PS: Of course with C support from the get go - maybe additionally other languages.
It would be awesome if zig implemented compiler hooks, for example: shoot off messages at various stages (sema of files, zir on individual functions, linker events), with some sort of defined binary structure (ok if it's defined in zig itself!) at say a tcp port.
Yes, that would be amazing, full classes of approaches to problem solving would be unlocked at once and we wouldn't have to invent new languages, DSLs, transpilers, etc, but just leverage zig.
When Golang switched to a self-hosted compiler, it lost compilation speed. When Zig switched to a self-hosted compiler, it gained compilation speed.
To all people complaining about bugs:
1. It's easy to find flaws in hard, ambitious projects. All ambitious projects start out as WIPs.
2. It is much harder to do an ambitious project.
Try to be in the 2nd group.
Try to be in the 2nd group.
But here's the thing about that though, when it comes to other languages which are not 1.0, various people will bash and troll to no end about any flaw, bug, or mistake they can find. Hell, even smear the developers at will.
How many times have we seen people dogging the hell out of Jai, Odin, Vlang, Carbon (that just started), and other newbies? People will dog other languages out and even compare them to 10 year old or older languages, without blinking.
How many times have we seen people dogging the hell out of Jai, Odin, Vlang, Carbon (that just started), and other newbies? People will dog other languages out and even compare them to 10 year old or older languages, without blinking.
Zig is not the first new language I try. I was an excited early adopted of a lot of many languages including D, Go, Rust and TypeScript. At no point to I remember running into miscompilations on rather fundamental issues and so quickly.
So I do think this is quite out of the ordinary.
So I do think this is quite out of the ordinary.
Consulting Wikipedia's "first appeared" field for each:
D: 8 December 2001; 20 years ago
Go: November 10, 2009; 12 years ago
Rust: July 7, 2010; 12 years ago
TypeScript: 1 October 2012; 10 years ago
Zig: 8 February 2016; 6 years ago
The only one of these which isn't at least twice as old as Zig is TypeScript, which has the might of Microsoft behind it, and is based on a well understood, pre-existing language.
This is not a fair comparison.
D: 8 December 2001; 20 years ago
Go: November 10, 2009; 12 years ago
Rust: July 7, 2010; 12 years ago
TypeScript: 1 October 2012; 10 years ago
Zig: 8 February 2016; 6 years ago
The only one of these which isn't at least twice as old as Zig is TypeScript, which has the might of Microsoft behind it, and is based on a well understood, pre-existing language.
This is not a fair comparison.
The more reasonable interpretation (backed up by his other posts on this article and his use of "was") is that he is comparing Zig as it stands today to D, Go, Rust, and TypeScript when they were similarly new.
Fair point.
However, when it comes to comparing just as old or even newer languages than Zig, such as Odin, Jai, Vlang, Carbon, etc... Various people have no mercy whatsoever.
I hope Zig continues to ship clang, because Zig is the easiest way to install clang.
And to build C and C++ projects!
https://kristoff.it/blog/maintain-it-with-zig/
https://kristoff.it/blog/maintain-it-with-zig/
If i am not mistaken, that's one of the larger selling points in Favour of Zig, isn't it?
I am pretty sure the C/C++ capabilities will stay in the Zig toolchain for a very long time :)
I am pretty sure the C/C++ capabilities will stay in the Zig toolchain for a very long time :)
> The package manager will not assume the presence of a central package index. We don’t plan to create an official package index.
This is great to hear!
This is great to hear!
> What’s interesting about this [C] backend is that it will play a role in our plan to replace the old bootstrap compiler implementation
That's super neat. I'm guessing the plan is to generate stage1 during CI or something with this backend?
That's super neat. I'm guessing the plan is to generate stage1 during CI or something with this backend?
I wonder if Zig is getting too much hype at the moment considering its immaturity (and so many easy to hit bugs as noted by other commenters). Is it getting hype before it's ready? Will it be able to recover from that?
> Is it getting hype before it's ready? Will it be able to recover from that?
Isn't that the very definition of hype? :^)
We can't hide in a cave to build everything in secret until it's ready because otherwise we wouldn't have the money to pay contributors, or we would be forced to get funding through other means (I mention that point in the blog post). And while we do want to be in the open, you can see that in my writing I never talk about a future that's distant and completely unproven. Even the most forward-looking stuff I mention, like incremental compilation, has at least had a proof of concept to show that it's not pure fiction.
I'm obviously biased, but I think we're in a pretty good spot when it comes to gathering public interest vs getting things done.
Isn't that the very definition of hype? :^)
We can't hide in a cave to build everything in secret until it's ready because otherwise we wouldn't have the money to pay contributors, or we would be forced to get funding through other means (I mention that point in the blog post). And while we do want to be in the open, you can see that in my writing I never talk about a future that's distant and completely unproven. Even the most forward-looking stuff I mention, like incremental compilation, has at least had a proof of concept to show that it's not pure fiction.
I'm obviously biased, but I think we're in a pretty good spot when it comes to gathering public interest vs getting things done.
> We can't hide in a cave to build everything in secret until it's ready because otherwise we wouldn't have the money to pay contributors, or we would be forced to get funding through other means (I mention that point in the blog post).
Can you see though how this model can cause a lack of trust in external observers? Your development model is reliant on attracting attention to drive funding. There are some major incentive problems here. I want to be clear that I am not suggesting that you are deliberately doing this but your model actually incentivizes building a lower quality product. You just need to make something appealing enough that people start using it and then, once they are invested in the language, some of them may be willing to pay you to fix the issues they find or build new features that would be useful to them. If the language didn't have issues, there would be no reason to pay you to fix them.
I know I will likely get downvoted for this but I believe in what I say.
Can you see though how this model can cause a lack of trust in external observers? Your development model is reliant on attracting attention to drive funding. There are some major incentive problems here. I want to be clear that I am not suggesting that you are deliberately doing this but your model actually incentivizes building a lower quality product. You just need to make something appealing enough that people start using it and then, once they are invested in the language, some of them may be willing to pay you to fix the issues they find or build new features that would be useful to them. If the language didn't have issues, there would be no reason to pay you to fix them.
I know I will likely get downvoted for this but I believe in what I say.
> Your development model is reliant on attracting attention to drive funding.
Yes, what alternative do you recommend? VC funding? You think any other financing model would result in being less reliant on attention as opposed to enhancing the effect?
> I want to be clear that I am not suggesting that you are deliberately doing this but your model actually incentivizes building a lower quality product.
You're writing this in a discussion about us having just spent two years reimplementing the compiler to pay tech debt, instead of doing any of the things that you mentioned.
I have to admit that your post smells a bit of concern trolling, especially considering that we've seen a bunch of that in the last few weeks.
Yes, what alternative do you recommend? VC funding? You think any other financing model would result in being less reliant on attention as opposed to enhancing the effect?
> I want to be clear that I am not suggesting that you are deliberately doing this but your model actually incentivizes building a lower quality product.
You're writing this in a discussion about us having just spent two years reimplementing the compiler to pay tech debt, instead of doing any of the things that you mentioned.
I have to admit that your post smells a bit of concern trolling, especially considering that we've seen a bunch of that in the last few weeks.
> Can you see though how this model can cause a lack of trust in external observers? Your development model is reliant on attracting attention to drive funding. There are some major incentive problems here... You just need to make something appealing enough that people start using it and then, once they are invested in the language, some of them may be willing to pay you to fix the issues they find or build new features that would be useful to them.
As also a 3rd party observer, there is an obvious partial counter to the argument that you made. In the case of free and open-source, people are investing money or time because they believe in the product and its potential usefulness. That the product will deliver, for all people invested and for all users, is clearly unrealistic. There is no such thing as a "sure thing". It's more a matter of if the product delivers for enough people. And if anything, if unsatisfied, people are free to jump to the next candidate or savior that floats by.
I think the more valid aspect of the point presented, is the lack of trust or suspicions coming from when the hype is paid for and doesn't come from the grassroots and actual users. To include, negative propaganda campaigns targeting competing languages.
> If the language didn't have issues, there would be no reason to pay you to fix them.
All languages will have issues. I've always found this curious about people having such expectations. Maybe the argument is more valid in terms of the total amount of issues and bugs, but then that would need to be a fair and relative comparison to other languages. There is no such thing as a bug-free perfect language. While other parts of your post were "more valid" or at least points worth considering, I think on this point, it kind of fell off the mark.
> I know I will likely get downvoted for this but I believe in what I say.
Odd, that's exactly what happened. The tribalism is arguably too severe, which may relate to the money involved, that the downvoting is so predictable.
As also a 3rd party observer, there is an obvious partial counter to the argument that you made. In the case of free and open-source, people are investing money or time because they believe in the product and its potential usefulness. That the product will deliver, for all people invested and for all users, is clearly unrealistic. There is no such thing as a "sure thing". It's more a matter of if the product delivers for enough people. And if anything, if unsatisfied, people are free to jump to the next candidate or savior that floats by.
I think the more valid aspect of the point presented, is the lack of trust or suspicions coming from when the hype is paid for and doesn't come from the grassroots and actual users. To include, negative propaganda campaigns targeting competing languages.
> If the language didn't have issues, there would be no reason to pay you to fix them.
All languages will have issues. I've always found this curious about people having such expectations. Maybe the argument is more valid in terms of the total amount of issues and bugs, but then that would need to be a fair and relative comparison to other languages. There is no such thing as a bug-free perfect language. While other parts of your post were "more valid" or at least points worth considering, I think on this point, it kind of fell off the mark.
> I know I will likely get downvoted for this but I believe in what I say.
Odd, that's exactly what happened. The tribalism is arguably too severe, which may relate to the money involved, that the downvoting is so predictable.
Disclaimer: Not a Zig fan.
I think Zig is getting too much hype. It's unfortunate because Zig could have potential, but hyping something before it's ready is a recipe for alienating early adopters and killing progress towards popularity.
The worst part of it is that I've heard people say that they've got Zig in production; with all of the bugs in the compiler, they are apt to get burned, and too many burns will lead to putting out the fire and rewriting.
It doesn't help that (in my opinion) there is confusion regarding basic things about Zig. [1]
[1]: https://gavinhoward.com/2022/04/i-believe-zig-has-function-c...
I think Zig is getting too much hype. It's unfortunate because Zig could have potential, but hyping something before it's ready is a recipe for alienating early adopters and killing progress towards popularity.
The worst part of it is that I've heard people say that they've got Zig in production; with all of the bugs in the compiler, they are apt to get burned, and too many burns will lead to putting out the fire and rewriting.
It doesn't help that (in my opinion) there is confusion regarding basic things about Zig. [1]
[1]: https://gavinhoward.com/2022/04/i-believe-zig-has-function-c...
Ugh that "Zig has function coloring" post again. The misunderstanding that post makes is that function coloring refers to at compile time. In other words, in Zig you don't need to write two versions of a function to use it in async vs sync context. Obviously the functions will have different runtime representations, so if you do something that cares about runtime calling conventions or w/e then yes you do need to worry about how the function is used, which is unavoidable.
The person you are replying to (me) is the author of that post.
> The misunderstanding that post makes is that function coloring refers to at compile time.
The original function colors post referred to runtime, so you're wrong.
> In other words, in Zig you don't need to write two versions of a function to use it in async vs sync context.
This is correct, if you only care about compile time.
> Obviously the functions will have different runtime representations, so if you do something that cares about runtime calling conventions or w/e then yes you do need to worry about how the function is used, which is unavoidable.
So you admit that Zig does have function colors?
Also, according to the actual Zig documentation, which I quote in my post, there is only one version of every function; if a function is used in an async context, even if it is sync otherwise, the compiler makes the function async and continues on its merry way, so there is only one runtime representation of a function.
Here's the quote from the documentation:
> Zig infers that a function is async when it observes that the function contains a suspension point. Async functions can be called the same as normal functions. A function call of an async function is a suspend point.
Saying that Zig does not have function colors because it hides them at compile time is a little disingenuous, in my opinion. It still has them, but like a statically-typed language with type inference, it just hides that fact, but only at compile time.
> The misunderstanding that post makes is that function coloring refers to at compile time.
The original function colors post referred to runtime, so you're wrong.
> In other words, in Zig you don't need to write two versions of a function to use it in async vs sync context.
This is correct, if you only care about compile time.
> Obviously the functions will have different runtime representations, so if you do something that cares about runtime calling conventions or w/e then yes you do need to worry about how the function is used, which is unavoidable.
So you admit that Zig does have function colors?
Also, according to the actual Zig documentation, which I quote in my post, there is only one version of every function; if a function is used in an async context, even if it is sync otherwise, the compiler makes the function async and continues on its merry way, so there is only one runtime representation of a function.
Here's the quote from the documentation:
> Zig infers that a function is async when it observes that the function contains a suspension point. Async functions can be called the same as normal functions. A function call of an async function is a suspend point.
Saying that Zig does not have function colors because it hides them at compile time is a little disingenuous, in my opinion. It still has them, but like a statically-typed language with type inference, it just hides that fact, but only at compile time.
> Saying that Zig does not have function colors because it hides them at compile time is a little disingenuous, in my opinion.
Not to impugn your character, but I feel the same way about the points you're making.
> So you admit that Zig does have function colors?
Not in any meaningful way, no. Again, the runtime representation of async vs sync functions is unavoidably different, and has nothing to do with the language. The fact that Zig lets you write one function definition that can be called in sync vs async contexts (unless you're specifically choosing to do sync vs async things) is what it means to be colorless!
> Also, according to the actual Zig documentation, which I quote in my post, there is only one version of every function; if a function is used in an async context, even if it is sync otherwise, the compiler makes the function async and continues on its merry way, so there is only one runtime representation of a function.
I dunno if that's accurate? Someone with more knowledge of Zig internals would have to confirm, but I assume if the same codebase uses a function in each context it'll be monomorphized into the specific version needed, same as generics. I don't think that's contradicted by what you quoted.
Not to impugn your character, but I feel the same way about the points you're making.
> So you admit that Zig does have function colors?
Not in any meaningful way, no. Again, the runtime representation of async vs sync functions is unavoidably different, and has nothing to do with the language. The fact that Zig lets you write one function definition that can be called in sync vs async contexts (unless you're specifically choosing to do sync vs async things) is what it means to be colorless!
> Also, according to the actual Zig documentation, which I quote in my post, there is only one version of every function; if a function is used in an async context, even if it is sync otherwise, the compiler makes the function async and continues on its merry way, so there is only one runtime representation of a function.
I dunno if that's accurate? Someone with more knowledge of Zig internals would have to confirm, but I assume if the same codebase uses a function in each context it'll be monomorphized into the specific version needed, same as generics. I don't think that's contradicted by what you quoted.
It's not accurate that there's only one representation of every function at runtime. Completely unrelated to async, Zig uses compile-time parameters to do stuff similar to templates which will get you lots and lots of monomorphized functions, and you obviously can't at runtime have a pointer to a function that takes a compile-time argument (just like you can't have a pointer to vector<T>::push, to which you would pass the type T at runtime, then a this pointer which is a vector<T>, then a T t to append, haha). Then, for async specifically, usually the way that the same function can be compiled to be async or non-async is for it to be generic over some other stuff at compile time and for that other stuff to either be async or not be async. For example, you could have a function that writes a websocket frame to a writer, and that function is async if the writer is async, but it's not async otherwise. But it's also specialized to each writer in other ways, so if you use it on a lot of types of writers, you'll get more than just 2 copies (one async and one non-async) of it in your compiled program.
In most programs you end up with all async functions or all non-async functions in a specific context, so if someone handed you a pointer you'd have no problem knowing which calling convention to use. But if you actually end up not knowing, because you really do have a mix of different types of pointers, you could call all of the pointers via the @asyncCall builtin. I have done this successfully, but I'm not actually sure whether it worked by having a pointer to a non-async function at runtime or by specializing the function that I was using that did not need to be async (because its body and callees did not use await or suspend) to use the async calling convention. Or you could use a tagged union of multiple function types and make your own `call` method for that union. This is pretty weird though, because you'll end up requiring the end user to provide storage for the async frame even when the function isn't async and doesn't actually need that storage.
In most programs you end up with all async functions or all non-async functions in a specific context, so if someone handed you a pointer you'd have no problem knowing which calling convention to use. But if you actually end up not knowing, because you really do have a mix of different types of pointers, you could call all of the pointers via the @asyncCall builtin. I have done this successfully, but I'm not actually sure whether it worked by having a pointer to a non-async function at runtime or by specializing the function that I was using that did not need to be async (because its body and callees did not use await or suspend) to use the async calling convention. Or you could use a tagged union of multiple function types and make your own `call` method for that union. This is pretty weird though, because you'll end up requiring the end user to provide storage for the async frame even when the function isn't async and doesn't actually need that storage.
> I dunno if that's accurate?
It's not, your understanding of async in Zig is sound. I would even add that in the blog post in question (my post, the one ghoward is replying to in his) I don't even dare to describe Zig as colorless, in fact I say colorblind.
It's not, your understanding of async in Zig is sound. I would even add that in the blog post in question (my post, the one ghoward is replying to in his) I don't even dare to describe Zig as colorless, in fact I say colorblind.
> It's not, your understanding of async in Zig is sound.
Then perhaps there is something wrong with the language reference?
> I don't even dare to describe Zig as colorless, in fact I say colorblind.
Doesn't that mean my blog post is right?
But my biggest beef is that people, including GP, used your blog post to then claim Zig is colorless. I mean, GP did it in GP.
Then perhaps there is something wrong with the language reference?
> I don't even dare to describe Zig as colorless, in fact I say colorblind.
Doesn't that mean my blog post is right?
But my biggest beef is that people, including GP, used your blog post to then claim Zig is colorless. I mean, GP did it in GP.
I don't really understand your crusade. I made this same observation in the past, it never satisfied you.
Your blog post is full of wrong information. I tried to explain to you what was wrong when you first posted it (so you can refer to those comments, if you want), but you keep seeing this as some kind of philosophical debate, and I have no interest in having this debate.
As I said to you already in the past, I just write software with Zig async and it works. Up to you what you want to do with your free time.
Your blog post is full of wrong information. I tried to explain to you what was wrong when you first posted it (so you can refer to those comments, if you want), but you keep seeing this as some kind of philosophical debate, and I have no interest in having this debate.
As I said to you already in the past, I just write software with Zig async and it works. Up to you what you want to do with your free time.
> I don't really understand your crusade.
Accuracy is important in the marketplace of ideas, and especially in programming. Software is too buggy already, and it would only add more bugs to have programmers not understand the languages they use.
> I made this same observation in the past, it never satisfied you.
Yes, you made that same observation, and I appreciate that. But as @kbd so unintentionally demonstrated, people still believe that Zig is colorless. I want to dispel that notion completely.
I think you are not adding to the problem, and that is great. But the notion is still there.
> Your blog post is full of wrong information. I tried to explain to you what was wrong when you first posted it (so you can refer to those comments, if you want), but you keep seeing this as some kind of philosophical debate, and I have no interest in having this debate.
Here is all of the comments you made on Hacker News on the comments [1] about my blog post.
> That's exactly it. It just enables code reuse. You still have to think about how your application will behave, but you won't have to use an async-flavored reimplementaion of another library. Case in point: zig-okredis works in both sync and async applicatons, and I don't have to maintain two codebases.
> https://github.com/kristoff-it/zig-okredis
> I thought using "colorblind" in the title of my original blogpost would be a clear enough hint to the reader that the colors still exist, but I guess you can never be too explicit.
and
> That's how it works in Zig. Calling an async function like this will also await it.
The closest thing to "explain[ing] to [me] what was wrong when [I] first posted it" is probably that first comment, which was in reply to
> I may be totally wrong with this assumption, but the way I understoo[d] Zig's color-less async support is that the compiler either creates a "red" or "blue" function body from the same source code based on how the function is called (so on the language level, function coloring doesn't matter, but it does in compiler output).
> The compiler still needs to stamp out colored function bodies because the generated code for a function with async support needs to look different - the compiler needs to turn the code into a state machine instead of a simple sequence).
> It's a bit unfortunate that red and blue functions appear to have a different "ABI signature", but I guess that's needed to pass an additional context pointer into a function with async support (which would otherwise be the implicit stack pointer).
(Original comment at [2] by flohofwoe.)
So if anybody explained anything, it's flohofwoe.
But flohofwoe's comment goes directly against the the language reference, so it's hard for me to believe.
The language reference says that sync functions are turned async if they call async functions. This implies virality of async on functions, which implies that many functions are definitely async-only.
If the compiler does something different, which it would have to if it actually makes two different versions of each function, then the language reference is wrong. Like I said, accuracy matters, so I would also like to see changes in the Zig language reference about this if that's the case.
> As I said to you already in the past, I just write software with Zig async and it works.
Yes, you write working software in Zig async, but you understand it better than most. People who go to the language reference and write based on that may not be able to write working software with Zig async as easily as you.
[1]: https://news.ycombinator.com/item?id=30965805
[2]: https://news.ycombinator.com/item?id=30967070
Accuracy is important in the marketplace of ideas, and especially in programming. Software is too buggy already, and it would only add more bugs to have programmers not understand the languages they use.
> I made this same observation in the past, it never satisfied you.
Yes, you made that same observation, and I appreciate that. But as @kbd so unintentionally demonstrated, people still believe that Zig is colorless. I want to dispel that notion completely.
I think you are not adding to the problem, and that is great. But the notion is still there.
> Your blog post is full of wrong information. I tried to explain to you what was wrong when you first posted it (so you can refer to those comments, if you want), but you keep seeing this as some kind of philosophical debate, and I have no interest in having this debate.
Here is all of the comments you made on Hacker News on the comments [1] about my blog post.
> That's exactly it. It just enables code reuse. You still have to think about how your application will behave, but you won't have to use an async-flavored reimplementaion of another library. Case in point: zig-okredis works in both sync and async applicatons, and I don't have to maintain two codebases.
> https://github.com/kristoff-it/zig-okredis
> I thought using "colorblind" in the title of my original blogpost would be a clear enough hint to the reader that the colors still exist, but I guess you can never be too explicit.
and
> That's how it works in Zig. Calling an async function like this will also await it.
The closest thing to "explain[ing] to [me] what was wrong when [I] first posted it" is probably that first comment, which was in reply to
> I may be totally wrong with this assumption, but the way I understoo[d] Zig's color-less async support is that the compiler either creates a "red" or "blue" function body from the same source code based on how the function is called (so on the language level, function coloring doesn't matter, but it does in compiler output).
> The compiler still needs to stamp out colored function bodies because the generated code for a function with async support needs to look different - the compiler needs to turn the code into a state machine instead of a simple sequence).
> It's a bit unfortunate that red and blue functions appear to have a different "ABI signature", but I guess that's needed to pass an additional context pointer into a function with async support (which would otherwise be the implicit stack pointer).
(Original comment at [2] by flohofwoe.)
So if anybody explained anything, it's flohofwoe.
But flohofwoe's comment goes directly against the the language reference, so it's hard for me to believe.
The language reference says that sync functions are turned async if they call async functions. This implies virality of async on functions, which implies that many functions are definitely async-only.
If the compiler does something different, which it would have to if it actually makes two different versions of each function, then the language reference is wrong. Like I said, accuracy matters, so I would also like to see changes in the Zig language reference about this if that's the case.
> As I said to you already in the past, I just write software with Zig async and it works.
Yes, you write working software in Zig async, but you understand it better than most. People who go to the language reference and write based on that may not be able to write working software with Zig async as easily as you.
[1]: https://news.ycombinator.com/item?id=30965805
[2]: https://news.ycombinator.com/item?id=30967070
> The language reference says that sync functions are turned async if they call async functions. This implies virality of async on functions, which implies that many functions are definitely async-only.
> If the compiler does something different, which it would have to if it actually makes two different versions of each function, then the language reference is wrong. Like I said, accuracy matters, so I would also like to see changes in the Zig language reference about this if that's the case.
To be clear, do you expect this function
> If the compiler does something different, which it would have to if it actually makes two different versions of each function, then the language reference is wrong. Like I said, accuracy matters, so I would also like to see changes in the Zig language reference about this if that's the case.
To be clear, do you expect this function
fn invokeFoo(a: anytype, b: anytype) @TypeOf(a) {
return a.foo(b);
}
That's a generic function. That's not what I'm talking about.
Thanks for confirming. So what does Zig actually do to execute async functions? Green threads in the background? An event loop? Is there documentation on the implementation?
When you set `const io_mode = .evented;` the stdlib observes that value and
- flips I/O functions to evented mode, both by setting the correct flags when creating, say, a socket, and also by having `suspend` and similar keywords in the winning branch of some comptime conditional statements, causing the functions in question to become async also in the language.
- instead of directly calling into main at program start, it starts an event loop and schedules main on it
This is all userland stuff based on a convention, none of this is ingrained in the programming language itself and it could very well be that in the future things will have to be done in a more explicit way.
Async functions are single frames (as opposed to being green threads).
- flips I/O functions to evented mode, both by setting the correct flags when creating, say, a socket, and also by having `suspend` and similar keywords in the winning branch of some comptime conditional statements, causing the functions in question to become async also in the language.
- instead of directly calling into main at program start, it starts an event loop and schedules main on it
This is all userland stuff based on a convention, none of this is ingrained in the programming language itself and it could very well be that in the future things will have to be done in a more explicit way.
Async functions are single frames (as opposed to being green threads).
> Not to impugn your character, but I feel the same way about the points you're making.
Again, the original post about function colors talks about runtime, so how could I be disingenuous about that?
> Not in any meaningful way, no. Again, the runtime representation of async vs sync functions is unavoidably different, and has nothing to do with the language. The fact that Zig lets you write one function definition that can be called in sync vs async contexts (unless you're specifically choosing to do sync vs async things) is what it means to be colorless!
I mean, this would be true if it were true. But I tried to use the "blue" function in my examples in both contexts, and it didn't work. In fact, it didn't even work to use the "red" function in both contexts.
> I dunno if that's accurate? Someone with more knowledge of Zig internals would have to confirm, but I assume if the same codebase uses a function in each context it'll be monomorphized into the specific version needed, same as generics. I don't think that's contradicted by what you quoted.
I worded this completely wrong, so let me fix that. I should have said,
> if an async function is used in a sync context, the compiler makes the context async and continues on its merry way, so there is only one runtime representation of the function that was the context.
But it means that the sync function that was the caller is made async no matter what because it calls an async function. So there is no monomorphization; there cannot be since there is a call to an async function that always makes an async context that always turns the caller async.
Again, the original post about function colors talks about runtime, so how could I be disingenuous about that?
> Not in any meaningful way, no. Again, the runtime representation of async vs sync functions is unavoidably different, and has nothing to do with the language. The fact that Zig lets you write one function definition that can be called in sync vs async contexts (unless you're specifically choosing to do sync vs async things) is what it means to be colorless!
I mean, this would be true if it were true. But I tried to use the "blue" function in my examples in both contexts, and it didn't work. In fact, it didn't even work to use the "red" function in both contexts.
> I dunno if that's accurate? Someone with more knowledge of Zig internals would have to confirm, but I assume if the same codebase uses a function in each context it'll be monomorphized into the specific version needed, same as generics. I don't think that's contradicted by what you quoted.
I worded this completely wrong, so let me fix that. I should have said,
> if an async function is used in a sync context, the compiler makes the context async and continues on its merry way, so there is only one runtime representation of the function that was the context.
But it means that the sync function that was the caller is made async no matter what because it calls an async function. So there is no monomorphization; there cannot be since there is a call to an async function that always makes an async context that always turns the caller async.
A number of those using zig in production have played with zig for a considerable amount of time (I started playing with 0.5.0 where large breaking changes came shortly after) thus know what to expect. Afaik most of those are within embedded, using it as a toolchain (zig cc not zig), or have similar constraints.
How would you expect a language as low-level as Zig is to support colorless async? It pretty much requires some kind of green threads.
I agree, and that's okay, but the problem is that many people assume Zig is colorless. I just want to dispel that notion.
Only by your definition. Colorless was always about DX, nowhere in the original article did anyone care about implementation. And this stuff does matter. I have created libraries which provides calls in both sync and async contexts to the same function and it works just fine.
Does developer experience not include using function pointers in Zig?
How is calling non-async functions via @asyncCall insufficient?
@asyncCall is exactly the sort of thing the original function colors calls out.
Actually, it's not... since you can use it on async and non async functions alike, which is what GP says. PLs with colored functions can't do the same thing at all, moreover the virality of @asynccall terminates at the point where you use it; if it were colored, the context of @asynccall would have to be async, and its parent and grandparent, etc.
Alright, and why should we care about it?
Maybe like Modula-2?
Compared to Rust at this stage of its development? No, Rust was far more popular.
If you search headline with Zig which reached HN front page, you will only see about 1 - 2 per month. Compared that to Rust, which isn't even as popular as it once was.
But still, I do wish the Zig community could quiet down a little bit. It would be nice if decent zig submission hit HN front page once every two months. Keeping the interest alive but not too much so people felt it is over hyped.
We will have a November Release soon and I can guarantee it will hit HN front page again.
If you search headline with Zig which reached HN front page, you will only see about 1 - 2 per month. Compared that to Rust, which isn't even as popular as it once was.
But still, I do wish the Zig community could quiet down a little bit. It would be nice if decent zig submission hit HN front page once every two months. Keeping the interest alive but not too much so people felt it is over hyped.
We will have a November Release soon and I can guarantee it will hit HN front page again.
It would be nice to have a public Zig webassembly compiler that anyone could import and use.
What is the reason to not have a central package index? I find that having one speeds up my development significantly. Is there more reason than decentralization?
Decentralization is good.
I'm really not a fan of what has happened in the Rust community with crates.
The fact that there is a single blessed Rust crate source and the fact that the Rust community doesn't take responsibility for vetting crates means that maintainership changes, name squatting gold rush, and a whole host of pathological behaviors show up.
If you're not going to take responsibility, then the only option is to leave everything decentralized.
I'm really not a fan of what has happened in the Rust community with crates.
The fact that there is a single blessed Rust crate source and the fact that the Rust community doesn't take responsibility for vetting crates means that maintainership changes, name squatting gold rush, and a whole host of pathological behaviors show up.
If you're not going to take responsibility, then the only option is to leave everything decentralized.
I don't know but hopefully there would be something like Go's module proxy, which can be replaced or turned off.
https://proxy.golang.org/
https://proxy.golang.org/
[deleted]
Port to 9front!
moffkalast(2)
Write a "Zig in Y minutes", to me a language doesn't exist if it's not there.
Add vector math types/functions/operators, those should be standard, in my view.
Add vector math types/functions/operators, those should be standard, in my view.
Then Zig is indeed real :^)
https://gist.github.com/ityonemo/769532c2017ed9143f3571e5ac1...
https://gist.github.com/ityonemo/769532c2017ed9143f3571e5ac1...
What do you mean by vector math types/functions/operators? There's built-in SIMD stuff but that may not be the right thing.
Zig seems to have captured a lot of attention in the new language design space, and I personally find this unfortunate. Frankly, I find it irresponsible that the team has marketed a pre 1.0 language so heavily. By constantly evolving the language and making frequent breaking changes, they are quite literally wasting people's time. They also have been very hostile to people who have raised fair points about the misleading claims that they have made.
From my perspective, the entire premise of the language is flawed. The main project goal seems to be to become a replacement for c. This means that in some sense the project is implicitly defined by c and doesn't really stand on its own conceptually. This could be ok if it actually fixed some of the worst problems with c, but most of what it offers appears to be relatively superficial syntactic improvements (which could easily be implemented by a simple transpiler to c). While I am sure that the language designers see things differently, it appears that Zig just lets c programmers keep writing c with some additional compiler checks. This is probably an improvement on c (at least in some respects), but it seems largely incremental. Also because they prematurely evangelized the language, they are likely to be weighed down by the additional complexity that seeped in from trying to support so many different use cases before they had really figured out what the language is.
That it took two years to write the self hosting compiler speaks to the fact that this language is already quite complex. My guess is that this complexity is always going to pose problems. Downstream users are always going to be finding obscure edge cases that don't work because the language supports so many different things that actually testing everything is going to be extremely difficult.
Having said all that, if you love Zig, that's great. I don't doubt that it offers some real quality of life improvements for you. I just personally believe that this project is an evolutionary dead end and that it is very unlikely to have significant mindshare/deployment in say, 10-20 years.
From my perspective, the entire premise of the language is flawed. The main project goal seems to be to become a replacement for c. This means that in some sense the project is implicitly defined by c and doesn't really stand on its own conceptually. This could be ok if it actually fixed some of the worst problems with c, but most of what it offers appears to be relatively superficial syntactic improvements (which could easily be implemented by a simple transpiler to c). While I am sure that the language designers see things differently, it appears that Zig just lets c programmers keep writing c with some additional compiler checks. This is probably an improvement on c (at least in some respects), but it seems largely incremental. Also because they prematurely evangelized the language, they are likely to be weighed down by the additional complexity that seeped in from trying to support so many different use cases before they had really figured out what the language is.
That it took two years to write the self hosting compiler speaks to the fact that this language is already quite complex. My guess is that this complexity is always going to pose problems. Downstream users are always going to be finding obscure edge cases that don't work because the language supports so many different things that actually testing everything is going to be extremely difficult.
Having said all that, if you love Zig, that's great. I don't doubt that it offers some real quality of life improvements for you. I just personally believe that this project is an evolutionary dead end and that it is very unlikely to have significant mindshare/deployment in say, 10-20 years.
Why do you care so much about a language you have no interest in? Just ignore it and move on, there's plenty of programming languages under the sky to have fun with and which probably fit your idea of a good programming language much better than Zig.
I haven't noticed miscompilation yet but I have noticed occasional unclear error messages such as it pointing at the wrong line. I've meant to make a note of these but since I'm just focused on learning I haven't done a great job of writing down these minor bugs.
I don't think the breaking changes (which haven't seemed that big) are a huge deal for most folks curious about PL in general. And I think the potential for being a better C is just so great that it's worth continuing to try out Zig and follow it's development. In particular I think it would make a lot of sense as a teaching language for the college courses that would otherwise use C. And I think it would be a pretty fancy compilation target for language implementations like Chicken Scheme that otherwise target C. (I'm sure it would be great in other situations too but these two are ones I can relate to most.)
But I'm still early on in my exploration of Zig. (Disclaimer: the company I work for uses Zig and sponsors ZSF.)