Anecdotally again, but I've been coding in Zig since 2020 and have hit I think 2-3 compiler bugs in all that time?
The first was fixed within 24 hours, in fact just before I reported it. The others had clear TODO error messages in the panic, and there were easy enough workarounds.
Static allocation has also made TigerBeetle's code cleaner, by eliminating branching at call sites where before a message might not always have been available. With static allocation, there's no branch because a message is always guaranteed to be available.
It's also made TigerBeetle's code more reliable, because tests can assert that limits are never exceeded. This has detected rare leaks that might otherwise have only been detected in production.
We use what we have available, according to the context: checksums, assertions, hash chains. You can't always use every technique. But anything that can possibly be verified online, we do.
Buffer bleeds also terrify me. In fact, I worked on static analysis tooling to detect zero day buffer bleed exploits in the Zip file format [1].
However, to be clear, the heart of a bleed is a logic error, and therefore even memory safe languages such as JavaScript can be vulnerable.
To be fair, we're certainly concerned about logic errors and buffer bleeds. The philosophy in TigerBeetle is always to downgrade a worse bug to a lesser. For example, if it's a choice between correctness and liveness, we'll downgrade the potential correctness bug to a crash.
In the specific case of message buffer reuse here, our last line of defense then is also TigerBeetle's assertions, hash chains and checksums. These exhaustively check all function pre/post-conditions, arguments, processing steps and return values. The assertion-function ratio is then also tracked for coverage, especially in critical sections like our consensus or storage engine.
So—apologies for the wince! I feel it too, this would certainly be a nasty bug if it were to happen.
Good to be back—Joran from the TigerBeetle team here!
Static allocation does make for some extremely hard guarantees on p100 latency. For example, for a batch of 8191 queries, the performance plot is like Y=10ms, i.e. a flat line.
And memory doesn't increase as throughput increases—it's just another flat line.
I personally find it also to be a fun way of coding, everything is explicit and limits are well-defined.
> I would imagine this would be potentially more efficient than having conventional mallocs.
Yes, in our experience, static allocation means we use sometimes 10x less memory. For example, TigerBeetle's storage engine can theoretically address up to 100 TiB with less than 1 GiB of RAM for the LSM in-memory manifest, which is efficient.
Because we think about memory so much upfront, in the design phase, we tend not to waste it.
TigerBeetle's storage engine is designed also for range queries, and we have some interesting ideas for our query engine in the works.
To be sure, there are some tricky things that crop up, such as pipeline blockers for queries. However, we do have limits on literally everything, so that makes it easier—static allocation is best when it's done viral.
We wanted to go into so much more detail on all this but there was only space for so much. Stay tuned.
TigerBeetle uses Deterministic Simulation Testing to test and keep testing these paths. Fuzzing and static allocation are force multipliers when applied together, because you can now flush out leaks and deadlocks in testing, rather than letting these spillover into production.
Without static allocation, it's a little harder to find leaks in testing, because the limits that would define a leak are not explicit.
This was in fact one of our motivations for static allocation—thinking about how best to handle overload from the network, while remaining stable. The Google SRE book has a great chapter on this called "Handling Overload" and this had an impact on us. We were thinking, well, how do we get this right for a database?
We also wanted to make explicit what is often implicit, so that the operator has a clear sense of how to provision their API layer around TigerBeetle.
As you zoom in, you will see differences. For example, TigerBeetle's data structures are all cache line aligned, and we use static allocation etc. The storage fault model is significantly different.
We use the same testing techniques though. FDB are pioneers in the space.
Thanks, it's a pleasure! Yes, with packed structs it's important to keep things carefully aligned.
Have you seen this post [1] about struct packing? This is what we did for TB's structures, so that we can switch them to `extern struct` for C ABI compatibility. It also helped side step the packed struct bugs.
Awesome to hear that you're excited about Zig.
Thanks for sharing the link to our repo also—would love to take you on a 1-on-1 tour of the codebase sometime if you'd be up for that!