One of the advantages of of protobuf I never see anyone highlight is how neat and well-designed the wireformat is, in terms of backward/forward compatibility and lowlevel stuff you can do with it. Very useful when building big and demanding systems over time.
For high performance and critical stuff, SBE is much more suitable, but it doesn't have as good of a schema evolution story as protobuf.
This brings back memories debugging an azul zing bug where an effectively final optimization ended up doing the wrong thing with zstd-jni. It was painful enough that I couldn’t convince the team to enable the optimization again for years after it was fixed.
I agree. It’s the enshittification of the internet. Luckily we still have infrastructure providers with more sensible offerings. We don’t have to use aws, gcp, etc.
I've used bazel for about 10 years, all in small orgs. Right now, single digit team.
I don't think there exists a better solution for a project mixing Java and C/C++ than Bazel. The new module system for bazel has matured a lot. As an example, it's trivial to add boringssl and access it from Java.
That’s the default in the monorepos I’ve worked on.
When a third party dep is broken or needs a workaround, just include a patch in the build (or fork). Then those patches can be upstreamed asynchronously without slowing down development.
I really appreciate the work done to evolve Java. They seem to be able to both fix earlier mistakes (and there were many!) and new ones from happening. While I see other languages struggling with the weight of complexity added over time, Java seems to get simpler and easier even with significant updates to the language spec.
I think the «best practices» found in Java enterprise has meant that a lot of people think all Java has to look like that.
High performance Java is awesome and can be competitive with C++ and Rust for non trivial systems, thanks to much better development cycle, debugging and high quality libraries.
Most the benefits is the JVM, so Kotlin has those too, but I don’t feel Kotlin is enough of an improvement to warrant the downsides (at least for me). But both Kotlin, Scala and Clojure are great for the Java ecosystem.
Mechanisms for getting the linux kernel out of the way is pretty decent these days, and CPUs with a lot of cores are common. That means you can isolate a bunch of cores and pin threads the way you want, and then use some kernel-bypass to access hardware directly. Communicate between cores using ring buffers.
This gives you best of both worlds - carefully designed system for the hardware with near optimal performance, and still with the ability to take advantage of the full linux kernel for management, monitoring, debugging, etc.
Isolating a core and then pinning a single thread is the way to go to get both low latency and high throughput, sacrificing efficiency.
This works fine on Linux, and common approach for trading systems where it’s fine to oversubscribe a bunch of cores for this type of stuff. The cores are mostly busy spinning and doing nothing, so it’s very inefficient in terms of actual work, but great for latency and throughput when you need it.
In a previous project we used fixedint32/64 instead of varint values in the schema for messages where performance was critical.
That left only varint used for tags. For encoding we already know the size at compile time. We also don’t have to decode them since we can match on the raw bytes (again, known at compile time).
A final optimization is to make sure the message has a fixed size and then just mutate the bytes of a template directly before shipping it off. Hard to beat.
In a newer project we just use SBE, but it lacks some of the ergonomics of protobuf.
For high performance and critical stuff, SBE is much more suitable, but it doesn't have as good of a schema evolution story as protobuf.