Are there performance worries about passing around two pointers for anything that needs to allocate, as well as storing these pointers in a struct? AFAICT this basically means two registers are eaten, and a lot of types have effectively 16 bytes of overhead. It seems like this could quickly change the calculus on what fits within cache lines and what doesn't, which people often care about for very high performance code.
I wonder if it's possible to change the compiler to detect that, if what is being used in arguments is the global default allocator, the first argument can be stripped and all references inside the function can be replaced with the global pointer. Potentially the same concept could apply to allocators that use thread local storage. (perhaps these optimizations already exist?)
I’ve tried to make the kgo package easier to use, but I think some libraries suffer from over abstracting the underlying protocol, which ultimately locks them out of some higher efficiency code or some correctness. So, I’ve tried to make the library simple yet not hide all of the details. It’s seemed to have panned out successfully for at least some people. Happy to answer any questions about it, and thanks for taking a look!
I spent quite some time writing github.com/twmb/franz-go due to wanting more from both Sarama and from Segment’s library. Pure Go as well.
I’d recommend evaluating it as an alternative, even to confluent-kafka-go due to comprehensive feature support and (afaict) even higher performance than the aforementioned libraries.
It isn't yet integrated with the schema registry, but I think doing so would be a fairly straightforward task. My primary goal before this "Show HN" was 2.6.0 feature completeness.
I'll look into integrating with the schema registry within the next month. Thanks for the reminder!
edit: there's a few other minor kafkacat features that I'd like to implement as well, such as start and end times for consuming, and producing entire files. I think those two, in addition to avro schema registry, are the main features this is missing, but otherwise kcl is much more featureful.
Long term, I think it'd be great to add support for secrets in vault, but I'd like a bit more usage before considering integrating that.
I wonder if it's possible to change the compiler to detect that, if what is being used in arguments is the global default allocator, the first argument can be stripped and all references inside the function can be replaced with the global pointer. Potentially the same concept could apply to allocators that use thread local storage. (perhaps these optimizations already exist?)