What makes Apache Kafka fast (2020)(medium.com)
medium.com
What makes Apache Kafka fast (2020)
https://medium.com/swlh/why-kafka-is-so-fast-bde0d987cd03
5 comments
TLDR: They use many good industry practices:
- Append-only log with sequential IO
- Record batching on client and broker side (works well with sequential IO)
- Batch compression (Json and other text formats could be compressed very well)
- Unflushed buffered writes
- Zero-copy approach for working with data
In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides.
The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that.
Edit: formatting
- Append-only log with sequential IO
- Record batching on client and broker side (works well with sequential IO)
- Batch compression (Json and other text formats could be compressed very well)
- Unflushed buffered writes
- Zero-copy approach for working with data
In addition to that the architecture of the Kafka also plays a major role so that the clients don't require much overhead. The code is written to not to put much pressure on GC and a number of other more specific optimizations was implemented on both client and server sides.
The most interesting part for me was how they achieve zero-copy when working with sockets. That whole section is worth reading and they could have made another blog post about only that.
Edit: formatting
So, if a system is predictable up to an upper bound of one year, is that still real-time? My understanding of real-time was pretty much as quickly as something happens with hard constraints on latency. I think "fast" is implied when we say real-time.