Scaling Kafka clusters has always been a challenging task. Kafka uses the ISR multi-replica mechanism to ensure data persistence, which was a natural choice when the cloud was not yet mature. However, in 2024, when cloud computing is very mature, this design seems a bit outdated. When the cluster is scaled, we must move the data of the partitions. Moving partition data will affect normal reading and writing, and this process may last a long time, tens of minutes or a few hours, depending on the amount of your data. This means that often, even when the business scale has expanded to the critical point, we still dare not carry out such operations as expansion, because there is a high execution risk.
AutoMQ is fully aware of the root cause of this problem, so it has redesigned and implemented the entire storage layer of Kafka based on the cloud (we call ourselves cloud-first Kafka). Offload data persistence to cloud storage, and ensure data persistence by the multi-replica mechanism inside cloud storage. This also gives us the ability to build a more powerful self-balancing ability than Cruise Control. This blog post details how we technically achieve this, and we hope it can bring some new insights to everyone.
In recent years, we have seen many challengers to Kafka emerge. Without exception, they all claim to be fully compatible with the Kafka API. Some challengers have even directly forked Kafka's code to innovate further. Has the Kafka API now become the de facto standard in the stream processing domain? How has it achieved this status?
I agree with your viewpoint. The crux of the matter is not whether to use tiered storage or not, but what trade-offs have been made in the specific storage architecture and what benefits have been gained. Here(https://github.com/AutoMQ/automq?tab=readme-ov-file#-automq-...) is a qualitative comparison chart of streaming systems including kafka/confluent/redpanda/warpstream/automq. This comparison chart does not have specific numerical comparisons, but purely based on their trade-offs at the storage level, I think this will be of some use to you.
Buddy, you've hit the nail on the head. Everything is a trade-off. For a stream processing system, I believe it's entirely possible to balance cost, ease of use, and latency. AutoMQ(https://github.com/AutoMQ/automq) is also a stream system built on top of S3. Its storage scheme introduces a very small size of EBS storage as a persistent write cache, and then asynchronously compacts the memory data to S3, taking into account latency while retaining the advantages of warpstream. Tiered-storage is just a form, how to implement it depends on you.
Examples of this are quite common. Many tech company blog pages offer tag-based searches. The articles published can be tagged either by the system or manually, allowing us to filter content we're interested in by searching for these tags.
That makes sense. Every choice has its pros and cons. Simplicity is the philosophy of HN. If complexity gets out of control, HN would cease to exist. However, perhaps some small experiments could be feasible while maintaining simplicity?
Thank you for sharing. However, I might be looking for more specific categorizations. For instance, if I'm only interested in AWS-related content, I would like to be able to search for the 'AWS' tag.
This is a tremendous feat. Your success should be greatly tied to the actions you've taken. You've listed a lot of experiences that contributed to this outcome. Many projects have done the same, but they haven't succeeded. What I'm curious about is, among the suggestions you've listed, which measure has contributed the most to this result?
The ban is not just about the issue of traffic fabrication. The fake story created by this influencer has occupied limited social media resources. This has prevented many more important news and events from being understood by the public.
The elasticity of the public cloud mainly manifests in the user's ability to use its resources on demand. When you need more resources to support the workload, you can quickly purchase resources; when you no longer need these resources, you can release them immediately. The elasticity of the public cloud mentioned in the article mainly refers to its ability to provide resources to users on demand.
Thanks for your kind action.I want to know is the post really dead? When I look at it in my personal account's submissions, I find that the post has not been marked as dead. My account has been set to showdead=yes. If it's really as you said, does this mean that I can't see the dead tag myself?"
It's so strange, now others can see my post as well. The title of the post is: 'Why it's difficult to utilize the elasticity of the public cloud?'. When I encountered the issue, I also sent an email to the moderator explaining the situation. I wonder if it has anything to do with their intervention. Does Hacker News have a content review process?