Regarding this issue, I have some observations of my own. I've noticed that systems based on queues, such as Kafka, AMQP, etc., are still very widespread, for example in vehicle networking, transaction systems, and so on. I recently encountered a customer deploying Kafka on AWS, with monthly consumption of Kafka-related computing storage exceeding $1 million. The cluster scale is huge, containing various system events, logs, etc. I've also seen customers building IoT platforms based on Kafka. Kafka has become very central to the IoT platform, and any problems can cause the entire IoT platform to be unavailable. I personally have written over 80% of the code for Apache RocketMQ, and today I have created a new project, AutoMQ (https://github.com/AutoMQ/automq). At the same time, we also see that competition in this field is very fierce. Redpanda, Confluent, WarpStream, StreamNative, etc., are all projects built based on the Kafka ecosystem. Therefore, the architecture based on message queues has not become obsolete. A large part of the business has transformed into a Streaming form. I think Streaming and MQ are highly related. Streaming leans more towards data flow, while MQ leans more towards individual messages.
Have you seen AutoMQ's approach(https://github.com/AutoMQ/automq)?
It is hard to believe that users can tolerate a produce message latency of hundreds of milliseconds with WarpStream. As the Co-founder & CEO of AutoMQ, we have engaged with hundreds of users, many of whom are seeking both speed and reliability. So We require a stateless broker solution that is fully compatible with Apache Kafka, while also excelling in terms of low latency and cost effectiveness on cloud infrastructure.
The following explanation from Strimzi about the difficulty of implementing autoscaling in Kubernetes can also be applied to answer this question. Using spot instances requires the computing layer to be stateless.
For Kafka brokers, auto-scaling is complicated because of their architecture. Adding or removing brokers is simple. But directing some load to them is complicated because they are in a way form of data storage. And moving the whole partitions between brokers is expensive. The partitions often contain huge amounts of data which need to be shifted from one broker to another - that will take time, it will have a performance penalty on the other traffic and possibly cost even real money for the data transfers.
The table summarizes the differences between various streaming platforms. It highlights how different technical architecture choices can lead to different end results. What are your thoughts on this table?
I completely empathize with your situation. It can indeed be frustrating when you're on a job hunt and going through what you're experiencing. But don't lose hope. You're not alone in this. The pandemic has had a profound impact on job markets globally, and many people are looking for new opportunities.
Firstly, I suggest revisiting your CV. Perhaps you could get a professional resume consultant or a trusted friend to take a look at it. They might provide some valuable feedback. Also, try tailoring your cover letters and CVs to match the requirements of each job. This could potentially increase your success rate.
Secondly, try broadening your job search channels. In addition to traditional job websites, consider participating in industry-related online events or forums. This could not only expand your network but also possibly uncover some hidden job opportunities.
Lastly, regarding your job gap, you can explain in interviews that it was due to the pandemic. I believe most employers would understand. Meanwhile, you could use this time to upskill or do some volunteer work, so your CV won't look empty.
Remember, everyone faces setbacks on their job-seeking journey. The key is to maintain a positive attitude and keep learning and improving. Good luck!
No job will be completely replaced. However, with the advent of AI, many jobs have become easier, and therefore the number of positions has decreased, such as designers.
scale is becoming more and more important, not just for cost, but also as a key technology feature to help deal with unexpected traffic and reduce the cost of manual operations.
Scaling Kafka has always been a headache for me, but seeing AutoMQ storing data entirely on S3, making Brokers stateless, no longer needing to migrate data for Kafka scaling. Awesome.