This part of the text caught my attention the most.
"There are a lot of tools out there (Gramarly, Antidote for naming the most famous) and I did not see someone mentioning he used this or that."
I was criticized in another thread because I used a translation assistant to improve my text, a tool that, long before the current AI hype, everyone used to write more effectively.
People need to stop believing that the watchdogs of reason are the all-seeing eye(1989). Many people, in general, seek to be ethical and utilize tools to enhance their ideas (such as a text in a non-native language), and that's okay.
No, man, this wasn't done by an LLM. I actually do Kafka implementations and follow exactly the same script I described.
Perhaps the fact that I'm not a native English speaker may have caused this confusion. I just made sure my text was written correctly in English with a translator.
In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational challenges.
Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:
Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.
Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.
Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.
The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.
From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.
Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.
You don't use Kafka just to move a large amount of data between systems. You can also make use of Kafka to decouple systems, make integration easier, and let distributed computing technology widely used by big companies like Facebook, Uber, Tinder, Via Varejo (Brazil), Banco Itau (Brazil), Banco Santander (Brazil), be used to act in what she is a specialist.
The big problem is that the learning curve of the entire ecosystem is high, many people do not have the minimum semantic requirements that a distributed computing strategy requires and do not even want to study and understand, and thus generate this discomfort, and some statements erroneous that it is a cannon to kill a fly.
Right now, I am using Apache Kafka as an integration strategy between systems to integrate large factories spread out geographically, there was a high curve for everyone involved (infrastructure, Developers, and Architects) to understand the ecosystem as a whole (Kafka Connect, Kafka Broker, Zookeeper, Streams strategy, MirrorMaker 2) we built the entire integration by moving files in just 45 days due to an emergency imposed by government regulations.
We have almost 35 thousand messages a day (far from being high volume) being transported between different locations, in a stable manner, and with the delivery time (milliseconds) necessary for the demand. And now when we need to integrate a new factory that was bought by the company, we can do this integration in a short time because the entire base (the core) of the integration has been consolidated.
Everything is a matter of strategy and the correct use of the solution.
Built a data collection pipeline of COVID-19 cough recordings through our website (opensigma.mit.edu) between April and May 2020 and created the largest audio COVID-19 cough balanced dataset reported to date with 5,320 subjects.
We have the same problem here in Brazil in São Lourenço city, near to São Paulo. Nestlé has acquired a large aquatic park and now sale the water and accelerate the process to extract water and with this eliminate the natural minerals proprieties on the water killing all the ecosystem.
"There are a lot of tools out there (Gramarly, Antidote for naming the most famous) and I did not see someone mentioning he used this or that."
I was criticized in another thread because I used a translation assistant to improve my text, a tool that, long before the current AI hype, everyone used to write more effectively.
People need to stop believing that the watchdogs of reason are the all-seeing eye(1989). Many people, in general, seek to be ethical and utilize tools to enhance their ideas (such as a text in a non-native language), and that's okay.