Building real-time ETL pipelines with Apache Kafka(datacater.io)
datacater.io
Building real-time ETL pipelines with Apache Kafka
https://datacater.io/blog/2022-02-11/etl-pipeline-with-apache-kafka.html
10 comments
basically an ad.
I wouldn't say that. Kafka is everywhere in the enterprise and it's adoption is still growing rapidly. This blog post is a decent overview for non-technical people at the very least.
agree
HN used to be strict about its "no self promotion" and "it must be interesting" rules. Dang probably got tired of constantly fighting about it. Now everyone with a blog treats HN like a dumping ground.
Edit: note that this is the CEO of the site. His account was inactive since 2012, and then in the past 7 months started dumping an ad every month here.
Edit: note that this is the CEO of the site. His account was inactive since 2012, and then in the past 7 months started dumping an ad every month here.
HN adblocker when?
It might be ad like, but there’s a pretty handy diagram explaining Kafka to beginners in there.
I’ve been working to grok Kafka the past little while and that diagram, if accurate stands out from most of the blabber on extended YouTube videos going for ad revenue.
I’ve been working to grok Kafka the past little while and that diagram, if accurate stands out from most of the blabber on extended YouTube videos going for ad revenue.
I always recommend the oreilly book 'kafka the definitive guide'. https://kafka.apache.org/books-and-papers
Thank you! Have so many oreillly books not sure why I didn’t think of it :)
ETL on streams? To be pedantic, does that even make sense? Extract, transform and load... a batch of data. But a stream?
I suggest to try thinking in terms of events instead of streams.
You can extract data change events (e.g., INSERTs) from a data source, transform them with a streaming application (e.g., built with Kafka Streams), and load the transformed events into a data sink.
You can extract data change events (e.g., INSERTs) from a data source, transform them with a streaming application (e.g., built with Kafka Streams), and load the transformed events into a data sink.