Good question. I wouldn’t say this replaces Flink in general. If you already run Flink and are comfortable with it, it’s a very powerful system.
Where we saw friction with Flink was mainly:
1.) Operational overhead (jobs, state backends, checkpointing)
2.) Generic sinks not being optimized for ClickHouse (batching, small inserts, etc.)
We focused on making scaling a property of the pipeline itself (just add replicas) and optimizing specifically for ClickHouse ingestion patterns.
So Flink is more general, this is more opinionated and focused on this specific use case.
I am seeing this pattern a lot lately. Teams start with a simple flow:
logs/metrics → Vector → ClickHouse
Works well as long as they run simple transformations via Vector. When they start adding things like dedupe, longer time windows, more data volume or joins, things start to break. They actually start using Vector as a stream processing engine.
Very typical issue that I see:
Time window limits: By default vector handles windowing in-memory. So with a higher load, it becomes too heavy to run there.
Missing support: When running in prod env, I have seen teams under pressure because there is no support available (except for Datadog customers). But most people I know run it self-hosted.
Scaling hits ceiling: I keep hearing similar numbers: 250k to 300k rec/sec per instance. Even by adding more resources, things do not scale. The consequences are: backpressure, latency spikes, etc.
At that point, it is no longer a “log pipeline.” It is a streaming system. Just not treated like one.
I wrote a deeper breakdown of this here if anyone’s curious:
This is interesting. Just wondering about your traffic volume and how long you have been running lcoalpdf?
For us, it is more like 5% of the traffic from GEO, but we have been running the company for 2 years and have created a lot of handwritten content for devs.
Where we saw friction with Flink was mainly: 1.) Operational overhead (jobs, state backends, checkpointing) 2.) Generic sinks not being optimized for ClickHouse (batching, small inserts, etc.)
We focused on making scaling a property of the pipeline itself (just add replicas) and optimizing specifically for ClickHouse ingestion patterns.
So Flink is more general, this is more opinionated and focused on this specific use case.