I am seeing this pattern a lot lately. Teams start with a simple flow:
logs/metrics → Vector → ClickHouse
Works well as long as they run simple transformations via Vector. When they start adding things like dedupe, longer time windows, more data volume or joins, things start to break. They actually start using Vector as a stream processing engine.
Very typical issue that I see:
Time window limits: By default vector handles windowing in-memory. So with a higher load, it becomes too heavy to run there.
Missing support: When running in prod env, I have seen teams under pressure because there is no support available (except for Datadog customers). But most people I know run it self-hosted.
Scaling hits ceiling: I keep hearing similar numbers: 250k to 300k rec/sec per instance. Even by adding more resources, things do not scale. The consequences are: backpressure, latency spikes, etc.
At that point, it is no longer a “log pipeline.” It is a streaming system. Just not treated like one.
I wrote a deeper breakdown of this here if anyone’s curious:
logs/metrics → Vector → ClickHouse
Works well as long as they run simple transformations via Vector. When they start adding things like dedupe, longer time windows, more data volume or joins, things start to break. They actually start using Vector as a stream processing engine.
Very typical issue that I see:
Time window limits: By default vector handles windowing in-memory. So with a higher load, it becomes too heavy to run there.
Missing support: When running in prod env, I have seen teams under pressure because there is no support available (except for Datadog customers). But most people I know run it self-hosted.
Scaling hits ceiling: I keep hearing similar numbers: 250k to 300k rec/sec per instance. Even by adding more resources, things do not scale. The consequences are: backpressure, latency spikes, etc.
At that point, it is no longer a “log pipeline.” It is a streaming system. Just not treated like one.
I wrote a deeper breakdown of this here if anyone’s curious:
https://www.glassflow.dev/blog/when-vector-becomes-your-stre...
Curious how people here are handling this.
Are you still pushing more logic into Vector, or have you split it out elsewhere?