Horizontally scaling Kafka consumers with rendezvous hashing(tinybird.co)
tinybird.co
Horizontally scaling Kafka consumers with rendezvous hashing
https://www.tinybird.co/blog-posts/kafka-horizontal-scaling
3 comments
The idea is nice but requires trusting their users a lot. How do they prevent users from setting super high "linker_workers" to be consumed by every agent? This could open the door for malicious users to saturate the entire system...
Hi! Alex here, Product Manager at Tinybird.
_peregrine_ is right, we do have control over the linker_workers value for every agent. Agents are not exposed to our users, everything is transparent for them.
As a user, you just need to create a new connection to your Kafka cluster providing your credentials, and then choose the topic you want to ingest into Tinybird. We take care of everything for your, we balance the worker using the approach explained in the blog post.
We can also fine tune it for specific users, or relevant events such as Black Friday. It's a bit different for Enterprise customers, in those cases we can set up dedicated agents and fine tune linker_workers and other parameters to optimize for their use case.
If you have any further questions you can reach us on our public Slack channel or via email.
Thanks!
_peregrine_ is right, we do have control over the linker_workers value for every agent. Agents are not exposed to our users, everything is transparent for them.
As a user, you just need to create a new connection to your Kafka cluster providing your credentials, and then choose the topic you want to ingest into Tinybird. We take care of everything for your, we balance the worker using the approach explained in the blog post.
We can also fine tune it for specific users, or relevant events such as Black Friday. It's a bit different for Enterprise customers, in those cases we can set up dedicated agents and fine tune linker_workers and other parameters to optimize for their use case.
If you have any further questions you can reach us on our public Slack channel or via email.
Thanks!
I'm pretty sure linker_workers is something that they configure, I don't think it's exposed to the user to configure.