Handling 100k consumers with one pulsar topic(streamnative.io)
streamnative.io
Handling 100k consumers with one pulsar topic
https://streamnative.io/blog/case/2023-01-10-handling-100k-consumers-with-one-pulsar-topic/
7 コメント
Interesting. That's the same path the article goes down. That is:
"Pulsar uses two individual tasks to complete the message-sending process - one for sending the message and the other for the ACK. For each message sent by the consumer, Pulsar generates these two tasks and pushes them to OrderedExecutor. To guarantee the order of messages, Pulsar always adds them to the same thread...As shown in Figure 12, Pulsar generates 200K tasks, all of which are inserted into a single thread."
It does, at the beginning, talk about other ways to address this, like "shadow topics". But it goes on to solve it instead with a combination of virtual cursors, and this:
"Therefore, to solve this problem, we used a random thread for ACK tasks instead of the same thread."
"Pulsar uses two individual tasks to complete the message-sending process - one for sending the message and the other for the ACK. For each message sent by the consumer, Pulsar generates these two tasks and pushes them to OrderedExecutor. To guarantee the order of messages, Pulsar always adds them to the same thread...As shown in Figure 12, Pulsar generates 200K tasks, all of which are inserted into a single thread."
It does, at the beginning, talk about other ways to address this, like "shadow topics". But it goes on to solve it instead with a combination of virtual cursors, and this:
"Therefore, to solve this problem, we used a random thread for ACK tasks instead of the same thread."
This looks like a blog post, but it's actually a writeup by a user/customer (aka case study or similar). Would recommend putting that into the post header somewhere, so as not to confuse readers who didn't navigate there by hand.
Any plans to release the broadcast capability upstream?
I asked the author the same thing. But they have not contributed it back to the community, but perhaps that's sth they will consider in the future.
Huh, I get a 404 on this.
It looks like the website has some problems. See this link: https://www.streamnative.io/blog/handling-100k-consumers-wit...
https://github.com/jgaa/pulsar-test-consumer-scale
I was a bit disappointed by the result. It's known that Pulsar shards it's topics so that one topic is handled by one broker - for subscribers and publishers. However, at least at that time, it appeared as the broker looped trough all it's subscribers in a single thread, effectively enforcing a linear limit on the number of subscribers it could handle on a topic.
I'm just mentioning it, because when I tested, I could definitely not handle 100k shared consumers.
(Eventually, I ended up writing a consumer-proxy that could scale the number of subscribers indefinitely, using just one or a few subscriber-connections to the broker. Unfortunately, it's not open source.)