This is cool. By the way I just tried it and ran a simple infinite loop (in Python) and the whole page goes unresponsive and I don't seem to have a way to terminate the process?
RQ is certainly useful (with finer control over messages), but we've been having a lot of problems with it lately in production due to it being memory bound (Redis). Improper code deploys or insufficient/stuck workers would quickly explode the queues and make the broker go oom in matter of hours. Memory is also very expensive. With KQ/Kafka I was hoping it would provide us with a lot more to buffer for human errors and scale better.
Thank you for the excellent feedback and insight. I will definitely give the pdf a read. I agree with all of your points, which admittedly I was not fully aware of when I first embarked on this project. As you've already implied, there are some nuances that may forever be inevitable due to the inherent design of Kafka. But I wouldn't want to dismiss it as unsuitable for job queues so early. It would depend strongly on the use case of course (e.g. jobs that are idempotent or without hard requirement to be processed), but as I am hoping that Kafka's API matures further with finer control over messages and that this could work fairly well for the most part. For now I will take note and update the documentation to clearly explain what KQ is (and what it is not), and what the best practises and use cases must be taken into account before using it. Thanks again!