Iggy.rs – building message streaming in Rust(blog.iggy.rs)
blog.iggy.rs
Iggy.rs – building message streaming in Rust
https://blog.iggy.rs/posts/building-message-streaming-in-rust/
62 comments
It seems like such a sharp, smart move to start with QUIC.
It provides really useful SCTP-like multi-streaming, which is a great base to start from. There's already so many good libraries the project or others can work from, and it'll just keep getting better (and better optimized, likely). https://github.com/xileteam/awesome-quic?tab=readme-ov-file#...
This seems like such a natural place where of course having a somewhat better transport protocol than we've had is a huge win. Looking forward to the next decade of QUIC.
It provides really useful SCTP-like multi-streaming, which is a great base to start from. There's already so many good libraries the project or others can work from, and it'll just keep getting better (and better optimized, likely). https://github.com/xileteam/awesome-quic?tab=readme-ov-file#...
This seems like such a natural place where of course having a somewhat better transport protocol than we've had is a huge win. Looking forward to the next decade of QUIC.
I've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when compared to Linux.
Is TCP + TLS still faster than QUIC, or do you mean TCP without encryption? (I'm assuming you're using Quinn with rustls, so encryption is always enabled.)
I'm also curious about your macOS vs Linux numbers. As a Quinn maintainer, we're always happy to hear feedback including about our performance!
I'm also curious about your macOS vs Linux numbers. As a Quinn maintainer, we're always happy to hear feedback including about our performance!
Right, I meant TCP without TLS, and when it comes to QUIC I only did benchmark using the "dangerous" mode (disabled local cert validation). I don't have the exact numbers, but AFAIR on MacOS raw TCP was a few times faster than QUIC, while on Linux (e.g. PopOS distro) TCP was maybe around 20-30% faster?
Still, there are plenty of options to be configured in Quinn (buffer size, send/receive window etc.), so it might be that with some tweaking the performance would be similar or QUIC would be even faster than TCP - this is also one of the things that I'd like to spend more time with in the future. We expose most of these options in the server configuration file, so it's easy to adjust.
Thank you for contributing to such a great library, really easy to work with :)
Still, there are plenty of options to be configured in Quinn (buffer size, send/receive window etc.), so it might be that with some tweaking the performance would be similar or QUIC would be even faster than TCP - this is also one of the things that I'd like to spend more time with in the future. We expose most of these options in the server configuration file, so it's easy to adjust.
Thank you for contributing to such a great library, really easy to work with :)
Looks like a direct competitor to JetStream [1]? Impressive progress in less than a year of work!
--
1: https://docs.nats.io/nats-concepts/jetstream
--
1: https://docs.nats.io/nats-concepts/jetstream
JetStream, Kafka, Redpanda, RabbitMQ Streams, Fluvio - quite a few message streaming solutions out there. Thanks!
I'm not quite sure how this compares to Kafka and fluvio [1], a Kafka competitor also written in Rust?
Is it more of a message queue like rabbitmq?
[1] https://www.fluvio.io/
Is it more of a message queue like rabbitmq?
[1] https://www.fluvio.io/
It's a message stream, so more like Kafka, Redpanda or RabbitMQ Streams (plugin). Fluvio is more mature (it's an actual product & company behind it), but we have our own ideas how to make Iggy a competitive message streaming solution :)
Doesn't Fluvio aim to replace both Flink and Kafka? I just heard about it so I am trying to understand.
I lead product at Fluvio.
If I remember correctly, I had seen iggy.rs in one of the recent applicant resumes to one of our Senior Rust Developer roles. Iggy seems like a cool project. All the best for the future of Iggy.
Fluvio has been in open source development for nearly 5 years now. The team has written hundreds of thousands of lines of code.
Fluvio core is a complete distributed streaming system for unbounded stream processing that is reliable, scalable, self healing. It’s the Kafka bit.
The Flink bit is our Stateful Service Development Kit which is currently in Dev Preview. Stateful Service Development Kit offers a scaffolding to compose complex chained event driven data pipelines.
Once we launch the stateful materialization bit, we will have a complete system for building end to end streaming data flows.
If I remember correctly, I had seen iggy.rs in one of the recent applicant resumes to one of our Senior Rust Developer roles. Iggy seems like a cool project. All the best for the future of Iggy.
Fluvio has been in open source development for nearly 5 years now. The team has written hundreds of thousands of lines of code.
Fluvio core is a complete distributed streaming system for unbounded stream processing that is reliable, scalable, self healing. It’s the Kafka bit.
The Flink bit is our Stateful Service Development Kit which is currently in Dev Preview. Stateful Service Development Kit offers a scaffolding to compose complex chained event driven data pipelines.
Once we launch the stateful materialization bit, we will have a complete system for building end to end streaming data flows.
Built something similar in Go with a friend a few years ago.
https://github.com/thibauts/styx
https://github.com/thibauts/styx
Nice one, looks quite similar indeed! May I ask why did you discontinue the further work?
Lack of traction basically. We ran out of time and money. Also we, maybe wrongly, did not feel like we could compete with the big ones without clustering. Yet we had a lot of fun building it. It was a thrill to see our product deliver performance on par or above in-memory messaging systems while being fully persisted and fsync’ed. I’m sure you know what I mean ! Good luck with Iggy anyway. Looks like you are on the right track !
Probably because there's a very low value in building message streaming framework once you've done the basics: how do you come to, say, a trading broker and say "your order management system could be just as fast with my other messaging framework".
He did it to learn and have fun, but finishing it to be prod-ready would require at least one user.
He did it to learn and have fun, but finishing it to be prod-ready would require at least one user.
That was indeed a big parameter. We missed someone on the team able to connect to us to one or two large early adopters. We learned that building a good product and building a company selling it are two very different things !
Would love to give it a go at some point (but I'll first need to learn Rust). On a side note, I love the aesthetic of the site.
Exactly, we have multiple SDKs, and the blog uses Rust Zola engine :)
I'm still trying to learn Zola well. Do you plan to open-source the website?
If you're talking about the blog, it's an open source Zola theme https://github.com/not-matthias/apollo (the original site https://secret.club which it's based on/inspired by is closed source though)
No, I really need to see how did he structure his website exactly -- to learn how I would do it as well.
I'll take a look at the theme as well, thank you.
I'll take a look at the theme as well, thank you.
The blog post mentions SDKs for other programming languages, so you may be able to use this without learning Rust
This post prompted me to resurface the Genesis of Fluvio. We are a small team going at it for a little while since we have long relationship with the data centric applications across multiple domains for the past decades and we are excited about data streaming on rust and web assembly instead of java and jvm.
Here is a blog that is from June 2021 from our CTO laying out the vision for Fluvio.
https://news.ycombinator.com/item?id=38880743
Since the comparison question keeps popping up, I would share whatever we have as well. It's a long drawn documentation exercise and I am happy to share what we have at Fluvio.
Again - Great job with Iggy!
Here is a blog that is from June 2021 from our CTO laying out the vision for Fluvio.
https://news.ycombinator.com/item?id=38880743
Since the comparison question keeps popping up, I would share whatever we have as well. It's a long drawn documentation exercise and I am happy to share what we have at Fluvio.
Again - Great job with Iggy!
Really cool idea and project! There are 2 things I would need to understand before trying it out though:
1. How can I run more than one instance of the server?
2. When running more than one instance of the server, how would the filesystem interaction work between servers?
1. How can I run more than one instance of the server?
2. When running more than one instance of the server, how would the filesystem interaction work between servers?
I'm surprised about the selection of monoio. AFAIK it requires the usage of the nightly compiler which I've found to not be a great choice when maintaining a project.
It does require nightly, but it only uses five features, one of which could be removed by adding an external crate, and the others are like, not anything super wild, for the most part. I haven't looked too deeply into the code but they all make sense to me, for example one of them is a standard library API for creating uninitialized containers, which allows you to eliminate a copy.
I have not compared monoio with glommio, which does run on stable. Would be interesting!
I have not compared monoio with glommio, which does run on stable. Would be interesting!
It’s not that the features are unreasonable but more that anything that uses it as a dependency also inherits the nightly requirement which makes things a bit more unstable because nightly is a floating version.
I did compare with glommio a bit and disliked the file API in monoio but that’s probably more a matter of taste.
I did compare with glommio a bit and disliked the file API in monoio but that’s probably more a matter of taste.
Yeah I hear you. I much prefer staying on stable for that reason. Sometimes (though much more rarely than non-Rust users seem to think) it is unavoidable though, and once you do go into nightly, you can kinda go two ways:
* only use what is absolutely necessary
* go wild and add tons of stuff
As well as there being two kinds of nightly features:
* stuff that's not stable but has a clear path to becoming stable on some time frame
* stuff that's not stable but who knows when that will change, it may never
And to me, this dependency feels kinda closer to choosing the first option from both lists, which is the most conservative way of using nightly.
* only use what is absolutely necessary
* go wild and add tons of stuff
As well as there being two kinds of nightly features:
* stuff that's not stable but has a clear path to becoming stable on some time frame
* stuff that's not stable but who knows when that will change, it may never
And to me, this dependency feels kinda closer to choosing the first option from both lists, which is the most conservative way of using nightly.
As already mentioned, we've decived to use monoio, just to have something to begin with, it will take at least a few months to rewrite the core parts to make use of io_uring and thread-per-core approach (if feasible), so staying on the bleeding edge is more like a sandbox for us - we might decide to go with another runtime, like glommio, mfio or something else.
Ah cool! Just to be clear, I think you should do whatever you feel is best for your project. I don't see this in any sort of moral terms, purely engineering ones, and there are good reasons to use nightly just like there are good reasons to use stable.
Monoio seems to be the most performant runtime, and actually easy to use - we have decided to go with "bleeding edge" approach, as it will still take at least a few months to implement io_uring and other optimizations, as we'll have to rewrite some of the core parts and most likely shift towards thread-per-core architecture.
For what it’s worth I don’t believe it meaningfully outperforms glommio. There are some spots where it outperforms glommio but that’s just a missing optimization that could be added. The biggest advantage monoio really has is that it currently has support for other platforms.
Not saying it’s the wrong choice, just highlighting the trade offs.
I’m not sure what you mean by io_uring as glommio/monoio should be hiding the io_uring details behind the runtime.
Not saying it’s the wrong choice, just highlighting the trade offs.
I’m not sure what you mean by io_uring as glommio/monoio should be hiding the io_uring details behind the runtime.
Of course, they hide the details behind io_uring, it's just that monoio seemed even easier to work with. It's not set in stone, though, actually there are new runtimes being developed as we speak, for example mfio - we'll see what will be our final choice, but we've decided to start with monoio.
Speaking of the other platforms, having the fallback to epoll or kqueue (they've even announced some Windows integration) is nice to have, however, at some point we might just purely focus on Linux development anyway (io_uring only), if there'd be any issues e.g. when it comes to the code design to provide the compatibility across multiple OS.
Speaking of the other platforms, having the fallback to epoll or kqueue (they've even announced some Windows integration) is nice to have, however, at some point we might just purely focus on Linux development anyway (io_uring only), if there'd be any issues e.g. when it comes to the code design to provide the compatibility across multiple OS.
How would this stack up to the likes of KeyDB’s streaming? More durable persistence is my take.
Is this project kind of an amalgamation of Kafka and something like Flight/nats?
Can't say for sure, we share some of the same concepts that can be found in Kafka or RabbitMQ Streams, but at the same time, try to build something from the ground up and focus on low-level I/O improvements :)
i'm a bit confused as to what exactly this is; a low-level, messaging system? what are the implications of it and the comparison of competitors? I'm obviously missing something. Could someone elaborate?
This is a message streaming infrastructure, similar to Kafka, Redpadna etc. You can think of it as a kind of a simple database (so-called Write Ahead Log), being responsible for appending the messages in a highly performant manner.
Will there be any public servers like xmpp? So that people can use it like normally who don't have expertise in hosting servers.
Iggy is a message streaming infrastructure (think of Kafka for example), not a messaging platform like Discord or so :)
How fast can one realistically expect to be able to build decent quality system-level rust code, provided the person is already familiar with C-style programming languages and has 0 rust experience ?
It's hard to judge what counts as "decent quality". However, much of the very low level understanding translates 1:1 from C to Rust. For example in term of the ordering rules to deliver a multi-threaded program with sequential consistency (which thus can be understood by humans) those are identical, both C and Rust "stole" the C++ 11 model.
For ABI Rust has an explicit way to say "Lay this out the way C does" which works for things that you could have done in C. How is everything else laid out? Not your business. So again, your C expertise applies.
Neither Rust nor C formally have agreed pointer provenance rules, if you've learned things that are promised to work in this domain in C most of them also work in Rust. Most things aren't promised to work in C, Aria's Strict Provenance experiment and associated APIs are more than you're going to get from ISO C any time soon.
Aliasing rules are stricter in Rust than C, but also I'd argue much simpler. So that seems not to be a huge amount of extra work.
For ABI Rust has an explicit way to say "Lay this out the way C does" which works for things that you could have done in C. How is everything else laid out? Not your business. So again, your C expertise applies.
Neither Rust nor C formally have agreed pointer provenance rules, if you've learned things that are promised to work in this domain in C most of them also work in Rust. Most things aren't promised to work in C, Aria's Strict Provenance experiment and associated APIs are more than you're going to get from ISO C any time soon.
Aliasing rules are stricter in Rust than C, but also I'd argue much simpler. So that seems not to be a huge amount of extra work.
It has gotten a little easier over time. IMO, a month or two depending on your aptitude.
If you have a little bit of experience with any higher level language (eg Python, Haskell), then I'd say fairly quicky (say, a week to a week and a half for the books and rustlings course), if you want an opinionated (and good enough, I recommend the book!) intro to production code theres the zero2prod Rust book. You can read the zero2prod book while working on one or two pet project (anything more complicated than a calculator would probably do) to help inform decisions. That should be another week or two, depending on how quickly you pace through the book.
That should get you started to writing decent code, maybe not aways idiomatic or neatly structured. That might take a few more weeks or months (depending on your interest domains and their inherent complexities). But like with all skills, it's an endless journey; you'll keep getting better as you're increasingly more familiar with the tools.
That should get you started to writing decent code, maybe not aways idiomatic or neatly structured. That might take a few more weeks or months (depending on your interest domains and their inherent complexities). But like with all skills, it's an endless journey; you'll keep getting better as you're increasingly more familiar with the tools.
[deleted]
The closest thing I have to a non-anecdotal answer to this question is from Google: https://opensource.googleblog.com/2023/06/rust-fact-vs-ficti...
> All survey participants are professional software developers (or a related field), employed at Google. While some of them had prior Rust experience (about 13%), most of them are coming from C/C++, Python, Java, Go, or Dart.
> Based on our studies, more than 2/3 of respondents are confident in contributing to a Rust codebase within two months or less when learning Rust. Further, a third of respondents become as productive using Rust as other languages in two months or less. Within four months, that number increased to over 50%. Anecdotally, these ramp-up numbers are in line with the time we’ve seen for developers to adopt other languages, both inside and outside of Google.
> Overall, we’ve seen no data to indicate that there is any productivity penalty for Rust relative to any other language these developers previously used at Google.
> All survey participants are professional software developers (or a related field), employed at Google. While some of them had prior Rust experience (about 13%), most of them are coming from C/C++, Python, Java, Go, or Dart.
> Based on our studies, more than 2/3 of respondents are confident in contributing to a Rust codebase within two months or less when learning Rust. Further, a third of respondents become as productive using Rust as other languages in two months or less. Within four months, that number increased to over 50%. Anecdotally, these ramp-up numbers are in line with the time we’ve seen for developers to adopt other languages, both inside and outside of Google.
> Overall, we’ve seen no data to indicate that there is any productivity penalty for Rust relative to any other language these developers previously used at Google.
I had 0 Rust experience and close to 0 of system programming experience (except playing with some lower level communication protocols etc. but nothing fancy), but I'd say that something like message streaming isn't the system programming - it's close to building the database, but nowhere close to making direct usage of low level system APIs.
Good luck with the project!
I also echo some of the others, it would be great to see a sort of comparison with the alternatives so that we can better understand how the project fits in.