Stream and Go: News feeds for 300M users, built on RocksDB and Raft(stackshare.io)
stackshare.io
Stream and Go: News feeds for 300M users, built on RocksDB and Raft
https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-million-end-users
65 comments
What issues did they run into with Cassandra? It's not easy to build a scalable database like Cassandra. There are good reasons to write your own distributed storage system but the authors need to add more details about Keevo and specific issues they ran into with Cassandra.
Main developer of Keevo at Stream here and you ask a very good question. I've asked myself the same question quite a lot. And I think it actually warrants it own dedicated blog post at some point. To give some idea though, the main reasons are, cost, simplicity, control and wanting to have a very good understanding of the database internals.
Cassandra is very scalable, but it's not very efficient. The hosting costs for our cassandra cluster were so big that it was infeasible to run it in another region as well.
Appart from that we've had (partial) downtime a couple of times because one node just started going crazy because of unclear reasons.
Keevo solves this by not trying to be cassandra, but be much simpler. It doesn't do schema's or indexes. All it is is a very fast ordered key value store that is stored to disk and replicated automatically to multiple servers (using raft). Any other features we need, we build on top of this usually outside of Keevo itself. This simplicity saves us a lot of hosting costs and makes its performance much more predictable and easier to debug.
Last but not least a very important advantage is control and understanding of the database internals. Because we build Keevo ourselves, we know the performance and consistency tradeoffs it has and can change/improve them when needed.
I hope this helps in understanding our choice. It's definitely not something I would recommend for most companies, but since our product is storage at its core it makes sense for us.
Cassandra is very scalable, but it's not very efficient. The hosting costs for our cassandra cluster were so big that it was infeasible to run it in another region as well.
Appart from that we've had (partial) downtime a couple of times because one node just started going crazy because of unclear reasons.
Keevo solves this by not trying to be cassandra, but be much simpler. It doesn't do schema's or indexes. All it is is a very fast ordered key value store that is stored to disk and replicated automatically to multiple servers (using raft). Any other features we need, we build on top of this usually outside of Keevo itself. This simplicity saves us a lot of hosting costs and makes its performance much more predictable and easier to debug.
Last but not least a very important advantage is control and understanding of the database internals. Because we build Keevo ourselves, we know the performance and consistency tradeoffs it has and can change/improve them when needed.
I hope this helps in understanding our choice. It's definitely not something I would recommend for most companies, but since our product is storage at its core it makes sense for us.
> Appart from that we've had (partial) downtime a couple of times because one node just started going crazy because of unclear reasons.
Do you have GC logs? GC lockups are the most common case. Have you used G1GC + Java8?
Do you have GC logs? GC lockups are the most common case. Have you used G1GC + Java8?
Thanks for sharing, super interesting stuff. I'd be curious to hear more about the design around availability in the case of a zone outage :D
I also think it's really interesting stuff and love working on it. I'll definitely write something about that if/when I do a blog post about it. But it's quite simple at its core. Like the current post mentions, we use Raft to do it. We simply have a cluster of 3 nodes, each in a different zone. If one zone goes down, there's still a majority of nodes up, so enough to keep the cluster running. I recommend reading the raft paper for more details, it's one of the easiest papers to read and understand I've ever found. https://raft.github.io/raft.pdf
> Keevo solves this by not trying to be cassandra, but be much simpler.
Sounds awfully similar to Riak. :)
Sounds awfully similar to Riak. :)
I have to say I'm not very familiar with Riak. A quick google makes it look similar in functionality. I'm not sure about speed though, RocksDB is really really fast and storage efficient. We tried out a lot of other embedded KV stores. Also Riak does seem to miss one important feature for us, iterating through the keys in lexical order. We use that a lot to build features on.
Even if this feature was/is available in Riak, I still think this was the better choice for us. Bringing this core component in house has been a real boon in gaining good and more importantly predictable performance for our API.
Even if this feature was/is available in Riak, I still think this was the better choice for us. Bringing this core component in house has been a real boon in gaining good and more importantly predictable performance for our API.
Iterating through keys is not trivial in Riak, you need a 2i to have that. If you have the manpower to maintain your solution and there are people with performance engineering skill it is probably good. Riak has very predictable performance and it is easy to tune and rock solid system this is why it is used in several healthcare systems and for the game that has the largest player base at the moment as well. The consistent hashing layer on the top of LevelDB which is essentially the foundation of RocksDB makes it super nice system. Anyways, I hope Keevo will be available for us as well.
The "you need 2i to have that" is too heavy.
See yugabyte that does cassandra+keevo+rocksdb+raft
See yugabyte that does cassandra+keevo+rocksdb+raft
Thanks for the comment and look forward to the dedicated blog post :)
Was Riak KV ever considered?
Was Riak KV ever considered?
Not as far as I know. We did benchmark a lot of embedded storages before deciding on Rocksdb though.
Wasn't riak, like, less efficient than cassandra ?
What do you mean? I think Riak is much simpler and after tuning it beats Cassandra on the same HW same use case in my anecdotal experience.
Well there are several issues with Cassandra:
- it is not ACID the worst possible way
"Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended."
"This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight transactions (LWT) are for, and that's what the authors of this piece used. However! Fundamentally, mixing serializable (LWT) and non-serializable (plain UPDATE) ops will produce unpredictable results and that's what bit them here.Basically the same as if you marked half the accesses to a concurrently-updated Java variable with "synchronized" and left it off of the other half as an "optimization."Don't take shortcuts and you won't get burned."
- there are several things that needs to be tuning (like GC) that is not trivial to do
- modeling is challenging say the least, easy to create hotspots that the developers are not aware of
- it is not ACID the worst possible way
"Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended."
"This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight transactions (LWT) are for, and that's what the authors of this piece used. However! Fundamentally, mixing serializable (LWT) and non-serializable (plain UPDATE) ops will produce unpredictable results and that's what bit them here.Basically the same as if you marked half the accesses to a concurrently-updated Java variable with "synchronized" and left it off of the other half as an "optimization."Don't take shortcuts and you won't get burned."
- there are several things that needs to be tuning (like GC) that is not trivial to do
- modeling is challenging say the least, easy to create hotspots that the developers are not aware of
"it is not ACID the worst possible way" Vow what a great revelation.
"Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." This is not true. Cassandra is row level atomic but I guess it also depends on the version you use maybe. Can you talk about which version and provide a test that satisfies your assertion ?
"there are several things that needs to be tuning (like GC) that is not trivial to do". Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead.
I guess these days people can make up whatever they want without providing any valid tests that prove their assertions. It all comes down to is either love & hate of a programming language or someone wants to put some fancy sounding tools in their resume!!
"Cassandra is not row level consistent,[21] meaning that inserts and updates into the table that affect the same row that are processed at approximately the same time may affect the non-key columns in inconsistent ways. One update may affect one column while another affects the other, resulting in sets of values within the row that were never specified or intended." This is not true. Cassandra is row level atomic but I guess it also depends on the version you use maybe. Can you talk about which version and provide a test that satisfies your assertion ?
"there are several things that needs to be tuning (like GC) that is not trivial to do". Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead.
I guess these days people can make up whatever they want without providing any valid tests that prove their assertions. It all comes down to is either love & hate of a programming language or someone wants to put some fancy sounding tools in their resume!!
> This is not true. Cassandra is row level atomic
You can go ahead and try to convince people who experienced this in production.
http://datanerds.io/post/cassandra-no-row-consistency/
I could provide you all of the versions but it is irrelevant from the angle that the version my clients have in production are affected.
> Do you know that Go's GC is way less advanced than Java GC as of today?
Not only I know, I work as a consultant who actually configures it so that it provides the best performance for the workload a customer has. I made a large sum by changing the settings that most Cassandra installations have. In fact, all of my clients were running Cassandra with the default settings in production and failed miserably to have a stable service. In one occasion there was almost 1 minute GC time on average, causing nodes marked as down, read and write tied up as well. Facebook and Netflix has production engineers with these skills so they can make it happen for them but clients not having such engineering resources are either reaching out consultancies or try to hire somebody with the skills. What does your question have to do with the topic?
> I guess these days people can make up whatever they want without providing any valid tests that prove their assertions.
I guess these days people can go on HN and write down some of their experiences that they encountered in their professional life.
You can go ahead and try to convince people who experienced this in production.
http://datanerds.io/post/cassandra-no-row-consistency/
I could provide you all of the versions but it is irrelevant from the angle that the version my clients have in production are affected.
> Do you know that Go's GC is way less advanced than Java GC as of today?
Not only I know, I work as a consultant who actually configures it so that it provides the best performance for the workload a customer has. I made a large sum by changing the settings that most Cassandra installations have. In fact, all of my clients were running Cassandra with the default settings in production and failed miserably to have a stable service. In one occasion there was almost 1 minute GC time on average, causing nodes marked as down, read and write tied up as well. Facebook and Netflix has production engineers with these skills so they can make it happen for them but clients not having such engineering resources are either reaching out consultancies or try to hire somebody with the skills. What does your question have to do with the topic?
> I guess these days people can make up whatever they want without providing any valid tests that prove their assertions.
I guess these days people can go on HN and write down some of their experiences that they encountered in their professional life.
> Do you know that Go's GC is way less advanced than Java GC as of today? This is admitted by Google's Go team lead.
I won't comment on the other points - but I managed a medium sized Cassandra cluster for a couple of years, and the GC point is valid. It has nothing to do with Java's GC being more advanced. It's easier to bypass the Go GC with stack allocations (not possible in Java), and many of Cassandra's processes (compaction, repair) end up being very GC heavy. GC tuning ends up being a function of your workload and if you ignore it, background processes like repair can adversely affect projection nodes, or throw nodes in a loop. - and adversely giving the node more memory can make things worse. Cassandra has left a bad taste in my mouth for Java-based databases.
I won't comment on the other points - but I managed a medium sized Cassandra cluster for a couple of years, and the GC point is valid. It has nothing to do with Java's GC being more advanced. It's easier to bypass the Go GC with stack allocations (not possible in Java), and many of Cassandra's processes (compaction, repair) end up being very GC heavy. GC tuning ends up being a function of your workload and if you ignore it, background processes like repair can adversely affect projection nodes, or throw nodes in a loop. - and adversely giving the node more memory can make things worse. Cassandra has left a bad taste in my mouth for Java-based databases.
Yeah I can show you one of the systems:
GC of death:
https://imgur.com/a/6RAX7
Write latency:
https://imgur.com/a/Us0MO
Typical Cassandra user has these sort of problems. I have one client where they identified the issue (data modeling) and fixed it by redesigning their tables before I got there. Some of the Cassandra users are not even aware. The company where the pictures are from engaged me because the system did not meet with business requirements anymore. Could not insert data into the cluster, their ETL jobs were running for 20 hours and if one failed they did not have data for their business. We could speed it up to run it for 4 hours without remodeling the data. With remodeling it Cassandra was not the bottleneck anymore.
GC of death:
https://imgur.com/a/6RAX7
Write latency:
https://imgur.com/a/Us0MO
Typical Cassandra user has these sort of problems. I have one client where they identified the issue (data modeling) and fixed it by redesigning their tables before I got there. Some of the Cassandra users are not even aware. The company where the pictures are from engaged me because the system did not meet with business requirements anymore. Could not insert data into the cluster, their ETL jobs were running for 20 hours and if one failed they did not have data for their business. We could speed it up to run it for 4 hours without remodeling the data. With remodeling it Cassandra was not the bottleneck anymore.
These graphs are meaningless in a vacuum, you don't know how would behave alternative Go based solution on similar hardware under the similar load.
Sure, this is why there is a production system surrounding them. I am not really interested in theoretical possibilities just observing reality here.
What I am saying is that you are observing one side of reality. It is very possible that even with this kind of graphs, Cassandra is capable to handle much larger traffic then custom Go solution.
Cassandra is usually competitive in open benchmarks.
Cassandra is usually competitive in open benchmarks.
> stack allocations (not possible in Java)
JVM does stack allocation whenever it is possible, you can learn about this by googling "escape analysis".
JVM does stack allocation whenever it is possible, you can learn about this by googling "escape analysis".
> This is not true. Cassandra is row level atomic
http://cassandra.apache.org/doc/latest/faq/index.html#what-h...
"What happens if two updates are made with the same timestamp? Updates must be commutative, since they may arrive in different orders on different replicas. As long as Cassandra has a deterministic way to pick the winner (in a timestamp tie), the one selected is as valid as any other, and the specifics should be treated as an implementation detail. That said, in the case of a timestamp tie, Cassandra follows two rules: first, deletes take precedence over inserts/updates. Second, if there are two updates, the one with the lexically larger value is selected."
Does this sound to you as atomic? My expectation of atomic is that 2 things changing the same value are ordered and both executed. "lexically larger value is selected" does not qualify for me as atomic.
http://cassandra.apache.org/doc/latest/faq/index.html#what-h...
"What happens if two updates are made with the same timestamp? Updates must be commutative, since they may arrive in different orders on different replicas. As long as Cassandra has a deterministic way to pick the winner (in a timestamp tie), the one selected is as valid as any other, and the specifics should be treated as an implementation detail. That said, in the case of a timestamp tie, Cassandra follows two rules: first, deletes take precedence over inserts/updates. Second, if there are two updates, the one with the lexically larger value is selected."
Does this sound to you as atomic? My expectation of atomic is that 2 things changing the same value are ordered and both executed. "lexically larger value is selected" does not qualify for me as atomic.
not cool enough to be used as clickbait any more.
@tschellenbach: congrats on your achievements!
This is yet another Python success story. You created a viable product with Python and moved critical parts to Go when there was real need for doing so.
Considering Go has only been your primary language for 9 months, as the beginner's style code base is optimized for all that Go has to offer you'll continue to see more performance gains, albeit probably not as significant as you saw between v1 and v2. Good on you for not being tied down with optimizations and actually releasing.
"With Python we often found ourselves delegating logic to the database layer purely for performance reasons. " -- you mean you wrote postgres functions.. or.. views!? gasp! heresy! Why would you migrate something working perfectly well in postgres away to Go, though? You probably didn't achieve any performance gains with that. Were postgres functions really gobbling that much memory up? Or.. maybe my assumptions are wrong here.
This is yet another Python success story. You created a viable product with Python and moved critical parts to Go when there was real need for doing so.
Considering Go has only been your primary language for 9 months, as the beginner's style code base is optimized for all that Go has to offer you'll continue to see more performance gains, albeit probably not as significant as you saw between v1 and v2. Good on you for not being tied down with optimizations and actually releasing.
"With Python we often found ourselves delegating logic to the database layer purely for performance reasons. " -- you mean you wrote postgres functions.. or.. views!? gasp! heresy! Why would you migrate something working perfectly well in postgres away to Go, though? You probably didn't achieve any performance gains with that. Were postgres functions really gobbling that much memory up? Or.. maybe my assumptions are wrong here.
I wonder about this sometimes. Keen, Sendgrid, Stream all started out with Python and switched later on. Algolia started out in C++. In retrospect I think Algolia made a better choice by starting out with a fast language. Yes we saved some time getting the prototype of the ground. On the other hand it made performance optimization more difficult. Both are amazing languages. I think it makes sense to learn and use both. They complement each other very well.
> Keen, Sendgrid, Stream all started out with Python and switched later on.
SendGrid engineer here.
IIRC, each of the three founders wrote components in the language that they were most productive in at the time. So there were services in Python, Perl, or PHP depending on who wrote it. Seems like a solid strategy to take the fastest path to MVP when you're still trying to prove there's a market for what you're building.
As the company grew, Python emerged as the most commonly chosen language for new projects.
As you said, the initial appeal of Go was its performance. An occasional side-effect of "fast" is needing less hardware to do the same work, so its efficiency was also attractive.
I think a key moment that got us interested in Go came at an internal hackathon. One of our engineers built a prototype replacement for a core piece of our MTA. It was something like 6x faster than the existing Python version. The ability to produce that in such a short period of time demonstrated that Go was also a surprisingly productive language.
These days, most of our engineering teams choose Go for new backend services -- even those that aren't performance-sensitive. However, we still have a lot of active development in Python (and Ruby).
Disclaimer: I've been at SendGrid about 4 years, so some of this is second-hand. I'll ask the founders if they're up for doing an article on our engineering blog about it.
SendGrid engineer here.
IIRC, each of the three founders wrote components in the language that they were most productive in at the time. So there were services in Python, Perl, or PHP depending on who wrote it. Seems like a solid strategy to take the fastest path to MVP when you're still trying to prove there's a market for what you're building.
As the company grew, Python emerged as the most commonly chosen language for new projects.
As you said, the initial appeal of Go was its performance. An occasional side-effect of "fast" is needing less hardware to do the same work, so its efficiency was also attractive.
I think a key moment that got us interested in Go came at an internal hackathon. One of our engineers built a prototype replacement for a core piece of our MTA. It was something like 6x faster than the existing Python version. The ability to produce that in such a short period of time demonstrated that Go was also a surprisingly productive language.
These days, most of our engineering teams choose Go for new backend services -- even those that aren't performance-sensitive. However, we still have a lot of active development in Python (and Ruby).
Disclaimer: I've been at SendGrid about 4 years, so some of this is second-hand. I'll ask the founders if they're up for doing an article on our engineering blog about it.
[deleted]
I actually think that's not a "Python" or "Go" success story. It's a story of how both of them lack something significant: Python lacks speed and light memory usage, Go lacks dev speed and expresiveness. There is no reason why a single modern language can't combine them, and that would be much better than managing a codebase in two languages, so it's a success story of a useful compromise
The article says they saw improved productivity after switching to Go, how does that point to Go lacking dev speed or expressiveness?
I am disagreeing with the parent's point. His point is that Python is better for prototyping which implies that in the beginning people can be quite more productive with it.
I write Python in my day job, and I also experience a velocity boost by using Go. It's easy like Python, but the tooling is better. I really find myself appreciating the static type system catching errors at compile time instead of runtime. And the performance improvements I've observed for the systems I've migrated to Go is 50-300X.
personal experience, as someone who's written a lot of python in the past and writes a lot of go now: go is more productive for me than python.
meh, it depends on the problem domain. it's quite nice for its niche, but I don't like it for general use
I picked up python and Go at the same time. Go is much easier, logical, faster, uses less memory, and has common-sense stuff like unicode handling built-in.
I only use Python for the same reason I use Java: all the NLP / ML frameworks built by people who don't know Go yet.
I only use Python for the same reason I use Java: all the NLP / ML frameworks built by people who don't know Go yet.
Go is nice for its niche, and it can be very cool after using Python. Still it feels too simplistic and limited as a language to me, which includes the "low level" part: it's not "optimally" fast and it's not very easy to interop with other languages
The title contains "rocksdb and raft", but the article doesn't give any details on how the Keevo key-value store is designed using rocksdb + raft. You can not figure out how they manage data with raft (how many raft groups in the system? do they split the group when it reaches certain stage?), there is no mention on the design choice of not using etcd raft which is the most popular one in Go. They didn't mention how they test the system, Jepsen or in-house monkey testing? They do have a Keevo link in the article but it says "Sorry, we couldn't find that page".
On the Go side, since they mentioned that rocksdb is written in C++, I am interested to know the cgo issues they might had when using Go and rocksdb together, but of course they didn't mention that. Same goes for the GC stw pauses.
I strongly believe that articles like this largely used as soft ads materials without much detailed technical meat should be banned from Hack News. If it is all about "we built system X using building blocks Y and Z in language T!", what is the real value for readers?
On the Go side, since they mentioned that rocksdb is written in C++, I am interested to know the cgo issues they might had when using Go and rocksdb together, but of course they didn't mention that. Same goes for the GC stw pauses.
I strongly believe that articles like this largely used as soft ads materials without much detailed technical meat should be banned from Hack News. If it is all about "we built system X using building blocks Y and Z in language T!", what is the real value for readers?
We currently use the Raft Go implementation from Hashicorp (the article links to the Github project). We found Hashicorp implementation mature enough and easier to work with than the one from etcd.
Using Rocksdb with Go has indeed its own challenges because CGO calls are far from cheap and have an important consequences on Go applications. For performance reasons, we ended up moving some logic from Go to C in a few hot-spots (eg. moving the retrieval of large amount of keys from Go to C).
Disclaimer: I work for Stream and I am one the two co-founders
Using Rocksdb with Go has indeed its own challenges because CGO calls are far from cheap and have an important consequences on Go applications. For performance reasons, we ended up moving some logic from Go to C in a few hot-spots (eg. moving the retrieval of large amount of keys from Go to C).
Disclaimer: I work for Stream and I am one the two co-founders
Could you elaborate on Keevo? Could you compare it to Riak or Cassandra for example? Do you have any plans to open source it?
> since they mentioned that rocksdb is written in C++, I am interested to know the cgo issues
On that subject, is there a Badger[0] based distributed key-value store, equivalent to etcd, ready for production?
[0]: https://github.com/dgraph-io/badger
On that subject, is there a Badger[0] based distributed key-value store, equivalent to etcd, ready for production?
[0]: https://github.com/dgraph-io/badger
I wonder why people don't use badger over rocksdb.
I disagree. I think it is great to see how people build stuff and everybody can get the conclusion. I am super interested in Keevo and hoping to get to know more. Rocksdb is a rock solid foundation (no pun intended), some of the most amazing developers are working on it. The only missing part is how to create a service on the top, sort of like LevelDB + Riak. Actually it would make sense to Riak guys to look into using RocksDB instead of LevelDB.
Why not badger since it's clearly superior to RocksDB in terms of performance and written in Go: https://github.com/dgraph-io/badger
Don't worry. This involves Go, a Google connection, so it is immune to criticism of bias.
I envy you guys to build such great technical product and being able to sell it.
I would love to know also the "business history" of the product: how you started, the first sale, the marketing approach and how you move from there...
I would love to know also the "business history" of the product: how you started, the first sale, the marketing approach and how you move from there...
Stream always looks interesting, until I hit their pricing page, where I see this:
Starter, $59/month, 5 million updates
Growth, $269/month, 9 million updates
Wait, what? To increase the updates by 80%, the price jumps by 356%.
I understand they're throwing in higher processing and custom ranking. That's still approximately the most absurd pricing variance on one plan to the next on a scaling service that I've ever seen.
Starter, $59/month, 5 million updates
Growth, $269/month, 9 million updates
Wait, what? To increase the updates by 80%, the price jumps by 356%.
I understand they're throwing in higher processing and custom ranking. That's still approximately the most absurd pricing variance on one plan to the next on a scaling service that I've ever seen.
The difference is large because we have a free tier of 3 million feed updates. So practically the first plan is paying for 2 and the 2nd for 6. The free and small plans make it easier for developers and smaller startups to start using Stream. I see that you're founder, want to send me some examples of pricing that you like? (email is in my profile)
You do realize that what you actually just said is: the difference is large because your valued customers in the second tier with the gigantic 356% price jump, are paying for your perhaps overly generous free plan.
What you maybe should have said is (speaking from a sales perspective): the reason there's a 356% jump in price, is because of an incredible value proposition step up of 1,000% in what you're getting!
I think that price gap is a classic, and very large, opening for a competitor. There's a ~$89-$129 style plan missing in there, that is a logical step up from $59. That's obviously just my opinion. I also happen to be a big fan of charging more rather than less, in the service space; that under-charging is a common mistake young companies make. And I still can't make sense of $59 to $269 as a step.
I launched a new business in the last quarter. It has a user feed system that is fairly standard. I've built several feed-like features in the last decade. I looked at Stream a while back, having run across it while trying to avoid building my own again. The good news is, I like your product, it got my attention. That price jump between plans, for a very modest increase in service volume, made it a non-starter. It's too easy and cheap to build my own, which then gives me tight control over its evolution over time as well. It's basically impossible at your price points to compete with what I can just do for myself relatively quickly; that $269 product can be run on a low-cost Digital Ocean droplet in terms of its actual resource demands. You're charging for the quality of your software product primarily, and some support, which makes perfect sense. The problem, is that I can build that and scale it to the $899 plan and save myself $10k a year. If you could do 10 million updates at ~$50-$75 / month, with normal plan stepping from there, it'd be interesting. 250k to 350k updates per day is not an immense sum, just 20,000 daily active users can saturate that pretty easily.
What you maybe should have said is (speaking from a sales perspective): the reason there's a 356% jump in price, is because of an incredible value proposition step up of 1,000% in what you're getting!
I think that price gap is a classic, and very large, opening for a competitor. There's a ~$89-$129 style plan missing in there, that is a logical step up from $59. That's obviously just my opinion. I also happen to be a big fan of charging more rather than less, in the service space; that under-charging is a common mistake young companies make. And I still can't make sense of $59 to $269 as a step.
I launched a new business in the last quarter. It has a user feed system that is fairly standard. I've built several feed-like features in the last decade. I looked at Stream a while back, having run across it while trying to avoid building my own again. The good news is, I like your product, it got my attention. That price jump between plans, for a very modest increase in service volume, made it a non-starter. It's too easy and cheap to build my own, which then gives me tight control over its evolution over time as well. It's basically impossible at your price points to compete with what I can just do for myself relatively quickly; that $269 product can be run on a low-cost Digital Ocean droplet in terms of its actual resource demands. You're charging for the quality of your software product primarily, and some support, which makes perfect sense. The problem, is that I can build that and scale it to the $899 plan and save myself $10k a year. If you could do 10 million updates at ~$50-$75 / month, with normal plan stepping from there, it'd be interesting. 250k to 350k updates per day is not an immense sum, just 20,000 daily active users can saturate that pretty easily.
Engineer at Stream here and I'm really not the guy to talk to about pricing.
However, I can tell you that our features that are hardest to implement in a scalabale way are aggregated feeds (e.g. one message with "tom watched 10 videos") and custom ranking (reddit/HN style ordering with time decay and points). If you're feed system doesn't have either of these features it might indeed be possible to do it cheaper than our more expensive plans.
However keep in mind, a lot of the cost is in the availability and reliability of our service. Basically for every type of server we run (including databases), we need to run at least two to be able to stay online in case one dies. Finally building and maintaining a feed system is not cheap in terms of developer cost (try finding a dev for $10k a year).
However, I can tell you that our features that are hardest to implement in a scalabale way are aggregated feeds (e.g. one message with "tom watched 10 videos") and custom ranking (reddit/HN style ordering with time decay and points). If you're feed system doesn't have either of these features it might indeed be possible to do it cheaper than our more expensive plans.
However keep in mind, a lot of the cost is in the availability and reliability of our service. Basically for every type of server we run (including databases), we need to run at least two to be able to stay online in case one dies. Finally building and maintaining a feed system is not cheap in terms of developer cost (try finding a dev for $10k a year).
But the price per update is getting higher. Usually you would expect a lower price per unit if you use more.
Let's do some math and take their metrics and divide by their number of servers:
- 6.3 million feed updates per day per server
- 111 API requests per minute per server
If you guess 4 cores per server it even sounds worse.
(Ps, will HN ever support better formatting?)
- 6.3 million feed updates per day per server
- 111 API requests per minute per server
If you guess 4 cores per server it even sounds worse.
(Ps, will HN ever support better formatting?)
They measured their response time using average. Collecting metrics to look good.
Since we can only speak in broad terms, those requests/updates aren't going to be evenly distributed throughout the day and they might not be consistent from day to day so some over-provisioning is reasonable.
Since they're on EC2 a) they shouldn't be that over-provisioned though and b) are getting comparatively (to bare metal) awful performance (say 2x-8x depending on exactly what they're doing) and paying for the privilege.
I'd like to better understand why "Fanout-on-write" wasn't their go-to solution. All they said was that it was "expensive".
Since we can only speak in broad terms, those requests/updates aren't going to be evenly distributed throughout the day and they might not be consistent from day to day so some over-provisioning is reasonable.
Since they're on EC2 a) they shouldn't be that over-provisioned though and b) are getting comparatively (to bare metal) awful performance (say 2x-8x depending on exactly what they're doing) and paying for the privilege.
I'd like to better understand why "Fanout-on-write" wasn't their go-to solution. All they said was that it was "expensive".
We started out with fanout on write but it was tricky. Fanout on write can get incredibly expensive. Customers would implement things like featured accounts, or topics that you can follow. Basically feeds that are followed by a significant percentage of your userbase and update all the time. Companies like Twitter and Instagram control their product and can avoid some of these pitfalls. A full fanout on write approach means you need to enforce certain product limitations, have a large budget and accept that feed propagation is not always as fast.
There are more factors to take into account than plain API / feed update traffic when looking at server count:
- High Availability requires 2x or 3x more servers
- We run our own monitoring, log collection and tracing infrastructure
- ML/Analytics features
- Distribution of traffic
- Disaster Recovery
Edit: fix list formatting
- High Availability requires 2x or 3x more servers
- We run our own monitoring, log collection and tracing infrastructure
- ML/Analytics features
- Distribution of traffic
- Disaster Recovery
Edit: fix list formatting
Looks like you guys have gotten some surly comments from people here but I enjoyed reading this. It was just enough information to get me interested in researching some things for my own project.
Interesting, what do you think about rocksplicator https://medium.com/@Pinterest_Engineering/open-sourcing-rock...? And also pika (https://github.com/Qihoo360/pika)
The Stream Framework[0] is still in Python with Cassandra. The Go version with RocksDB is not open source.
[0]: https://github.com/tschellenbach/Stream-Framework
[0]: https://github.com/tschellenbach/Stream-Framework
This blog post does raise the question of what’s going to happen to Stream Framework, now that the hosted version is diverging so clearly...
I'd be interested in reading, even if very high-level, some additional details about Keevo. I'm most interested in what the API exposed by Keevo looks like, how it's utilized, and what parts of it are nice/lacking.
The jagger logo is indeed awesome! (https://github.com/jaegertracing/jaeger)
Sometimes it's so hard for me to determine whether half of the stuff on HN is real news or just advertisement.
A news feed for 300M is child's play when you consider that the content is pretty much static.
"Go and RocksDB and Raft" hit all the keywords but what makes this post relevant for HN?
A news feed for 300M is child's play when you consider that the content is pretty much static.
"Go and RocksDB and Raft" hit all the keywords but what makes this post relevant for HN?
Yeah! More product ads masquerading as technical blog posts. Even submitted by the CEO.