The tl;dr is it's just a lockless hashmap attached to a TCP server with a log. Simple Get/Set operations are highly optimized, so with high batching they are able to efficiently fetch a lot of data efficiently. The architectures scales very well when you add threads and data access that is uniform.
It struggles a bit on certain types of workloads like hot keys, think heavy hitting a single sorted set. It's a cool architecture.
> So, you are practically saying you were using, at Amazon, all the BSD code we provided, provide an important part of the code to us? You see how broken such model was? At least stop defending it.
I'm not defending it, I'm trying to fix it. I want Amazon to contribute back. That's what I spend most of my time doing, but I can't just sit in a meeting and tell people we should give away code. It takes time to convince people that we should collaborate on the core and just compete on what we want to differentiate on. It takes time to convince people that building open-source in a vendor neutral space makes software that is better for everyone.
Do you have data to back up your claims? I see a lot of customer claims for Valkey here, https://aws.amazon.com/elasticache/customers/. Neither of the AWS or GCP offerings are in preview.
The Valkey implementation of multi-threading is fundamentally different than what existed in Redis. The history dates back to work done in ElastiCache that was released as "Enhanced IO", https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-el.... The version released in Redis could only do about 350k RPS because of poor memory locality of operations, the inability to do command processing while handling I/O, and the inability to offload much of the TCP read path. The new version in Valkey can achieve 1.2M RPS.
"They made certain improvements later", should be "we threw away the old implementation and built a better one."
It's not just a fork, there have been two releases on Valkey that improved performance and memory efficiency. There is a lie that Redis likes to spread that only their own employees were working on the core engine at the time of the fork, but most of the engineers on Valkey came directly from having worked on Redis OSS. A recent example is we modernized the hash table a bit: https://valkey.io/blog/new-hash-table/.
Yeah, there has been a lot of stuff like performance [1] and efficiency improvements [2]. A lot of the contributors, that didn't work for Redis labs but worked on Redis OSS before the fork, moved to Valkey and they continued to contribute.
Everything we do is open-source, so you're free to take it and incorporate it back into Redis. (Although I'm more into rewriting the data structures to more efficient on modern CPU hardware). Keep some comments out in the open like your discussion about vector sets to keep the conversation going. For the record, I really don't like the vector set idea, but I'm not one to dish out hot takes on hacker news.
As one of those 12 other people, number of commits is not a good metric in this context. When you have a single maintainer, like antirez was for a long time, he could commit 10+ commits for a single small feature but I needed to get someone else to approve my commit before merging.
I know this doesn't mean much, but I spent a lot of time outside of work helping to maintain the Redis codebase. I attended our meetings and responded to issues on vacation because I cared about the community, it was more than just a paycheck for me. I don't regret doing any of that.
I worked with antirez a bit before he left the project in 2020, and I still look fondly back on that time since he was a wonderful person and he helped me learn a lot. I wish he had considered coming to work with Valkey instead of Redis, but maybe we'll find some some way to work together that benefits both communities.
You might be able to get near 100% of Redis performance without a lot of tuning. Redis has a lot of overhead to make it a production ready system like performance counters, safety checking, and conditionals for features like replication/clustering/etc which slows it down.
Hell, you might even be able to use a concurrent hash map implementation and show multi-threaded performance much better than Redis :).
> Fair point. Was that foreseeable in 2009? A lot of the actions we take now are based on what's reasonably foreseeable in the future. I honestly don't recall.
The company now known as Redis didn't acquire the redis trademark until 2018, and didn't change their name to include redis until after that. That is probably the better timeframe to think about.