Jepsen: Aerospike 3.99.0.3(jepsen.io)
jepsen.io
Jepsen: Aerospike 3.99.0.3
https://jepsen.io/analyses/aerospike-3-99-0-3
45 comments
I highly recommend people watch Kyle Kingsbury's talks on YouTube. They're extremely interesting, especially to see the differences in issues that affect each database. He's also a great speaker, and makes it entertaining.
Anyone using AeroSpike in production? It's not a project you hear about much these days.
We do and we‘re still okay with it. It‘s an incredible piece of engineering of a database, all optimized towards a single use case:
Guaranteed single-millisecond reads and writes on a key-value store that‘s too expensive to fit in RAM.
That being said, nowadays it‘s hard to have a dataset with exactly these properties, and honestly they messed up all other use cases pretty hard.
Secondary indices are hard in practice, and so is any other structure, like sorted sets, or lists, or trees, or any kinds of analytics. They tried a lot of things (LDT) and abandoned them again.
Also, AS can be a diva at operations. The amount of brains and engineering within Aerospike is mindblowing, but it‘s like a F1 car: The fastest you can build, but everything is special and custom made and expensive to replace.
We‘re looking currently at Scylla, which is still decent at K/V (also thanks to recent hardware improvements), but wayyy more flexible from the data model side.
Guaranteed single-millisecond reads and writes on a key-value store that‘s too expensive to fit in RAM.
That being said, nowadays it‘s hard to have a dataset with exactly these properties, and honestly they messed up all other use cases pretty hard.
Secondary indices are hard in practice, and so is any other structure, like sorted sets, or lists, or trees, or any kinds of analytics. They tried a lot of things (LDT) and abandoned them again.
Also, AS can be a diva at operations. The amount of brains and engineering within Aerospike is mindblowing, but it‘s like a F1 car: The fastest you can build, but everything is special and custom made and expensive to replace.
We‘re looking currently at Scylla, which is still decent at K/V (also thanks to recent hardware improvements), but wayyy more flexible from the data model side.
MySQL Cluster (NDB) is a better choice for many looking for a near real-time key-value store. It support cross-partition transactions, partition-pruned index scans, partition-aware transactions, multi-master asynchronous geographic replication between clusters, an Event API, on-disk columns, on-line add node, and has been around the block a lot longer.
We benchmarked AeroSpike vs NDB (a key-value store on top of NDB), and NDB out-performed Aerospike for most workloads with equivalent hardware:
http://www.diva-portal.org/smash/get/diva2:849736/FULLTEXT01...
Is this tested with Jepsen?
I don't think so. I believe it would fail quite a few of Jespen's tests, as it is a real-time DB. Typically, you set the transaction deadlock detection timeout to be a couple of seconds. Jepsen thinks transactions can take minutes to complete, and it's ok - the messages are just late.
In fact, in NDB, most people think 2PC is a blocking protocol, but it isn't in NDB. If the Transaction Coordinator (TC) fails, there is a quick leader election protocol and a new one takes over failed TCs. Participant failures cause blocking - but only a few seconds (the configured deadlock detection timeout). So, it's behaviour is similar to abortable consensus (aka Paxos), if automatically retry failed transactions.
> Jepsen thinks transactions can take minutes to complete, and it's ok
Jepsen takes great pains to make no assumptions about the time it takes for transactions to execute. We test systems with millisecond-level timeouts, or multi-minute timeouts, and both work fine.
Jepsen takes great pains to make no assumptions about the time it takes for transactions to execute. We test systems with millisecond-level timeouts, or multi-minute timeouts, and both work fine.
Ok, my mistake. I saw you tried to install NDB several years ago, without any luck.
Nowadays, there's mysql cluster mgr, severalnines installer, and we wrote some chef code to automate it. So it's pretty straightforward.
can you comment on ease of operations between NDB cluster and aero spike? from my experience NDB cluster is pretty difficult. curious if aero is the same.
Ndb is pretty stable if your workload is stable. Most HLRs at network operators around the world run NDB 24x7, so it is the go-to HA OLTP real-time DB. If, however, you start overloading it, it can be temperamental. So, don't treat it like innodb and do full table scans while expecting millisecond PK lookups.
I can't really comment on running Aerospike in production, because i haven't done it - just run experiments with it.
We did in the past, and lots of AdTech companies currently do. In fact Oracle is one of the biggest users because they acquired BlueKai, and AppNexus is probably their oldest customer.
It's an interesting database that has some strange quirks and is primarily good at a narrow use-case of fast key/value and low-latency on SSDs. It's grown since then but there are better options now like ScyllaDB, Redis Enterprise, Apache Ignite, MemSQL, etc. depending on scenario. I'm not sure there's a situation I would choose Aerospike for anymore.
It's an interesting database that has some strange quirks and is primarily good at a narrow use-case of fast key/value and low-latency on SSDs. It's grown since then but there are better options now like ScyllaDB, Redis Enterprise, Apache Ignite, MemSQL, etc. depending on scenario. I'm not sure there's a situation I would choose Aerospike for anymore.
Memsql doesn't get mentioned around HN that often, and I always wondered why. It's a pretty incredible piece of software.
Yes, we used it for years and it's a very good system and one of the most polished datawarehouse/column-store systems around with a great interface. It's also very fast and reliable.
Unfortunately it's not open-source and doesn't have a production license for the community edition so it costs money. We never had a problem as we always paid for software (even as a startup) but it can be an issue considering their customers are usually bigger enterprises.
Unfortunately it's not open-source and doesn't have a production license for the community edition so it costs money. We never had a problem as we always paid for software (even as a startup) but it can be an issue considering their customers are usually bigger enterprises.
The "developer" edition of Memsql does not permit you to put it into production and doesn't have any high availability features. So basically you can only run the enterprise version which starts at a minimum commitment of $25k annually and goes up according to RAM usage. Many small projects and startups will opt for a database that lets them get started cheaper/free and with less requirements. Also being not open source might make people worried as too many closed source databases folded in the past, leaving users stuck or forced to migrate to something else.
I evaluated Memsql for a project which would have made good use of both the row storage and columnar storage engines. But with Clickhouse, there's now a columnar database that performs extremely well and is completely free and open source. So half the usecase for Memsql went away. For the row based engine, the competition is a bit tougher. If one doesn't need extreme performance, CockroachDB provides a super easy to cluster consistent SQL db. And for people with more performance need, there's Mysql Cluster (NDB) for example or several NoSQL solutions.
Memsql is aiming for the enterprise market with well paying customers. They are not targeting the HN startup scene that much.
I evaluated Memsql for a project which would have made good use of both the row storage and columnar storage engines. But with Clickhouse, there's now a columnar database that performs extremely well and is completely free and open source. So half the usecase for Memsql went away. For the row based engine, the competition is a bit tougher. If one doesn't need extreme performance, CockroachDB provides a super easy to cluster consistent SQL db. And for people with more performance need, there's Mysql Cluster (NDB) for example or several NoSQL solutions.
Memsql is aiming for the enterprise market with well paying customers. They are not targeting the HN startup scene that much.
Do you know of any open-source db that folded and is still alive ? I know of rethinkdb, riak and don't know how alive they really are.
You already mentioned two examples. But open sourcing a database doesn't just prevent against the folding of the company behind it. It creates a community that drives improvements and helps prevent the project from folding. Clickhouse for example has a growing community that files bugs and authors improvements. CloudFlare for example contributed some very useful features.
Open source also means one can examine the bug tracker (in most cases, some don't provide an open bug tracker) for known bugs and dive into the implementation in order to understand the inner workings in more detail if needed. I've made good use of this ability numerous times in the past.
The gist is: if a database which is always a key part of a software architecture, is not open source, then it better provide extremely convincing arguments to choose it over other products. Being open source on the other hand doesn't mean choosing a partical software is a no-brainer. There are tons and tons of open source databases with questionable quality. Open Source as a criterion is just one amongst many. I would for example not hesitate to use some hosted proprietary DB on AWS if it fit the project because I know AWS is unlikely to go away. But some smaller/young companies? The risk that they'll disappear unfortunately is very real in this industry.
Open source also means one can examine the bug tracker (in most cases, some don't provide an open bug tracker) for known bugs and dive into the implementation in order to understand the inner workings in more detail if needed. I've made good use of this ability numerous times in the past.
The gist is: if a database which is always a key part of a software architecture, is not open source, then it better provide extremely convincing arguments to choose it over other products. Being open source on the other hand doesn't mean choosing a partical software is a no-brainer. There are tons and tons of open source databases with questionable quality. Open Source as a criterion is just one amongst many. I would for example not hesitate to use some hosted proprietary DB on AWS if it fit the project because I know AWS is unlikely to go away. But some smaller/young companies? The risk that they'll disappear unfortunately is very real in this industry.
Did riak fold? I'm pretty sure Basho is alive and well.
Basho went bankrupt and all its assets were sold to one of it's customers, bet365, and everything made open-source: http://lists.basho.com/pipermail/riak-users_lists.basho.com/...
And HBase
I did in my previous job. It was a good experience except around the time we started seeing corrupt LDTs (Large Data Type). I think a proper solution was never found and they eventually deprecated the feature. I'd recommend it IF you really think you can model your business on top of the KV interface.
[deleted]
We did at my last place, an ad tech company. It replaced Cassandra as a KV store in some streaming pipelines.
We do use it in production here at Adform.
More about it here: https://www.aerospike.com/adform-divorces-cassandra-scales-4...
Aerospike Founder here. Does anyone care that, after working with Kyle and generating since new versions through the development process, Jepsen found no real consistancy problems?
ie, “In hundreds of tests of SC mode through network partitions, 3.99.1.5 and higher versions have not shown any sign of nonlinearizable histories, lost increments to counters, or lost updates to sets.” – Kyle Kingsbury, Aerospike 3.99.0.3, 12-27-2017
Curious.
ie, “In hundreds of tests of SC mode through network partitions, 3.99.1.5 and higher versions have not shown any sign of nonlinearizable histories, lost increments to counters, or lost updates to sets.” – Kyle Kingsbury, Aerospike 3.99.0.3, 12-27-2017
Curious.
As someone who follows Kyle's work I can say that with Jepsen it's only impressive and shows how serious you are about consistency and distributed systems if there are no real consistency problems on the first try.
Be aware it's nothing to do with rocket science.
There are so many software libraries named after normal concepts from physics and engineering. While I see it as dreamfull flattery, it is really damned annoying when you travel in all of these worlds, especially here on HN where people post all of it.
(Best Regards, Grumpy software-, (defense-)systems engineer and former particle physicist).
(Best Regards, Grumpy software-, (defense-)systems engineer and former particle physicist).
I find this one especially egregious because Jepsen looks a lot like Jeppesen the aerospace products company. They mostly do charts and stuff though so I was pretty confused that they were mentioned alongside the term 'aerospike'
IIRC, Jepsen here is named after Carly Rae Jepsen, the pop singer.
As in "call me maybe", both a reference to the song lyrics and what might happen in a distributed system when a network partition occurs.
My understanding is that the folks who are responsible for this name were also former aerospace engineers. Name overloading is always tricky, and I hope you can forgive their choices.
Best regards, another grumpy software & physics person. :)
Best regards, another grumpy software & physics person. :)
Community edition is not a production ready product.
Use Aerospike only if you are ready to buy enterprise edition. Jepsen test results are applicable to enterprise edition only.
Use Aerospike only if you are ready to buy enterprise edition. Jepsen test results are applicable to enterprise edition only.
Can you elaborate on that? What's missing from CE that makes it not suitable for production? Thanks for your input.
One of the biggest issues is that deletes aren't durable in the CE edition. The index entry will be removed but the key and data remains, and may or may not be reclaimed by a background compaction process. If the server restarts before the data is compacted, the deleted data is revived because the index is recreated.
Amadeus uses community in production, they recently published a blog about it. I believe Rubicon project does too. Expiration works great for a lot of data sets, compared to explicit deletes.
I would like to run the jepsen tests on my own with the current versions of Mongo and Redis, does anyone know how to do that?
The Redis test was quite early in Jepsen's history, and I totally rewrote the library after that--you can still find the original code in the `old` branch, if you'd like to experiment. The Mongo tests are reasonably current; you'll just need to pass a new --package-url on the CLI, I thiiiink. No promises, of course, compatibility is always a moving target. ;-)
I am asking this because I think it's funny (so don't be mad) but does anyone know what this means?
"AEROSPIKE 4.0 NOW GA
Strong Consistency with High Performance for Systems of Record and Systems of Engagement"
It seems like Marketing 3.0 speak, maybe their target market would understand it intuitively?
"AEROSPIKE 4.0 NOW GA
Strong Consistency with High Performance for Systems of Record and Systems of Engagement"
It seems like Marketing 3.0 speak, maybe their target market would understand it intuitively?
> Systems of Record
OLTP
> Systems of Engagement
OLAP
OLTP
> Systems of Engagement
OLAP
ty, knowing they actually mean something is reassuring. I've been seeing too much of this kind of buzzword-mashups lately. Glad to see I am wrong. :)
This kind of analysis is awesome. In a past job I might have deployed Aerospike, then eventually realised it sometimes lost my data. But that would be years later, after much midnight oil had been burned.
This reminds me of the turbo encabulator.
[deleted]
My take - if you didn't test your app in nodr loss scenario, you are likely to lose data when that happens, even if your DB won't.