It does seem odd, especially since in real world cases I'm more accustomed to seeing Druid and ClickHouse be in the same ballpark of performance. Sometimes one is somewhat faster than the other. But in my experience that's more like 2–3x difference in one direction or the other depending on workload, not 100x.
Hard to say why more of a difference shows up here, since I haven't analyzed the benchmark. It's possible the Druid configuration is suboptimal in some way. It's also possible it has something to do with the setup. It appears that the ClickHouse tests were done using a local table, which there isn't an equivalent of in Druid. Druid treats every table like what ClickHouse would call a "distributed" table. My understanding is using a distributed table in ClickHouse adds overhead since the system can no longer assume all data is on a single server. It may be that using distributed queries in both systems would yield a different result. And of course it may be that some of the test queries exercise functionality where ClickHouse is legitimately better optimized. But, again, hard to say anything for certain without detailed analysis.
This is impressive work: it's time consuming to set up and benchmark so many different systems!
Impressiveness of the effort notwithstanding, I also want to encourage people to do their own research. As a database author myself (I work on Apache Druid) I have really mixed feelings about publishing benchmarks. They're fun, especially when you win. But I always want to caution people not to put too much stock in them. We published one a few months ago showing Druid being faster than Clickhouse (https://imply.io/blog/druid-nails-cost-efficiency-challenge-...) on a different workload, but we couldn't resist writing it in a tongue-in-cheek way that poked fun at the whole concept of published benchmarks. It just seems wrong to take them too seriously. I hope most readers took the closing message to heart: benchmarks are just one data point among many.
That's why I appreciate the comment "All Benchmarks Are Liars" on the "limitations" section of this benchmark -- something we can agree on :)
I'm a committer on Apache Druid and generally a big fan of observability. I'm glad that you found Druid useful in building this!
A tip, if you aren't already doing it: with metric and trace data, it helps a ton to set up partitioning and sorting according to the query patterns you expect. Timeseries databases usually do this out of the box, because they can make assumptions about your query patterns, but general purpose databases like Druid usually need an extra step or two. Some references:
IMO Druid is most well-differentiated if you want to power an online, real-time, high-concurrency analytical application at scale. It is the use case Druid was originally designed for and still the one where the project shines the brightest. The reason mostly isn't related to things that database people usually talk about (storage format, indexes, etc). That stuff is important but isn't a major differentiator between systems in today's world. The reason is more related to the pieces in between servers, like locking, replication, fault tolerance, data partitioning and balancing, and resource management. Druid's approach to these things is relatively unique and gives it characteristics that allow it to do well at powering these sorts of apps at scale. I think it will remain an important advantage of Druid over other systems. Maybe one day the details would make a good blog post :)
As far as the roadmap goes, most of the work we're doing to make Druid better falls into two categories: first, stuff that makes it even better at this core analytical app engine use case; second, stuff that better supports new use cases, like the work on building out SQL. They are both important so usually each release has a bit of both.
Hey Mani. Druid committer here. It actually is a column store! The project makes a big deal about its ability to do indexes and pre-aggregation because those are important capabilities and, while not unique, are also not universally supported by every column store out there. So they are interesting differentiators. But architecturally they are really just extra icing on the cake.
Personally I see stuff like Druid, MemSQL, Clickhouse, Redshift, BigQuery, and Snowflake as technological siblings in the space. These systems are all evolving rapidly too (well, the healthy ones are anyway) so it's definitely a good time to be an analytical database enthusiast.
With regard to the operational complexity, that's an interesting point. It shows up in two main ways, I think -- the multi-process architecture and usage of external deep storage. On huge clusters, which is what Druid was designed for, the idea is that explicitly separating components in this way gives you three benefits: they don't interfere with each other (spikes in ingestion load won't interfere with ability to query historical data), you can scale each one individually, and it makes most components "disposable" (as long as your storage is reliable, the other Druid components can be blown away and recreated without losing any data). It helps when you're trying to run a big cluster in a stateless / containerized environment.
But these aspects are less good on small clusters or single servers, where it just feels like a bunch of overhead. So we're currently working on simplifying some of this for people that aren't running huge clusters.
We're also expanding SQL support rapidly. Almost every release adds additional SQL capabilities. The next release is a big one, adding JOIN and GROUPING SETS operators. The project's goal is to support it all before too long -- up next after this release will likely be analytic functions.
If you're interested in checking out the community, we do meetups pretty often (all virtual now, though, due to COVID-19). We're also planning our first user conference later in the year @ https://druidsummit.org/.
Logically, an array of booleans and a set of integers are equivalent. So in the Druid developer community we usually use the terms interchangeably. But to be precise, our indexes are all stored as bitmaps and compressed with bitmap compression libraries.
Druid committer here. (Also, I think we've met before in SF!)
One thing I wanted to add with regard to performance. Druid does indeed get a big boost from the fact that it uses inverted indexes for filtering. It also gets a boost from having a wide variety of approximate algorithms you can use if you want (for things like topN, count distinct, set difference/intersection, quantiles, etc). But straight scan performance has been improving quite a bit recently too.
The biggest change related to straight scan perf is fully vectorizing the query engine, which is partially done as of the latest release (0.17): https://druid.apache.org/docs/latest/querying/query-context..... In benchmarks, the implementation so far has been posting row scan rate improvements in the 2-3x range. I expect we'll be able to round it out and have it work for all queries over the next couple of releases. The multiples involved mean this is quite meaningful if you do a lot of straight scans.
There's plenty of other stuff going on too: our latest release added parallel merging of large result sets. Our next one (0.18) is going to add a new, more efficient hash aggregation engine. That next release is also going to add a JOIN operator -- not perf related, but probably the number one most requested feature.
Druid committer here. Fwiw, Druid was designed to run on huge clusters and that really shows up in the multi-process architecture. The idea is that if you separate the components needed for ingestion, historical processing, query routing, and coordination, then there are two benefits: they don't interfere with each other (spikes in ingestion load won't interfere with ability to query historical data), and also you can scale each one individually for your workload. You could even auto-scale some of them. For example, the original Druid cluster was operated with load-based auto-scaling for the ingestion processes.
That being said we are currently working on reducing the number of processes to 4 (from the current 6) for a "standard" setup. The main reason is that at smaller scale there isn't as much of a purpose to having a larger number of processes.
We're also working on removing some of the knobs. Actually, depending on what version you originally looked at, many of them might already be gone.
Check out Druid [1], an open-source analytical database with tightly-coupled storage and processing engines designed for OLAP. In particular it implements a memory-mappable storage format, indexes, compression, late tuple materialization, and query engines that can operate directly on compressed data. There is a patch out to add vectorized processing as well, so you should expect to see that show up in a future release.
Its storage format and processing engine aren't designed to be embedded in the same way as RocksDB and SQLite are, but you certainly could if you wanted to, since the code is fairly modular. Or you could use it as a standalone service as it was designed to be used.
Fwiw, more recent versions of Druid have a no-rollup mode that does ingestion row-for-row. It ended up being useful for cases where you _do_ care about every row, maybe because you want to retrieve individual rows or maybe because you don't want to define your rollups at ingestion time. And in that mode, Druid behaves like the other DBs you mention.
The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).
Hard to say why more of a difference shows up here, since I haven't analyzed the benchmark. It's possible the Druid configuration is suboptimal in some way. It's also possible it has something to do with the setup. It appears that the ClickHouse tests were done using a local table, which there isn't an equivalent of in Druid. Druid treats every table like what ClickHouse would call a "distributed" table. My understanding is using a distributed table in ClickHouse adds overhead since the system can no longer assume all data is on a single server. It may be that using distributed queries in both systems would yield a different result. And of course it may be that some of the test queries exercise functionality where ClickHouse is legitimately better optimized. But, again, hard to say anything for certain without detailed analysis.