> If a beginner wanted to do anything "cool-looking" today, this same beginner programmer would need to set up render to texture on an OpenGL surface. Yuck! And so they lose interest, believing programming is the mysterious and complex domain of gods.
Or just use the web platform: JavaScript, HTML5 Canvas API, WebGL, all of that. I think it's pretty close in terms of getting a pretty (motivating) result quickly without much prior experience.
(Unrelated, but I can also remember writing the first lines of code of my life in qbasic on a dos machine in ~1998; the blue screen definitely invokes childhood memories!)
If you can't (or don't want to!) afford that kind of money for data analytics, please consider giving the FOSS alternative EventQL [0] a try some time.
It's super simple to set up and tries to be efficient on commodity hardware, so you can run large clusters (>100TB scale) for a couple hundred dollars a month.
>> For example in germany FF is still the widest used desktop browser.
Do you have any references to back that up? I'm asking because I suspect your assumption is completely wrong.
(Based on a bunch of analytics data that I have access to, which might not be representative but still contains some very large german web properties, chrome has more than twice the market share compared to FF in germany).
It's not an entirely new art form. The demoscene [0] has been going strong far longer than most of us have even been coding. So apparently some people _will_ execute interactive art.
Also, you can still listen to their music on soundcloud [1] if you don't want to review the code. It's open source after all, which can't be said for most demos...
Speaking for myself: Berlin has a special kind of poor-is-sexy culture and this would not be too unusual. Lived there for 4yrs, most of them without a smartphone (had a 20eur burner phone for communication and to receive monitoring alerts). Had an absolute blast and didn't feel like I was missing a thing.
>> So it's a completely different class of application than splunk or elasticsearch
Sure it takes a somewhat different approach (i.e. it requires an explicit schema), but for the use case discussed in this thread it _is_ completely relevant and a comparable open-source/on-premise alternative which parent was asking about.
>> one that you have a commercial interest in
Yes, I'm involved in the EventQL project but I thought that it was obvious from the way I phrased my posting. Usually I always include a disclaimer to prevent misunderstandings but I didn't consider it necessary in this case.
> I'm still surprised that there's no startups offering comparable (particularly on-prem) products
Please consider giving EventQL [0] a try some time! It's completely open-source and self-hostable. Still a new project though, just released this summer and still in beta.
Yes, but this is only a solution if your library is only targeting windows.
If you want users of any standards-compliant C++ implementation to be able to use your library today, you'll still have to go with c-abi symbols or ship the sources. All other worakrounds are vendor-specific and not part of the standard.
[Of course even objects containing only C symbols are not portable across platforms either, but at least the C ABI/calling convention is more or less strictly defined for any given target platform. Assuming no other platform-specific stuff like glibc is used]
Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers]
EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!
Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).
Actually, you can even have incompatibilities with the same compiler and different compile flags. So to reliably build a "semi-portable" c++ object one would have to ensure that all objects are compiled with the exact same (i.e. same version of the compiler/same compiler source code) and with the exact same flags. I'm sure there are some compiler vendors that offer ABI backwards-compatibility but it's not part of the C++ language per-se.
That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes.
Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).
> Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets.
The C++ ABI is currently not portable anyway. So the concern about templates (or any C++ features) forcing you to put the implementation into the header file would not apply in the scenario dllthomas was referring to:
If you want to distribute your library as a binary object and a separate source-form interface/header today, you'll have to use a (wrapper) C API. Regardless of how "modern" the C++ code is.
Of course, it would be nice to have portable C++ objects some time in the future which would change things... :)
I think what they meant with "1000 nodes" is that the dataset they're using for the benchmark is synthetic monitoring data (where the thing being monitored are servers).
And the way they generated the synthetic data set is by having 1000 imaginative servers produce one sample per second, (i.e. have a script that writes out 1000 * duration_in_sec fake samples -- I believe this is the code that does it https://github.com/influxdata/influxdb-comparisons/tree/mast...)
Just so sad that Vertica is proprietary so we can't see how they did it! ;)
On a serious note: Please check out EventQL [0] some time. It's very similar to Vertica in some ways and completely open-source. It's a new project (beta) and not nearly as mature as vertica yet though (still a long way to go).
> The point of the comparison is that InfluxDB gives you a ton of functionality out of the box and has great performance. [...] if you want to do time series
on Cassandra, you're going to write a bunch of the code yourself.
Fair enough. I'm sure InfluxDB is very good/fast at timeseries data (allthough I have to admit to not actually having tried it out so far). Still, if that was your point, consider removing these statements from the blog.
> InfluxDB outperformed Cassandra by 4.5x when it came to data ingestion.
> InfluxDB outperformed Cassandra by delivering 10.8x better compression.
> InfluxDB outperformed Cassandra by delivering up to 168x better query performance.
I think it would help make the point and not put the reader in a defensive position (when the statements are clearly not based on a fair comparison of the two products and will not hold under most conditions). Just my two cents.
Yes, the delta encoding scheme you described (and other fancy coding schemes such as bitpacking, varints, RLE or a combination thereof) are frequently employed in columnar storage formats and databases. Columnar storage is basically a generalization that allows one to apply these optimizations to all kinds of data (not just timeseries). One popular open-source implementation of columnar storage that I am not affiliated with is https://parquet.apache.org/.
(On the other hand, columnar storage also has a bunch of tradeoffs/downsides so it's not a superior choice for every db product.)
My point about no "superior compression technology here" was specific to the linked benchmark. I.e. the lack of this potential optimization in cassandra does not appear to be the reason for the space blowup in the benchmark, but rather that they're duplicating the series ID for each sample.
The linked article is an obviously bullshit benchmark that makes influxdb look good and cassandra look bad (by, surprise, the influxdb folks).
I'm far from a cassandra fanboy, but this really is just dishonest marketing. Not sure if that will work if your product is open source and the target audience are developers.
Some thoughts:
- The reason why cassandra uses so much more space to store the same data is that they've set up the cassandra table schema in such a way that cassandra needs to write the series ID string for each sample (while influxdb only needs to write the values). You easily get a 10-100x blowup just from that. There is no superior "compression" technology here but just an apples-to-oranges comparison.
- Then, comparing the queries is even worse, because they are testing a kind of query (aggregation) that cassandra does not support. To still get a benchmark where they're much faster, they just wrote some code that retrieves all the data from cassandra into a process and then executes the query within their own process. If anything, they're benchmarking one query tool they've written against another one of their own tools.
- Also, if I didn't miss anythin, the article doesn't say on what kind of cluster they actually ran this on or even if they ran both tests on the same hardware. There definitely are cassandra clusters handling more than 100k writes/sec in production right now. So I guess they picked a peculiar configuration in which they outperform cassandra in terms of write ops (given a good distribution of keys, cassandra is more or less linearly scalable in this dimension)
- A better target to benchmark against would probably be http://opentsdb.net/ or http://prometheus.io/ - both seem to have somewhat similar semantics to InfluxDB (which cassandra and elasticsearch do not)
DISC: I also work on a distributed database product (https://eventql.io) but it's neither a direct competitor to Cassandra nor InfluxDB nor any of the other products I've mentioned. I hope the comment doesn't come across as too harsh. The article raised some very big (and harsh) claims so I think it's fair to respond in tone.
It seems to solve a different usecase. LLVM is (by their own documentation) not really intended for ad-hoc JIT compilation for queries. It doesn't put a lot of priority on making the compilation fast (but rather makes the produced code fast).
On top of that, LLVM is a massive dependency.
I only looked at NativeJIT a few minutes but it seems to solve both of these issues. It's pretty small in terms of API surface // total amount of code and is explicitly designed for a usecase where compilation happens as part of a user-issued query (that needs to be fast).
Or just use the web platform: JavaScript, HTML5 Canvas API, WebGL, all of that. I think it's pretty close in terms of getting a pretty (motivating) result quickly without much prior experience.
(Unrelated, but I can also remember writing the first lines of code of my life in qbasic on a dos machine in ~1998; the blue screen definitely invokes childhood memories!)