Can house multi-tenant-safe prometheus long-tail storage as well as others (graphite, opentsdb, influx, etc.). Put it all in one place, scale it up, radically storage efficient and has strong non-stop HA guarantees. It is state of the art, it isn't free.
Disclaimer: I'm the CEO of Circonus (IRONdb's home).
I am an IEEE member. I am not sure why anymore, mostly I think it is important to support the industry.
I am also an ACM member and I find that very valuable. The community is good, the policy engagement is good, the publications are excellent, and their employer subsidy program is excellent. I work at Circonus and we pay for everyone's ACM membership as an employee benefit (including the digital library) and it makes each and every one a more valuable employee -- such an easy investment to justify.
I am also highly active with the ACM. I am currently an ACM Member at Large, so if anyone would like to see any changes within the ACM (and you are a member) I'm here to represent you practitioners! The ACM Turing award (amongst all the other distinguished awards) and very important to rewarding innovation and progress in computing. The ACM is aces.
We have customers that generate tens of millions of measurements per second. Lots of low-level systems latencies can be collected at high volume. Also, high volume online services can easily generate this order of magnitude.
I find this discussion fascinating. When I hear people advocate for SLOs (and SLIs) they are often quite rigorous in how they approach it... that is until the very last step where they hand-wave the math and produce numbers that don't mean anything (like averages of percentiles and such). Often times (specifically for sites/services that have undulating traffic volume like more users in the day than at night), the incorrect mathematics can produce wildly inaccurate outputs... so all that rigor and you end up determining that you've failed or succeeded when, in fact, the opposite is true.
I appreciate the rigor in this post because it provides clear and simple instructions on doing the last step (your calculations) correctly so that all those fancy SLOs are actually honest.
Reducing "Other TSDBs" to log-structured-merge trees is misleading. Any large-scale TSDB has something sophisticated underneath and LSM is often just one tiny part of that. I would argue (as most do) that any TSDB "simply used an LSM" it would be doomed at any scale over time.
Storing and retrieving data has never been all that hard. The challenge is having user-interactive performance on complex queries against the data. Comparing and correlating and deriving and integrating and ... (lots of other analysis). For many "scaled" systems, 100M records/minute isn't uncommon... and while that's very likely possible with your design the question of economic feasibility enters. Solving these problems at scale with good economics is the playground of TSDB vendors today.
Relentless change is most likely the motivator for these innovations. However, I don't think it is needless innovation or innovation for the same of innovation. Personal opinion that there isn't much innovation in what this article covers, but still it is happening elsewhere.
Changes in telemetry production (IoT and others) have fundamentally changed the requirements... millions (and often billions) of data points points per-second are now happening. This is being driven by RSM and IoT. RSM is alluded to in my ACM Queue article: https://queue.acm.org/detail.cfm?id=3178371
Find a group of fellow founders that you can talk with. Even if it is just one or two. As a founder, I can relate to the stress and suffering. Sharing can make all the difference.
Ummm https://github.com/keithf4/pg_partman has been around for a long time and is used all around the world... AND it doesn't require cron jobs. Much tighter solution
If you care about correctness of data, solid data retention and good analytics (prediction, forecasting, etc.) then you should take a look at Circonus.
not premise