Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service(cncf.io)
cncf.io
Cortex: a multi-tenant, horizontally scalable Prometheus-as-a-Service
https://www.cncf.io/blog/2018/12/18/cortex-a-multi-tenant-horizontally-scalable-prometheus-as-a-service/
25 comments
What's the current state of the art for space-efficient long term Prometheus storage?
PromHouse looks very promising: https://github.com/Percona-Lab/PromHouse
PromHouse looks very promising: https://github.com/Percona-Lab/PromHouse
Thanos (https://github.com/improbable-eng/thanos) or Cortex are both great projects depending on your needs.
I would be wary of PromHouse since at a glance it seems to be storing each timestamp in a separate row, which will not be as space efficient as either of the above options.
Full disclosure, I am a maintainer of the Cortex project.
I would be wary of PromHouse since at a glance it seems to be storing each timestamp in a separate row, which will not be as space efficient as either of the above options.
Full disclosure, I am a maintainer of the Cortex project.
IRONdb (https://www.irondb.io/)
Can house multi-tenant-safe prometheus long-tail storage as well as others (graphite, opentsdb, influx, etc.). Put it all in one place, scale it up, radically storage efficient and has strong non-stop HA guarantees. It is state of the art, it isn't free.
Disclaimer: I'm the CEO of Circonus (IRONdb's home).
Can house multi-tenant-safe prometheus long-tail storage as well as others (graphite, opentsdb, influx, etc.). Put it all in one place, scale it up, radically storage efficient and has strong non-stop HA guarantees. It is state of the art, it isn't free.
Disclaimer: I'm the CEO of Circonus (IRONdb's home).
Looking at this architecture, I wonder if it would make sense to replace Prometheus itself with something simpler? Normally Prometheus pulls metrics, evaluates rules against them, stores them, and answers queries on them. It looks like in Cortex the latter three functions are handled by other services, and Prometheus is just pulling and forwarding metrics.
Also, the article says Cortex "completes" Prometheus. While it does fill many of the gaps, there is still a big one: First-class support for pushing metrics. This seems especially important if you want to offer Prometheus as SaaS. Your customers won't want to run their own Prometheus installs (that's what they pay you for, after all), and asking customers to open their firewalls so that your Prometheus can discover and scrape their infrastructure is probably a non-starter.
Also, the article says Cortex "completes" Prometheus. While it does fill many of the gaps, there is still a big one: First-class support for pushing metrics. This seems especially important if you want to offer Prometheus as SaaS. Your customers won't want to run their own Prometheus installs (that's what they pay you for, after all), and asking customers to open their firewalls so that your Prometheus can discover and scrape their infrastructure is probably a non-starter.
I am not sure why are you saying that the customers don't want to run their own instances as they are very lightweight. It makes architecture really flexible as one Prom instance can scrape whole environment. New applications just need to expose /metrics endpoint and set some annotations on their deployment manifests. The main problems with running Prometheus yourself are just storage capacity and query speed.
I played with Cortex for a bit, built my own gateway to match & assign user IDs for incoming Prometheus push requests but from operational perspective running Cortex can be very expensive (if you want it just as a side project) :) GCP Bigtable seems like the best way to store & query data but even the dev instance is prohibitively expensive (last time I checked it was ~1k USD)
I played with Cortex for a bit, built my own gateway to match & assign user IDs for incoming Prometheus push requests but from operational perspective running Cortex can be very expensive (if you want it just as a side project) :) GCP Bigtable seems like the best way to store & query data but even the dev instance is prohibitively expensive (last time I checked it was ~1k USD)
At that point you're just treating Prometheus as an agent, like telegraf or collectd, running in the customers infrastructure to collect metrics from it and publish them to Cortex. That gets to my first point that Prometheus does a lot more than that, so maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example.
It's flexible for some architectures, but not others. If your potential customer runs their own servers or something like k8s that they can deploy Prometheus to, sure. But what if your potential customer runs on a PaaS like Heroku? Or a FaaS like lambda? Or what if what they primarily care about getting metrics from are jobs on some managed data processing service, like cloud dataflow? Etc.
In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you.
It's flexible for some architectures, but not others. If your potential customer runs their own servers or something like k8s that they can deploy Prometheus to, sure. But what if your potential customer runs on a PaaS like Heroku? Or a FaaS like lambda? Or what if what they primarily care about getting metrics from are jobs on some managed data processing service, like cloud dataflow? Etc.
In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you.
maybe it would be beneficial to replace it or strip it down to ensure your customer never has
to worry about storage performance, for example.
what if your potential customer runs on a PaaS like Heroku
In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players
like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code
against to push data to you.
In Weaveworks case we've build integrations into our product that help connect and pull data from other systems. Overall I think Prometheus metrics have a strong future, and there will be lots of SaaS options for sending and consuming the data.
> The way the user runs it means that they don't store any metrics locally
Ah, that's great! I was mistakenly assuming that Prometheus would still have to write the data to disk; I didn't realize you could disable local storage.
> The range of exporters is very extensive
Sure, but that doesn't solve the problem that your heroku dyno, or lambda function, etc cannot expose metrics for Prometheus to pull, but it could easily push metrics somewhere.
> it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples.
That's fair; I was thinking of Cortex as something to adopt off the shelf, instead of something to integrate with. However, as a SaaS provider you can always build more custom functionality alongside Cortex instead of making it a direct part of the open-source project.
Ah, that's great! I was mistakenly assuming that Prometheus would still have to write the data to disk; I didn't realize you could disable local storage.
> The range of exporters is very extensive
Sure, but that doesn't solve the problem that your heroku dyno, or lambda function, etc cannot expose metrics for Prometheus to pull, but it could easily push metrics somewhere.
> it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples.
That's fair; I was thinking of Cortex as something to adopt off the shelf, instead of something to integrate with. However, as a SaaS provider you can always build more custom functionality alongside Cortex instead of making it a direct part of the open-source project.
> I didn't realize you could disable local storage.
You are correct: there is no option to disable so we set to discard after 2 hours. Prometheus doesn’t write continuously to disk so the io is not noticeable.
To your original question, yes we could produce another agent from the Prometheus code which did discovery, scraping and sending but not disk storage. We choose not to as it’s another thing to maintain.
Also we want an option to queue on disk when the network link to push is down.
(I am a Cortex maintainer)
You are correct: there is no option to disable so we set to discard after 2 hours. Prometheus doesn’t write continuously to disk so the io is not noticeable.
To your original question, yes we could produce another agent from the Prometheus code which did discovery, scraping and sending but not disk storage. We choose not to as it’s another thing to maintain.
Also we want an option to queue on disk when the network link to push is down.
(I am a Cortex maintainer)
> I'd expect you'd have to run an API that your customers can code against to push data to you.
Thats a good point; Cortex does accept pushes of metrics but its not that simple, and its definitely not easy right now. We're going to be working with the OpenMetrics project to have a better story about push and serverless - I expect we'll have this pretty soon, ie a matter of months.
Thats a good point; Cortex does accept pushes of metrics but its not that simple, and its definitely not easy right now. We're going to be working with the OpenMetrics project to have a better story about push and serverless - I expect we'll have this pretty soon, ie a matter of months.
Nice! I agree Cortex's operational overhead can be quite high. Good news! We are going to fix that. We've got PRs open to add local storage (based on BoltDB) to remove the dependancy on DynamoDB/Bigtable, and I've almost finished a single binary, non-microservices version of Cortex.
If you want a sneak peak, checkout grafana/loki; its based on Cortex and has all theses features - buts it for logs, not metrics.
If you want a sneak peak, checkout grafana/loki; its based on Cortex and has all theses features - buts it for logs, not metrics.
Hi,
It is my understanding that Prometheus's storage engine is LSM based, and uses a compression scheme inspired by gorilla.
Boltdb is basically a btree which causes more write amplification.
Maybe you should try badgerdb.
It is my understanding that Prometheus's storage engine is LSM based, and uses a compression scheme inspired by gorilla.
Boltdb is basically a btree which causes more write amplification.
Maybe you should try badgerdb.
Cortex also uses gorilla-inspired compression, using the Prometheus source code.
After compressing timeseries they are stored and indexed using a database-like thing, typically DynamoDB or Bigtable, but soon BoltDB will be an option.
(I am a Cortex maintainer)
After compressing timeseries they are stored and indexed using a database-like thing, typically DynamoDB or Bigtable, but soon BoltDB will be an option.
(I am a Cortex maintainer)
This seems especially important if you want to offer Prometheus as SaaS.
Your customers won't want to run their own Prometheus installs (that's what they pay you for, after all),
and asking customers to open their firewalls so that your Prometheus can discover and scrape their
infrastructure is probably a non-starter.
The upsides of running the agent on their side is that they benefit from Prometheus' automatic service discovery. They can also use standard Prometheus configuration and scrapers to get the full capabilities for their environment.
There are also situations where the user wants to continue to run their own Prometheus, but down sample and send a limited number of metrics for long-term storage in the Weave Cloud SaaS. This can be done with one line in a standard Prometheus.
It's true that running the agent takes resources, I think the benefits are worth it. For most people the key question is getting the full value from their metrics data to enable better development and operations.
[NOTE]: Full Disclosure I work at Weaveworks where we provide Prometheus-as-a-Service (https://www.weave.works/features/prometheus-monitoring/)
Supporting push metrics is not a Gap. It is literally the design goal of Prometheus to have a pull based metrics model.
Whatsmore, Cortex is push based...
I recently opened a PR to add support for pushing metrics to Cortex from Telegraf, so you don't need Prometheus as the agent: https://github.com/influxdata/telegraf/pull/5003
NB I'm the main author of Cortex...
NB I'm the main author of Cortex...
Prometheus is winning because push-based metrics are the wrong approach. Integration with service discovery to tell if your metrics are operating or not is, in my experience, the #2 or #3 preventer of outages.
I love Prometheus but in some cases push-based metrics have they advantages. E.g. I'm using push based system in my home automation, where apart from metrics (stats of temp and humidity) it also works as a state machine (door open, switch off) so that I can build logic on top of that by listening to changes. I think having realtime data helps in evaluating some server problems too. Obviously everything depends on use case and not every solution scales well.
Re having downtime indicator that's really a few lines of code, if no event received in the last X seconds then the host is down or busy enough to need attention. With websocket connection you can react to such downtime pretty much immediately instead of waiting for the next poll.
I know there are workarounds but I really wish Prometheus would support some push based system out of the box. It also helps with tracking simple scripts that only run occasionally etc. And yes, I know, they provide workarounds, the docs are really good.
It seems to me that having a realtime system with push based metrics is cheap enough that we should have it already. Imagine ideal monitoring system of the future. Does it poll and refresh data every so often or are you able to see everything as it's happening?
Re having downtime indicator that's really a few lines of code, if no event received in the last X seconds then the host is down or busy enough to need attention. With websocket connection you can react to such downtime pretty much immediately instead of waiting for the next poll.
I know there are workarounds but I really wish Prometheus would support some push based system out of the box. It also helps with tracking simple scripts that only run occasionally etc. And yes, I know, they provide workarounds, the docs are really good.
It seems to me that having a realtime system with push based metrics is cheap enough that we should have it already. Imagine ideal monitoring system of the future. Does it poll and refresh data every so often or are you able to see everything as it's happening?
If I had to speculate why Prometheus is doing well, I'd put the simplicity of deployment and ongoing operations first, and probably the great integration with Kubernetes second.
I think that if it had gone with a push-based model it would have been just as successful.
Push vs Pull is unrelated to integrating service discovery. Right now I push metrics to Datadog. It can tell if my metrics are operating or not, because it's aware of the state of the Datadog agent and of what the EC2 API reports.
I think that if it had gone with a push-based model it would have been just as successful.
Push vs Pull is unrelated to integrating service discovery. Right now I push metrics to Datadog. It can tell if my metrics are operating or not, because it's aware of the state of the Datadog agent and of what the EC2 API reports.
[deleted]
There are reasons for push-vs-pull but one huge advantage of pull vs push is that you're less likely to get overwhelmed by useless metrics. You have to make a very deliberate decision to take the metrics you want. It also makes sense that the observer decides what to observe.
Also, if you really, really want push, there's always the push gateway in the Prometheus tool kit. Also, you can use Telegraf from InfluxDB with Prometheus.
Also, if you really, really want push, there's always the push gateway in the Prometheus tool kit. Also, you can use Telegraf from InfluxDB with Prometheus.
I think this misrepresents the pull vs. push issue.
I wrote something up two years about this that was pretty well received. https://www.circonus.com/2016/06/pully-mcpushface/
I wrote something up two years about this that was pretty well received. https://www.circonus.com/2016/06/pully-mcpushface/
Thanks for that amazing insight! Super helpful. I like how you framed your thinking.
There is already an amazingly popular tech usage of Cortex, you almost certainly have one either on your person or feet from you night now.
For SEO reasons alone it makes no sense to call your new service by a well known name belonging to someone else.