[untitled]
46 comments
I really don't get why "scrape the prometheus endpoint" is a go-to now, push model seems to be way less PITA to manage at scale.
> If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/
Any idea how it compares with https://victoriametrics.com/ ?
We're slowly looking for a replacement for InfluxDB (as 1.8 is essentially on life support), the low disk footprint is pretty big advantage here.
> If you get serious about Prometheus, eventually you will want longer data retention, checkout https://thanos.io/
Any idea how it compares with https://victoriametrics.com/ ?
We're slowly looking for a replacement for InfluxDB (as 1.8 is essentially on life support), the low disk footprint is pretty big advantage here.
Scrape is typically just how people get started and works well for small and medium things, it gets you a long way before you need to consider it.
Prometheus remote_write is what people graduate to, this gets you the rest of the way, and you are correct it's less PITA at scale.
If you're looking for retention your choices are large, there's Cortex (CNCF), Mimir (most Cortex work moved here), Thanos, VictoriaMetrics, TimeScale, Chronosphere, and many others.
All seek to do a similar thing from a distance, they all store metrics (likely from Prometheus) and allow retention and some variety of how to query it (if you want SQL you got it, if you want non-standard functions you go it, if your reads are more important than your writes you got it, if you need a billion active series you got it, etc).
If what you want is "Prometheus but bigger" then the Prometheus project provides a compliance suite that you can run to help you evaluate your options: https://github.com/prometheus/compliance
I work for Grafana Labs, and we have maintainers working for us who have touched Prometheus, Thanos, Cortex and Mimir. Mimir is currently the largest investment we have https://github.com/grafana/mimir and it is 100% compliant with Prometheus (though that is about to be temporarily untrue as Native Histograms is landing in Prometheus soon https://github.com/prometheus/prometheus/milestone/10 and we'll need to add a perfectly compliant support to Mimir to get back to being compliant).
Prometheus remote_write is what people graduate to, this gets you the rest of the way, and you are correct it's less PITA at scale.
If you're looking for retention your choices are large, there's Cortex (CNCF), Mimir (most Cortex work moved here), Thanos, VictoriaMetrics, TimeScale, Chronosphere, and many others.
All seek to do a similar thing from a distance, they all store metrics (likely from Prometheus) and allow retention and some variety of how to query it (if you want SQL you got it, if you want non-standard functions you go it, if your reads are more important than your writes you got it, if you need a billion active series you got it, etc).
If what you want is "Prometheus but bigger" then the Prometheus project provides a compliance suite that you can run to help you evaluate your options: https://github.com/prometheus/compliance
I work for Grafana Labs, and we have maintainers working for us who have touched Prometheus, Thanos, Cortex and Mimir. Mimir is currently the largest investment we have https://github.com/grafana/mimir and it is 100% compliant with Prometheus (though that is about to be temporarily untrue as Native Histograms is landing in Prometheus soon https://github.com/prometheus/prometheus/milestone/10 and we'll need to add a perfectly compliant support to Mimir to get back to being compliant).
Interesting... I recall the prom creators being very, very convinced that pull was the one true way (push gateway was viewed as a compatibility concession).
I mean, in most cases it just works, for typical "Here is my HTTP server" services. But for batch processes, not so well, as you're not exposing a HTTP endpoint.
Pull does work like 90% of the time, + you get a handy uptime check built-in as well, as if the pull didn't work, the service is probably offline and you'll notice it immediately. With Push you still need a way of checking if the service is online/offline.
Pull does work like 90% of the time, + you get a handy uptime check built-in as well, as if the pull didn't work, the service is probably offline and you'll notice it immediately. With Push you still need a way of checking if the service is online/offline.
We began a transition from a single behemoth Prometheus instance to Victoria Metrics about a month ago after evaluating a couple of similar tools. VM won out because of its relative operational simplicity and eye-popping benchmark results in resource usage comparisons. Documentation can be a little rough around the edges, but overall we're impressed so far.
VictoriaMetrics is super simple to use and reliable. It's a breath of fresh air compared to running other tools.
> push model seems to be way less PITA to manage
They have a blog post explaining their reasoning:
https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or...
They have a blog post explaining their reasoning:
https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or...
"Hey, I'm trying to figure out some business metrics related to our infrastructure and application performance / uptime; do we have the data anywhere?"
"Sure, Bob, just write a promql query against this interface"
https://news.ycombinator.com/item?id=1636348
(in case my point is too subtle: I believe that the value of long term metric retention is to support business decisions more than supporting availability related monitoring.
The tools "business deciders" use to look at "the things they call metrics" are simply unable to look at data lodged in a prometheus silo. So the data you have for long term retention is simply unusable for the thing it's most valuable)
(edited for clarity)
"Sure, Bob, just write a promql query against this interface"
https://news.ycombinator.com/item?id=1636348
(in case my point is too subtle: I believe that the value of long term metric retention is to support business decisions more than supporting availability related monitoring.
The tools "business deciders" use to look at "the things they call metrics" are simply unable to look at data lodged in a prometheus silo. So the data you have for long term retention is simply unusable for the thing it's most valuable)
(edited for clarity)
I use thanos for availability related concerns. eg, Some metric crossed the threshold for saturation, and I can go back as far as I need to see when the metric began saturating. We have seasonal business patterns that I can show the high water-mark for resource consumption. Thanos also allows a CQRS-type pattern if that's important to folks.
Do I point the biz at it? Only when I'm trying to show them availability related metrics.
Do I point the biz at it? Only when I'm trying to show them availability related metrics.
[deleted]
What tools do they use? It's not hard to build a dashboard in Grafana and show them there the metrics they want, in an easily-digestible format.
There's an entire universe of people making business decisions who live in excel and "BI" tools. If your thing doesn't speak ODBC / JDBC it's invisible.
Further, the vast majority of workaday devs are stressed out and under tremendous pressure to thing all the things in their ticket queue. They're probably pretty fluent in their thinging language and likely also conversant in SQL. Asking them "hey learn this other query language, because" is just adding stress to them. Sure, some are going to rise above and promql things too, but for lots, it's telling them to "write a distributed map reduce query in erlang."
Further, the vast majority of workaday devs are stressed out and under tremendous pressure to thing all the things in their ticket queue. They're probably pretty fluent in their thinging language and likely also conversant in SQL. Asking them "hey learn this other query language, because" is just adding stress to them. Sure, some are going to rise above and promql things too, but for lots, it's telling them to "write a distributed map reduce query in erlang."
PromQL is barely a language. For availability, the query would be as simple as the series name and it would return everything. Also, there's an adapter so that Prometheus writes to PostgreSQL.
However, Prometheus is not a tool that specializes in reporting availability to people who live in Excel. Prometheus scrapes metric from a lot of systems in a standard format, and enables you to see what's happening and write alerts to fix things when they break, without requiring complex frameworks. Reporting availability to people is just a small, tiny thing that Prometheus is able to do.
However, Prometheus is not a tool that specializes in reporting availability to people who live in Excel. Prometheus scrapes metric from a lot of systems in a standard format, and enables you to see what's happening and write alerts to fix things when they break, without requiring complex frameworks. Reporting availability to people is just a small, tiny thing that Prometheus is able to do.
Batch jobs are arguably a lot better served with old-school logging than metrics. If you're using prometheus for alerts, I'll maybe go as far as considering incrementing a "the batch job failed"-counter a good idea. But all this is very much jamming a square peg into a round hole.
Also, there are tools like Cronitor that can help.
I would highly recommend looking at Cortex (or Grafana Labs' fork, Mimir) instead of Thanos.
It doesn't rely on reading the Prometheus WAL (rather, it uses Prometheus's built-in remote-write functionality). This is both less fragile and has the benefit of being about to run Prometheus in stateless mode (or replace it entirely with something like Grafana Agent, vmagent, etc.
Grafana Labs has done a proof of concept showing Mimir scaling to 1 billion active timeseries. There are also "real" users serving 700+ million metrics in production.
It doesn't rely on reading the Prometheus WAL (rather, it uses Prometheus's built-in remote-write functionality). This is both less fragile and has the benefit of being about to run Prometheus in stateless mode (or replace it entirely with something like Grafana Agent, vmagent, etc.
Grafana Labs has done a proof of concept showing Mimir scaling to 1 billion active timeseries. There are also "real" users serving 700+ million metrics in production.
How do you “import” existing Prometheus metrics block into Cortex/Mimir using the remote-write, then?
For Thanos, you can (mostly) just copy the blocks from the existing s3 bucket into Mimir's bucket. There's some cleanup prep around external labels, but nothing too difficult, and the process is well-documented. I just finished migrating a year of Thanos metrics into Mimir without issue.
I note that a lot of people on this thread are saying that there are better TSDBs out there. There probably are (I personally don't like Prometheus), but they are missing the point which is: having metrics is better than not.
Now, for something constructive:
what to metric and how?
Because time series DBs tend to "bin" metrics, that is put metrics into 1s/1m/5m "holes" you will end up being at the mercy of quantisation/averaging. To get round this, use counters for most thing. This has the advantage that its much simpler implement accurately as a programmer.
every time an error happens, increment an error, every time you get a 200, increment.
but why?
because if you do the math locally and work out the hit/second and emit that number, if you get a spike, and you aren't aware of the time binning, your 45k 1 minute spike will be averaged out as the retention lowers the temporal resolution. This means that your 45k spike could be halved each time you go down a temporal step.
Using incremental counters overcomes this, even if temporal resolution is thinned. (you can use derivative (well non negative derivative) to turn it into hit/s later on)
Microservices and metrics
Microservices tend to be simpler to add monitoring, as they _should_ be simple. Yes, by all means record memory, CPU, network and that, but you will most likely not look at that unless you have a difficult bug. make sure each of your instances of the service has a unique ID in the metric address. This will save you many hours of fuckery.
Have counters for all upstream and down stream requests(and error states). Then have gauges for response times of those requests. (beware that gauges will be averaged by time bin.) Do the same for all of the important actions your service does as well.
These metrics will give you a good base to make alerts from:
service x has had 1500 request errors from service y
conversely it can allow you to summarise all your metrics into a green/orange/red tile (ie if no errors in the last minute, be green)
Now, for something constructive:
what to metric and how?
Because time series DBs tend to "bin" metrics, that is put metrics into 1s/1m/5m "holes" you will end up being at the mercy of quantisation/averaging. To get round this, use counters for most thing. This has the advantage that its much simpler implement accurately as a programmer.
every time an error happens, increment an error, every time you get a 200, increment.
but why?
because if you do the math locally and work out the hit/second and emit that number, if you get a spike, and you aren't aware of the time binning, your 45k 1 minute spike will be averaged out as the retention lowers the temporal resolution. This means that your 45k spike could be halved each time you go down a temporal step.
Using incremental counters overcomes this, even if temporal resolution is thinned. (you can use derivative (well non negative derivative) to turn it into hit/s later on)
Microservices and metrics
Microservices tend to be simpler to add monitoring, as they _should_ be simple. Yes, by all means record memory, CPU, network and that, but you will most likely not look at that unless you have a difficult bug. make sure each of your instances of the service has a unique ID in the metric address. This will save you many hours of fuckery.
Have counters for all upstream and down stream requests(and error states). Then have gauges for response times of those requests. (beware that gauges will be averaged by time bin.) Do the same for all of the important actions your service does as well.
These metrics will give you a good base to make alerts from:
service x has had 1500 request errors from service y
conversely it can allow you to summarise all your metrics into a green/orange/red tile (ie if no errors in the last minute, be green)
Telegraf => InfluxDB => Grafana is also a well-known stack for monitoring which I like
See for example this (that also mentions Prometheus):
https://stanislas.blog/2018/04/monitoring-telegraf-influxdb-...
See for example this (that also mentions Prometheus):
https://stanislas.blog/2018/04/monitoring-telegraf-influxdb-...
I like this setup because you don't need to pull the stats like with Prometheus.
But the InfluxDB 2.0 migration have been really bad for me, how have it worked out for you?
But the InfluxDB 2.0 migration have been really bad for me, how have it worked out for you?
What problems did you have?
I've stuck with InfluxDB v1.8. I dislike the new Flux language. I know there is a way to do InfluxQL queries using InfluxDB v2, but I haven't set it up yet.
https://docs.influxdata.com/influxdb/v2.5/query-data/influxq...
I've stuck with InfluxDB v1.8. I dislike the new Flux language. I know there is a way to do InfluxQL queries using InfluxDB v2, but I haven't set it up yet.
https://docs.influxdata.com/influxdb/v2.5/query-data/influxq...
Personally I've also used Apache Skywalking for a decent out of the box experience: https://skywalking.apache.org/
I've also heard good things about Sentry, though if you need to self-host it, then there's a bit of complexity to deal with: https://sentry.io/welcome/
Though of course, Prometheus and Grafana is a rather popular option as well!
I've also heard good things about Sentry, though if you need to self-host it, then there's a bit of complexity to deal with: https://sentry.io/welcome/
Though of course, Prometheus and Grafana is a rather popular option as well!
TIL Skywalking exists. It seems like a complete package and very capable, why aren't more folks using it?
> It seems like a complete package and very capable, why aren't more folks using it?
Personal opinion: it doesn't seem to have gotten as much love or attention as many other commercial projects, currently being under Apache Foundation and just slowly trudging ahead. Someone else mentioned it being more popular in China which is fair, but also means that the docs and blog posts about it etc. aren't getting as much attention in the west and other places.
Personally, for many of my use cases (typically simpler projects, <20 services per environment), it's good enough, the platform support is mostly there for most of the popular languages: https://skywalking.apache.org/docs/
The docs are also there and configuring it is actually easier than Sentry, because of fewer moving parts: https://skywalking.apache.org/docs/main/next/en/setup/backen...
I don't particularly like using ElasticSearch which seems like the "happy path" but ES just loves to eat memory in general. Thankfully, there are actually some nice options for alternatives here, I wonder why other projects haven't embraced such modularity: https://skywalking.apache.org/docs/main/next/en/setup/backen...
Personal opinion: it doesn't seem to have gotten as much love or attention as many other commercial projects, currently being under Apache Foundation and just slowly trudging ahead. Someone else mentioned it being more popular in China which is fair, but also means that the docs and blog posts about it etc. aren't getting as much attention in the west and other places.
Personally, for many of my use cases (typically simpler projects, <20 services per environment), it's good enough, the platform support is mostly there for most of the popular languages: https://skywalking.apache.org/docs/
The docs are also there and configuring it is actually easier than Sentry, because of fewer moving parts: https://skywalking.apache.org/docs/main/next/en/setup/backen...
I don't particularly like using ElasticSearch which seems like the "happy path" but ES just loves to eat memory in general. Thankfully, there are actually some nice options for alternatives here, I wonder why other projects haven't embraced such modularity: https://skywalking.apache.org/docs/main/next/en/setup/backen...
Looks like it's popular in the tech scene in China. Sometimes tech adoption is about marketing and word of mouth, and that can leave the west vs the east pretty insulated I'm thinking?
SkyWalking definitely does it all. But like you mentioned Prometheus plus Grafana is the most popular solution. But it does not compare to the complete observability solution provided by SkyWalking.
Sentry have some issues with main sentry database in postgresql that's why I'm not recommend to use it as self hosted.
Sentry solves tangential but different use cases. E.g does not allow you to define custom metrics to e.g. monitor resource utilization
> E.g does not allow you to define custom metrics to e.g. monitor resource utilization
I think that might not quite be the case in the latest versions: https://docs.sentry.io/product/performance/metrics/#custom-p...
I think that might not quite be the case in the latest versions: https://docs.sentry.io/product/performance/metrics/#custom-p...
In addition to the automatic performance metrics described above, Sentry supports setting custom performance metrics on transactions.
Custom performance metrics allow you to define metrics (beyond the ones mentioned above) that are important to your application and send them to Sentry.
For example, you might want to set a custom metric to track:
- Total memory usage during a transaction
- The amount of time being queried
- Number of times a user performed an action during a transaction
You define and configure custom metrics in the SDK.
I use Sentry to track errors in my web apps :)
In my experience over time dev teams stop looking at that metrics on a board and all we really need are proper monitors/alarms that cover all relevant cases of our domain and writes a slack message or sends an email... KISS!
I tend to find the Grafana dashboards solve a different issue to Prometheus's alerts and getting the balance right between the two really helps.
I'll use alerts are for things that need acting on in the immediate to short term (e.g. a server's load being very high for over 10 minutes or an SSL certificate will need renewing in the next couple of weeks).
And I'll use Grafana dashboards to provide insight in to how a system is running (e.g. graphing number of events triggered over time or a heat map of web servers response times).
As a team we've got a handful of Grafana dashboards that we look at after we've walked our Kanban board. It only takes a few minutes and we all get to know what the current state of our systems are, and over time, what their norm is as well.
I'll use alerts are for things that need acting on in the immediate to short term (e.g. a server's load being very high for over 10 minutes or an SSL certificate will need renewing in the next couple of weeks).
And I'll use Grafana dashboards to provide insight in to how a system is running (e.g. graphing number of events triggered over time or a heat map of web servers response times).
As a team we've got a handful of Grafana dashboards that we look at after we've walked our Kanban board. It only takes a few minutes and we all get to know what the current state of our systems are, and over time, what their norm is as well.
Ability to see trends in prod data and fix issues before they appear => Prom/Grafana
Ability to detect when something is broken right now => Monit, nagios, etc
Ideally, you have both.
Ability to detect when something is broken right now => Monit, nagios, etc
Ideally, you have both.
Personally would have used the OpenTelemetry Metrics approach though
Not that widely supported, but I do prefer the push model over having one node just scrape thousands of URLs... much easier to provision.
We just pipe everything to collectd then riemann for downsampling and influxdb for storage but we're shopping for new stack as influxdb 1.8 is essentially on life support
We just pipe everything to collectd then riemann for downsampling and influxdb for storage but we're shopping for new stack as influxdb 1.8 is essentially on life support
If you are looking for a good replacement for InfluxDB, then take a look at VictoriaMetrics. It accepts data in Influx line protocol [1] and it needs less RAM and disk space comparing to InfluxDB [2].
[1] https://docs.victoriametrics.com/#how-to-send-data-from-infl...
[2] https://valyala.medium.com/insert-benchmarks-with-inch-influ...
[1] https://docs.victoriametrics.com/#how-to-send-data-from-infl...
[2] https://valyala.medium.com/insert-benchmarks-with-inch-influ...
You could use the Metrics API of OpenTelemetry and use the Prometheus exporter etc instead of hard dependency on prom-client
One of the nicest recent addition to Fluentbit has been support for Prom metrics. The same way you can buffer and re-send logs, you can now handle metrics as well. It's always really nice to see metrics data back-filling after something takes down your metrics server.
https://docs.fluentbit.io/manual/pipeline/inputs/prometheus-...
https://docs.fluentbit.io/manual/pipeline/inputs/prometheus-...
I recently tried a bit self-hosted https://www.netdata.cloud, looks very good but I wonder why it's not more popular?
It does have 64k stars on GitHub[0], but I had to google for months to find them.
[0]: https://github.com/netdata/netdata
It does have 64k stars on GitHub[0], but I had to google for months to find them.
[0]: https://github.com/netdata/netdata
Take a look at https://github.com/coroot/coroot (Apache 2.0), a zero-instrumentation observability tool for microservice architectures.
Thanks to eBPF, it can be integrated in minutes.
Live demo: https://community-demo.coroot.com
Live demo: https://community-demo.coroot.com
Pretty sure that application flow is a distributed monolith and not microservices. Food for thought.
azlyrics(2)
Also sometimes you'll have some sort of a batch job, or a cron that can't be scrapped the regular way, checkout Prometheus Pushgateway https://prometheus.io/docs/practices/pushing/