Explaining modern server monitoring stacks for self-hosting(dataswamp.org)
dataswamp.org
Explaining modern server monitoring stacks for self-hosting
https://dataswamp.org/~solene/2022-09-11-exploring-monitoring-stacks.html
35 comments
My first exposure to netdata was part of the default FreeNAS install. It was super easy to use and effective to understand the basic server health and telemetry.
Simple yes; simplistic no.
It can provide you with thousands of metrics you never thought of monitoring. On top of that comes with pre-configured alerts, eBPF support, metric correlations and anomaly rates for every single metric it collects.
Self hosting in general tends to be overkill. It’s mostly people practicing for enterprise tasks.
If you wanted to be practical about things you very likely would just give up self hosting entirely.
If you wanted to be practical about things you very likely would just give up self hosting entirely.
I think there's a pretty large middle ground between what I'd consider a home server setup (or self hosting it wherever) and practicing for enterprise tasks.
Much of the crowd around here and certain subreddits IS probably practicing for more, but I have a counterexample. You won't find him around the internet. Not interested.
I have a friend who's recently learned just enough Linux and Docker to get through a Debian install with some prompting, and run a Matrix home server, Nextcloud and Jellyfin because he thought they were cool. Its an old (for him) repurposed desktop.
He's a mechanic. He doesn't care about anything enterprise. He doesn't want monitoring because he'll know if it's down. He doesn't care if backups run because he's sure he's got the important stuff on an old disk somewhere, probably.
Much of the crowd around here and certain subreddits IS probably practicing for more, but I have a counterexample. You won't find him around the internet. Not interested.
I have a friend who's recently learned just enough Linux and Docker to get through a Debian install with some prompting, and run a Matrix home server, Nextcloud and Jellyfin because he thought they were cool. Its an old (for him) repurposed desktop.
He's a mechanic. He doesn't care about anything enterprise. He doesn't want monitoring because he'll know if it's down. He doesn't care if backups run because he's sure he's got the important stuff on an old disk somewhere, probably.
based on?
I have tried multiple monitoring tools, but I have found Uptime Kuma most useful for me.
Alerts work well and it is not too complicated for simple things.
https://github.com/louislam/uptime-kuma
Alerts work well and it is not too complicated for simple things.
https://github.com/louislam/uptime-kuma
My monitoring needs are very simple and I've had good success with Uptime Kuma.
For checking server metrics i built a simple go service [1] to expose metrics with simple rules applied so uptime kurma will notify me when issues arise (Since the go service will change http status upon alert scenarios).
[1]: https://github.com/ssddanbrown/wirestat
[1]: https://github.com/ssddanbrown/wirestat
Prometheus and other modern application monitoring oriented timeseries solution stacks seem to be an overkill for simple server infrastructure monitoring, Zabbix is easier to setup and use and has all the batteries included.
I hadn't known about Zabbix (or netdata) and I'm currently evaluating our options for monitoring. I was going to explore in the direction of Prometheus (+ Grafana), so I'm glad to know about these other seemingly more straightforward and simple options. Thanks for mentioning it.
Currently using Zabbix for monitoring my own servers. It's... okay.
The UI sometimes feels a bit dated and not everything is as straightforward as one might expect, but for my use case (monitoring a bunch of GNU/Linux hosts) it's sufficient.
Some of the things that are good about it:
Edit: That said, the docs of Zabbix are pretty decent.
Here's how configuring something in it typically looks like in the UI: https://www.zabbix.com/documentation/current/en/manual/web_m...
Here's an example of some of the dashboard functionality: https://www.zabbix.com/documentation/6.2/en/manual/web_inter...
And here's maps that you can embed in the dashboard: https://www.zabbix.com/documentation/6.2/en/manual/web_inter...
The UI sometimes feels a bit dated and not everything is as straightforward as one might expect, but for my use case (monitoring a bunch of GNU/Linux hosts) it's sufficient.
Some of the things that are good about it:
- can be run in containers if you want to, use a familiar DB like MySQL/MariaDB/PostgreSQL
- the agent installation on the hosts that you want to manage is also pretty simple
- supports both active (monitored host sends data to Zabbix) and passive (Zabbix asks the host for data) configurations
- depending on the template that you use, has lots of built in metrics out of the box
- easy to integrate with something like e-mail or SMS messaging for alerting, other plugins exist AFAIK
- also has built in alerts, such as when disk space is low, CPU load is high, memory usage is high, swap space is low, host is unreachable etc.
- has the ability to build dashboards with graphs, network maps etc.
- last I checked (older version, now using Uptime Kuma for this use case) the web monitoring didn't send alerts by default
- last I checked (since haven't bothered to set up again) maintenance windows straight up didn't work and still sent notifications about the agent going down
- in general the UI can be a bit cumbersome and there is some legacy to be found, at least a while back there were legacy graphs and the "modern" ones, each of which had different sets of functionality available
- sometimes certain parts of the OS template also decided not to work, e.g. currently have disk usage not showing up in like 1/6 graphs, even though the configuration is pretty much the same for all of the nodes in question
- in general, it's just not as popular as other solutions and might not have as much tutorials around setting things up in it
Edit: That said, the docs of Zabbix are pretty decent.
Here's how configuring something in it typically looks like in the UI: https://www.zabbix.com/documentation/current/en/manual/web_m...
Here's an example of some of the dashboard functionality: https://www.zabbix.com/documentation/6.2/en/manual/web_inter...
And here's maps that you can embed in the dashboard: https://www.zabbix.com/documentation/6.2/en/manual/web_inter...
> - supports both active (monitored host sends data to Zabbix) and passive (Zabbix asks the host for data) configurations
But this requires different templates for each mode.
Why they did this way is beyond me... though I have a suspicion what they aren't using it themselves on anything than a tiny lab with a couple dozens of hosts.
But this requires different templates for each mode.
Why they did this way is beyond me... though I have a suspicion what they aren't using it themselves on anything than a tiny lab with a couple dozens of hosts.
Thanks for the thorough review and for sharing your experience. Very helpful!
I'm working on a side project that will essentially cover uptime and metrics monitoring for servers, as I find that trying to roll out prometheus monitoring across large numbers of servers across multiple organisations and setups is too much effort. I want to be able to quickly and easily add servers to a dashboard, set a number of alarms, and then not have to worry. Anything happens, an alarm trips, offline etc, and I get an email/app/slack/discord notification, etc.
I looked around but didn't really find anything that fit for me. There are a lot of complicated (albeit powerful) options, but I want simple, easy, lightweight, quick. These days I'm juggling so much, I want to be as efficient as possible with my time.
It's still early days but hoping to be able to onboard people towards the end of the year, for anyone who is interested feel free to join the waitlist: https://serverduty.co
Interestingly, I'm using ServerDuty to monitor ServerDuty as I build ServerDuty. I mean, if that isn't dogfooding, I don't know what is.
I looked around but didn't really find anything that fit for me. There are a lot of complicated (albeit powerful) options, but I want simple, easy, lightweight, quick. These days I'm juggling so much, I want to be as efficient as possible with my time.
It's still early days but hoping to be able to onboard people towards the end of the year, for anyone who is interested feel free to join the waitlist: https://serverduty.co
Interestingly, I'm using ServerDuty to monitor ServerDuty as I build ServerDuty. I mean, if that isn't dogfooding, I don't know what is.
In what way do you think it's an overkill? As in, what's too much / gets in the way? I've gone exactly the opposite way with: I don't want to deal with a highly opinionated and integrated thing like zabbix if I can put a 3 small things together (grafana, influx, telegraf in my case) and have a simple system monitoring that also can handle anything I want to throw at it.
You already need to deal with 3 things as opposed to one and learn PromQL to boot. If you want to have dashboards you also need to learn and use Grafana. I used both Prometheus+Thanos+Grafana stack and Zabbix in production and can say with certainty that the latter is much easier to use and set up for infra monitoring than the former. You need a whole dedicated observability team to use Prometheus effeciently.
I'd disagree as someone running grafana+influx+telegraf for a few reasons.
Dealing with 3 things... feels about as much work as zabbix. That one requires server + db + agents. Basically the same components I'm running.
Promql/flux - while I know it, I almost never use it - instead in grafana click the database name, metric name, aggregation and I'm done.
While you need people to use it effectively, it's the same for everything, including mysql for zabbix. You can dump prometheus/influx somewhere with no configuration and survive for quite a long time.
Dealing with 3 things... feels about as much work as zabbix. That one requires server + db + agents. Basically the same components I'm running.
Promql/flux - while I know it, I almost never use it - instead in grafana click the database name, metric name, aggregation and I'm done.
While you need people to use it effectively, it's the same for everything, including mysql for zabbix. You can dump prometheus/influx somewhere with no configuration and survive for quite a long time.
Agree with this take, depending on just how simple a home setup is.
If you're going to be working with it on the job, Prometheus is relatively simple to provision, though, and can provide some experience if you use it at home.
If you're going to be working with it on the job, Prometheus is relatively simple to provision, though, and can provide some experience if you use it at home.
For server monitoring, Have a look at Prometheus, Grafana, Zabbix
I made my own monitor in 500 lines of code.
It uses a HTTP client to poll all instances every minute which return /proc/stat so that I can see how much CPU they use:
http://monitor.rupy.se
It uses a HTTP client to poll all instances every minute which return /proc/stat so that I can see how much CPU they use:
http://monitor.rupy.se
Since this post was interested in resource usage, I'm using telegraf for pushing system information (similar idea as collectd) and it's at 24kB of res memory. It reads from / writes to pretty much anything you can think of https://docs.influxdata.com/telegraf/v1.23/plugins/
Also a telegraf user here, been using it for ~5 years across 100+ machines, and it's been rock solid.
I find it very difficult to understand how "modern server monitoring" requires 6 or 7 components, when you can just deploy netdata on all your nodes with a 1-line installer and get way higher fidelity monitoring.
Is anyone using netdata in production? Has anyone looked at their data sharing/selling policies? How does it compare with other tools (https://github.com/louislam/uptime-kuma) ? Looks very impressive.
They aren't even in the same category though ( except the "monitoring" super-category ). Uptime-kuma monitors uptime only while netdata doesn't even (afaik) monitor uptime, it's mostly machine related metrics.
It monitors uptime too. Netdata has over 200 integrations, including all Prometheus endpoints. So yeah, it's not in the same category.
Push model is only great for short lived jobs (lambdas, cronjobs). Pull model scales much more easily if you’ve large number of servers in your fleet. Configuration is managed centrally, only the agent (like node-exporter) has to be running on the target node.
How do you deal with security? Are all nodes accepting connections from the monitoring node? With push, the monitoring node is exposed, but it is a less valuable target than every node in the network.
This is my main gripe with Prometheus (didn't use it yet).
This is my main gripe with Prometheus (didn't use it yet).
Exactly. I feel that monitoring solutions have to be the opposite of backup ones.
For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups.
But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
For backups, you ideally want the backup server to do a pull from the backed-up servers, to avoid that a security incident in one of them could damage your backups.
But for monitoring, I feel that push is the way to go. I want only the minimal, indispensable connections into my production servers.
For backups you can push with no security problems (unless you are doing a straight copy). If your backup client has rights to overwrite previous backups, that's a design problem.
Feels like some combination of VPCs and bastion edges would work
At what scale is that a problem? I imagine that a simple solution is to just have aggregators. Vector, for example, can ship logs to another instance of Vector. So you can have N endpoint Vectors that ship to N/K Vector aggregators. Your aggregators can buffer aggressively as well. Plus, if your data can be condensed, like metrics, I believe a Vector transform would work as well (but I'm not sure).
I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache.
This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.
I like this approach because, for security data, you want it off of the box as soon as possible. In an ideal world a log would go straight from the kernel to another box, or as close to that as possible (to avoid tampering/ DOS'ing). So for your security data you're already going to want to push that latency metric down, and now the question is what to do with the rest of them - obviously your service logs are less sensitive, but at the same time getting things shipped off of a box can save you a lot of headache.
This is hand wavy though, I'm honestly very curious to hear what others think as I haven't built the system I'm describing.
Part of what parent is describing is pushing configurations down to the agents from a centrally managed admin panel. I think this works well for basic system metrics but not for anything off the beaten path requiring integrations and scripts.
In response to your vector solution:. What I found highly adaptable is having all servers forward to a central vector/fluent-bit agent, have that agent forward to kinesis firehose, then attach a transform lambda to create the final output. You end up with 2 configuration points from which you can control and transform all your logs. Instead of having to push this down to the servers through config management.
In response to your vector solution:. What I found highly adaptable is having all servers forward to a central vector/fluent-bit agent, have that agent forward to kinesis firehose, then attach a transform lambda to create the final output. You end up with 2 configuration points from which you can control and transform all your logs. Instead of having to push this down to the servers through config management.
I always default to Netdata as it is batteries-included and can rarely outgrow it.
Other stacks can do the same, but I haven’t found anything as simple to set up; netdata is just a simple service.