Journald messages have almost infinite cardinality at their labels and all of them are indexed, even if every single message has unique fields and values.
When you send journald logs to Loki, even if you use the `relabel_rules` argument to `loki.source.journal` with a JSON format, you need to specify which of the fields from journald you want inherited by Loki. This means you loose all the flexibility journald provides: indexing on all fields and all their values.
Loki generally assumes that all logs are like a table. All entries in a stream share the same fields. But journald does exactly the opposite. Each log entry is unique and may have its own unique fields.
Of course someone could use filters and relabel rules to create multiple streams of logs out of a single journald stream (never tried it), but it sounds a lot of work and again assumes you know all the possible fields beforehand.
So, Loki and systemd-journal are good for different use cases. The good thing with systemd-journal, is that you already have it and use it. It is there inside your systems.
Do not compare Netdata with other monitoring solutions that centralize everything to one place, or with single installation applications.
Netdata is a distributed application, and it is installed all over the place. So we needed to find a way to provide SSO.
There are a few alternatives:
1. PAM (then LDAP or a DB), but this would significantly increase the attack surface of your Network, making Netdata an ideal component to test your security. We didn't want this.
2. LDAP, similar to the above and increased complexity. Probably too complex for the average user out there, and it would over-complicate things when you need to run Netdata in private and public clouds concurrently.
We chose to provide a free service to everyone using Netdata, where we manage all this complexity and simplify the process.
Netdata Cloud uses Google SSO, Github SSO, and email verification to authenticate users. It does not store user passwords. Combined with the claiming process of the Netdata Agents:
a) it ensures you are the admin of each server you want to manage
b) it verifies your identify
c) it provides centralized control on who of the authenticated users has access to your servers.
What happens when you use Netdata Cloud to access a Netdata agent, is that your web browser asks from Netdata Cloud to access this Netdata agent, Netdata Cloud verifies you and if this succeeds and you have trusted the agent before, it asks the agent (via their link) to generate a unique token for you, which is sent back to your browser and is then used as an authorization bearer to access the agent directly. So, your data do not flow through Netdata Cloud. You only get a token from the agent, via Netdata Cloud.
Netdata v1.39 just released, introducing a major change in monitoring. Now Netdata not only trains multiple ML models for each and every metric collected, but also Netdata Charts use an ML-first approach, revealing all the power of ML instantly!
> if the number of people required to maintain netbase is low (relatively speaking that is)
The Netdata agent is a robust and mature product. We maintain it and we constantly improve it, but:
- most of our efforts go to Netdata.Cloud
- most of the action in the agent is in internal forks we have. For example, we are currently testing ML at the edge. This will eventually go into the agent, but is not there yet. Same with EBPF. We do a lot of work to streamline the process of providing the best EBPF experience out there.
> I can see them not really needing to worry about making money
We are going to make money on top of the free tier of Netdata.Cloud. We are currently building the free tier. In about a year from now we will start introducing new paid features to Netdata.Cloud. Whatever we will have released by then, will always be free.
> I'm guessing they are finding value in gathering data for ML
No, we are not gathering any data for any purpose. Our database is distributed. Your data are your data. We don't need them.
The same way GitHub, Slack or Cloudflare provide massively free-forever SaaS offerings while making money.
We believe that the world will greatly benefit by a monitoring solution that is massively battle tested, highly opinionated, incorporating all the knowledge and experience of the community for monitoring infrastructure, systems and applications. A solution that is installed in seconds, even mid-crisis and is immediately effective in identifying performance and stability issues.
The tricky part is to find a way to support this and sustain it indefinitely. We believe we nailed it!
So, we plan to avoid selling monitoring features. Our free offering will never have a limit on the number of nodes monitored, the number users using it, the number of metrics collected, analyzed and presented, no limit on the granularity of data, the number of war-rooms, of dashboards, the number of alarms configured, the notifications sent, etc. All these will always be free.
And no, we are not collecting any data for ML or any other purpose. The opposite actually: we plan to release ML at the edge, so that each server will learn its own behavior.
We plan to eventually sell increased convenience features, enforcement of compliance to business policies and enterprise specific integrations, all of them on top of the free offering.
We complement the Netdata agent with Netdata.Cloud, a free-forever SaaS offering that maintains all the principles of the Netdata agent, while providing infrastructure level monitoring and several additional convenience features.
In Netdata.Cloud, infrastructure is organized in war-rooms. On each war-room you will find the "Overview" page, that provides a fully automated dashboard, very similar to the one provided by the agent, in which every chart presented aggregates data from all servers in the war-room! Magic! Zero configuration! Fully automated!
Keep in mind that Netdata.Cloud is a thin convenience layer on top of the Netdata agent. We don't aggregate your data. Your data stay inside your servers. We only collect and store a few metadata (how many netdata agents you have, which metrics they collect, what alarms have been configured, when they triggered - but not the actual metric and log data of your systems).
Thank you for this feedback. I am the founder of Netdata.
Netdata is about making our lives easier. If you need to tweak Netdata, please open a github issue to let us know. It is a bug. Netdata should provide the best possible dashboards and alerts out of the box. If it does not for you, we missed something and we need your help to fix it, so please open a github issue to let us know of your use case. We want Netdata to be installed and effectively used with zero configuration, even mid-crisis, so although tweaking is possible and we support plenty of it, it should not be required.
An "incident" is a way to organize people, an issue management tool for monitoring, a collaboration feature. Netdata's primary goal however, is about exploring and understanding our infrastructure. We are trying to be amazingly effective in this by providing unlimited high resolution metrics, real-time dashboards and battle tested alarms. In our roadmap we have many features that we believe will change the way we understand monitoring. We are changing even the most fundamental features of a chart.
Of course at the same time we are trying to improve collaboration. This is why Netdata.Cloud, our free-forever SaaS offering that complements the open-source agent to provide out of the box infrastructure-level monitoring along side several convenience features, organizes our infra in war-rooms. In these war-rooms we have added metrics correlation tools that can help us find the most relevant metrics for something that got our attention, an alarm, a spike or a dive on a chart.
For Netdata, the high level incident panel you are looking for, will be based on a mix of charts and alarms. And we hope it is going to be also fully automated, autodetected and provided with zero configuration and tweaking. Stay tuned. We are baking it...