We’ve just released Uptrace v2.0, a major re-architecture of our open-source observability platform (traces, logs, metrics).
This release focuses on performance and scalability:
- 5–10x faster queries using ClickHouse’s new JSON column type (nested fields can be queried directly with dot-notation, no ETL or flattening required)
- Multi-project support – manage projects, users, and orgs via UI or YAML seed files
- Real-time data transformations (rename, filter, enrich spans/logs, sample dynamically) using an [Expr-based](https://github.com/expr-lang/expr) scripting system
- Redesigned dashboards and query builder for faster incident response and exploration
- Handles massive traces (10k–1M spans) with tunable resource limits
It’s still open-source and free to self-host. We’ve published migration guides, Ansible and Kubernetes deploy instructions, and full [release details](https://uptrace.dev/blog/uptrace-v20.html).
Would love feedback from the HN crowd — especially from folks scaling ClickHouse or building custom observability pipelines.
Uptrace is an open-source APM tool that supports distributed tracing, metrics, and logs. You can use it to monitor applications and set up alerts to receive notifications via email, Slack, Telegram, and more.
In v1.4 release Uptrace got built-in alerting capabilities that allow to monitor metrics, manage alerts, and receive notifications without using AlertManager. But AlertManager is still supported.
While OpenTelemetry remains to be the main source of telemetry data, Uptrace also provides integrations for Prometheus metrics, Vector logs, FluentBit, Sentry, and CloudWatch.
I use memory profiling with Go and it is indeed very useful. I think that whatever Go already provides is enough, but I guess Uptrace could try to automate some things and/or provide some fancy UI.
But I find CPU profiling a lot less useful, because production profiles tend to be too broad and it is hard to make sense from them, for example, Uptrace profile mostly consists of memory allocations and network calls.
So I would not say that CPU profiling is superior/better/can replace tracing.
When compared with ElasticSearch, ClickHouse can handle the same amount data using 10x less resources and that is not an exaggeration. It is even worse with MariaDB/MySQL/PostgreSQL.
I guess ElasticSearch is still relevant when it comes to searching text, but ClickHouse is much faster when it comes to filtering and analyzing the data.
Give ClickHouse a try and you won't be disappointed.
My understanding is that APM became or always was a marketing term which is used rather freely. For that reason I try to avoid it, but search engines love it and I don't know a better alternative.
>Whether I can find out why that component is slow might not be as easy (not sure what granularity tracing happens inside a component).
It is true that you can't always guess what operation going to be slow and instrument it, but it is almost always a network or a database call. There is still no way to tell *why* it is slow, but the more data you have the more hints you get.
>What's the usual overhead of a single tracing
Depending on what is your base comparison point the answer can be very different.
Usually, you trace or instrument network/filesystem/database calls and in those cases the overhead is negligible (few percents at most).
>But does that mean taking a trace is very cheap?
What you've described is tail-based sampling and it only helps with reducing storage requirements. It does not reduce the sampling overhead. Check https://uptrace.dev/opentelemetry/sampling.html
But is taking a trace cheap? Definitely. Billions of traces? Not so.
>request sampling decision something that's common or that's a totally unique capability some have.
It is a common practice to reduce cost of sampling when you have billions of traces, but it is an advanced feature because it requires backends to buffer incoming spans in memory so you can decide if the trace is anomalous or not.
Besides, you can't store only anomalous traces because you will lose a lot of useful details and you can't really detect anomaly without knowing what is the norm.
Uptrace uses the BSL license to forbid or rather not allow other companies creating a cloud service using Uptrace code, because that is how we are planning to monetize. But you can self-host Uptrace and use it as you want to monitor your (production) application. I think this is fair enough.
If there is another license that better reflects our intentions, let us know.
I am not a lawyer so I can't keep up the discussion on the necessary level so I will just clarify few things.
Uptrace uses the BSL license to forbid or rather not allow other companies creating a cloud service using Uptrace code, because that is how we are planning to monetize. But you can self-host Uptrace and use it as you want to monitor your (production) application. I think this is fair enough.
I am sure there are many complications with re-licensing, but it happens in practice, for example, Sentry now uses BSL license. And you can't do anything about it except to fork old Sentry. But then you will have to maintain it yourself.
And I am not arguing with anything you've said, but there are not that many financially thriving truly OSS projects. That's why people like me have to complicate their lives with BSL, AGPL, and others.
You can self-host Uptrace and use it in production environment. This is explicitly stated in the FAQ - https://github.com/uptrace/uptrace#faq . But you can't resell Uptrace to others.
I can only repeat that the same BSL license is used by MariaDB, Sentry, CockroachDB, Couchbase, and others. I am not a lawyer and thus not qualified to discuss details.
It is true for all licenses, for example, it is possible to keep old code available under old permissive license but release all new code under a more restricted license.
We already use JWT tokens that are passed in a HTTP cookie so perhaps we could document how it works and let users sign JWT tokens themselves. That way your app only needs to set a cookie and the user should be authorized.
It is the same license used by MariaDB, Sentry, CockroachDB, Couchbase and many others. Technically it is not open source and instead is called source available, but you can enjoy pretty much the same benefits.
Out of curiosity, what makes you uncomfortable about the license?
DataDog has a high learning curve and can be rather expensive if you need to monitor a lot of hosts and microservices.
Uptrace tries hard to stay simple while providing almost the same set of features. It can also be self-hosted without paying anything which can save a lot of money.
Uptrace aims to be an open source alternative to DataDog, but realistically we are not there yet.
>- Are there any recommendations for scaling (e.g. benchmarks) on how many spans/s are supported on what hardware?
With Uptrace, I was able to achieve 10k spans / second on a single core by running the binary with GOMAXPROCS=1. That is 1-3 terrabytes of compressed data each month which is more than most users need.
Practically, you are limited by the $$$ you are willing to spend on ClickHouse servers, not by Uptrace ingestion speed.
So my recommendation is to scale Uptrace vertically by throwing more cores at it. That will allow you to go very very far.
>- Is there support for SSO for self-hosted installation?
So far the only way to add news users is via the YAML config. We are considering to add a REST API or a CLI tool for the same purpose, but it is not clear how that would work with the YAML.
Regarding the SSO, it would be nice if you can provide an app that already does that so we can better estimate the complexity. But so far we don't have such plans.
>- Can I easily disable certain features? (e.g. alerting)
Yes, most YAML sections just can be removed / commented out to disable the feature.
You can ingest data using OpenTelemetry Protocol (OTLP), Vector Logs, and Zipkin API. You can also use OpenTelemetry Collector to collect Prometheus metrics or receive data from Jaeger, X Ray, Apache, PostgreSQL, MySQL and many more.
The latest Uptrace release introduces support for OpenTelemetry Metrics which includes:
- User interface to build table-based and grid-based dashboards.
- Pre-built dashboard templates for Golang, Redis, PostgreSQL, MySQL, and host metrics.
- Metrics monitoring aka alerting rules inspired by Prometheus.
- Notifications via email/Slack/PagerDuty using AlertManager integration.
So far my experience is that it is best to avoid trying to solve such problems with a query language and instead provide a much simpler UI to achieve the same. Solving such problems with SQL is tedious enough and learning another custom language is not fun / too much to ask from users.
Sometimes using a UI is not possible, for example, if you want to automate such checks. In that case, I would build a custom metric or two and would use that metric for monitoring purposes. That requires some programming / instrumentation, but it still looks like a better solution to me.
Thanks for the answer - that all makes sense... "but" :) there is also some sense in NOT having to support different storages / plugins API and instead supporting more features that work out of the box. At least I hope so.
I will try Grafana Tempo & Loki - thanks for reminding. It is just that all their products look the same like the "original" Grafana for metrics which TBH does not look especially ergonomic too... Somehow Jaeger still comes first when people talk about tracing.
This release focuses on performance and scalability:
- 5–10x faster queries using ClickHouse’s new JSON column type (nested fields can be queried directly with dot-notation, no ETL or flattening required)
- Multi-project support – manage projects, users, and orgs via UI or YAML seed files
- Real-time data transformations (rename, filter, enrich spans/logs, sample dynamically) using an [Expr-based](https://github.com/expr-lang/expr) scripting system
- Redesigned dashboards and query builder for faster incident response and exploration
- Handles massive traces (10k–1M spans) with tunable resource limits
It’s still open-source and free to self-host. We’ve published migration guides, Ansible and Kubernetes deploy instructions, and full [release details](https://uptrace.dev/blog/uptrace-v20.html).
Would love feedback from the HN crowd — especially from folks scaling ClickHouse or building custom observability pipelines.
Links:
- Blog & details: https://uptrace.dev/blog/uptrace-v20.html
- Install guide: https://uptrace.dev/get/hosted/install
- GitHub: https://github.com/uptrace/uptrace