Improving Observability of Golang Services(flow.com)
flow.com
Improving Observability of Golang Services
https://flow.com/engineering-blogs/golang-services-improving-observability
4 comments
This blog post is aimed at GoLang developers who are looking to improve the observability of their services. It skips the basics and jumps straight to advanced topics, such as asynchronous structured logging, metrics with exemplars, tracing with TraceQL, aggregating pprof and continuous profiling, microbenchmarks and basic statistics with benchstat, blackbox performance tests, and basic PID controllers for determining a system's maximum load. We'll also briefly touch on current research in the observability space, including active casual profiling and passive critical section detection.
Great, well-researched write-up with practical examples. Well done - Alexey Ivanov!
There are quite a few different options described in the paper. And I guess it was a long journey to find a solution which you are happy with.
Which configuration you ended up with?
We're mostly on the Grafana stack, so we tend to favor their open-source solutions for metric backends. For client libraries, we're currently using Prometheus for metrics and OpenTelemetry (OTEL) for tracing, but we're looking to migrate fully to OTEL once metrics are stable there. As for logging, it depends on the project, but generally we use either Zerolog or Zap, with either Loki Docker plugin or Promtail.