Pg_tracing: Distributed Tracing for PostgreSQL(github.com)
github.com
Pg_tracing: Distributed Tracing for PostgreSQL
https://github.com/DataDog/pg_tracing
15 comments
It would be awesome if this were to make it in core. Jelte did a bunch of work to improve query protocol changes. Having Postgres accept Otel trace propagation would be a gamechanger for debugging the clients and DB together. Especially since ~80%+ of DB related issues I see come down to some sort of long running open transaction which Postgres knows nothing about server-side.
This looks very useful for our database heavy teams.
Getting this information is certainly already possible, but there is a bit of a barrier in front of it. You need to realize the query is slow, then you need to re-run it with the right EXPLAIN and/or ANALYZE incantation with 8-9 parameters for a query visualizer, paste it into a query visualizer and then you get some nice, easily digested overview of what is going on.
Teams either don't know how to do that, or don't do that, due to permissions or because it's a hassle. Having a slow "calculateFooReport()" trace go straight into a bunch of slow SequentialScan- and NestedLoop-Nodes would remove one excuse from that equation.
Kinda bummed that we're updating out of the supported versions starting next month.
Getting this information is certainly already possible, but there is a bit of a barrier in front of it. You need to realize the query is slow, then you need to re-run it with the right EXPLAIN and/or ANALYZE incantation with 8-9 parameters for a query visualizer, paste it into a query visualizer and then you get some nice, easily digested overview of what is going on.
Teams either don't know how to do that, or don't do that, due to permissions or because it's a hassle. Having a slow "calculateFooReport()" trace go straight into a bunch of slow SequentialScan- and NestedLoop-Nodes would remove one excuse from that equation.
Kinda bummed that we're updating out of the supported versions starting next month.
Great idea. Currently, people have to rely on client-side spans in OpenTelemetry. However, it would be awesome if we could get spans for slow SQL queries, along with explanations.
It looks abandoned.
I've had this with multiple datadog libraries. Even offered to implement some features for the python one for free because we were missing a feature but they just never responded. Note that they had this feature in their php lib so it was not some outrageous demand.
Or complete?
> pg_tracing only supports PostgreSQL 14, 15 and 16 for the moment.
PostgresSQL is already at 18.
PostgresSQL is already at 18.
The last couple commits were about pg18 support. Maybe development stalled, but didn't completely stop
Also, tracing support is being upstreamed into Postgres proper https://github.com/DataDog/pg_tracing/issues/86 which would make this extension obsolete
Also, tracing support is being upstreamed into Postgres proper https://github.com/DataDog/pg_tracing/issues/86 which would make this extension obsolete
Unlikely, as it states:
Warning
This extension is still in early development and may be unstable.
Warning
This extension is still in early development and may be unstable.
Is someone from Sentry here? It’d be amazing if you’d pick this up and extend your tracing into Postgres…
There's nothing "distributed" about it (the only word that got me click).
The database you send the traces to may be distributed, just like any database, but pg_tracing does nothing for it.
The database you send the traces to may be distributed, just like any database, but pg_tracing does nothing for it.
Distributed tracing is a term for tracing that span between systems. With distributed tracing you can follow request traces across your various services. With pg_tracing enabled in postgres this would extend to your database.
Yes, I know I've been using it for a while.
The hard part is to merge one trace between storage workers, that's where distributed part comes. pg_tracing does nothing for it.
UPDATE: ah I believe I see what you mean -- that it passes down the trace ID.
The hard part is to merge one trace between storage workers, that's where distributed part comes. pg_tracing does nothing for it.
UPDATE: ah I believe I see what you mean -- that it passes down the trace ID.
Yeah, this is for integration with OTEL standard distributed tracing. So you configure some OTEL compliant endpoint for postgres to send trace spans to and that endpoint will collect traces from different services and store them for later retrieval.
You have a service that talks to postgres, probably other services that talk to that service, a client that talks to these services through some gateway. When a user clicks a button, distributed tracing allows you to see the whole request right from the button click to API gateway, to every service it goes through to the DB.. and all the way back. So you can see exactly the path it took, where it slowed down or failed. DBs are usually seen as black boxes in such systems. We just see the DB call took 500ms. We don't know what happened inside though. This allows such distributed traces to also get visibility to the internals of the DB so you can tell the exact joins, index scans etc that took place by looking at your distributed trace. I don't know what level of visibility they've built but that's the general idea.