Show HN: pgwire-replication - pure rust client for Postgres CDC(github.com)
github.com
Show HN: pgwire-replication - pure rust client for Postgres CDC
https://github.com/vnvo/pgwire-replication
9 comments
> Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure.
Would I use this if I host my own postgres and want to use replication for „real time backups“ into a hot standby?
Would I use this if I host my own postgres and want to use replication for „real time backups“ into a hot standby?
Not OP - but definitely no. In such a case use just physical or - if you have special use case - logical replication that’s built in.
I agree. pgwire-replication is useful when you need to build a customized and closely controlled pipeline. In fact, it will give you the first part of handling the data (reading from the source), you still need to implement the rest yourself.
Nice one, great to see this addition to the Rust ecosystem!
Reading through the README, this piqued my curiosity:
> Small or fast transactions may share the same WAL position.
I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN.
> LSNs should be treated as monotonic but not dense.
That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.
Reading through the README, this piqued my curiosity:
> Small or fast transactions may share the same WAL position.
I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN.
> LSNs should be treated as monotonic but not dense.
That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.
Good catch, you are correct. I did mix a few things there and the statements were incorrect or at least very misleading.
To demo your point I created a gist, for myself and others to see the (commit-LSN, event-LSN) ordering in action:
https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9...
I'll update the readme to reflect this more accurately. Appreciate you taking the time to point it out.
To demo your point I created a gist, for myself and others to see the (commit-LSN, event-LSN) ordering in action:
https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9...
I'll update the readme to reflect this more accurately. Appreciate you taking the time to point it out.
This is just the transport - raw XLogData frames and LSNs. Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure.
What it does:
- Explicit LSN control - start/stop at specific WAL positions for deterministic recovery
- Automatic standby feedback - no more forgotten heartbeats filling your disk with WAL
- Bounded channels - backpressure propagates to Postgres naturally
- Pure Rust, no libpq
What it doesn't do: pgoutput decoding (intentionally). That belongs in a higher layer. Simplest way of using this:
while let Some(event) = client.recv().await? { match event { ReplicationEvent::XLogData { wal_end, data, .. } => { process(&data); client.update_applied_lsn(wal_end); } _ => {} } }