We've been using the Ubicloud runner for a while at PeerDB[1]. Great value and specially the ARM runners have been helpful to get our CI costs down. The team is really responsive and added the arm runner support within a few weeks of us requesting it.
Author of the blog here: I'm a huge proponent of Rust, but in this project while interfacing with BigQuery and Snowflake go proved to be the right choice. There weren't official drivers for these in Rust and also generally onboarding new devs in Go was easier. I also personally think async in rust has a few more releases to go before I would consider it stable.
To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.
Not all of our code is in Go. PeerDB has multiple components: the workers, the UI and the query layer. Some of which is in Rust, Go and Typescript.
While it would certainly be possible to package them into individual binaries, I found it significantly easier to define the stack in a docker compose file with the requisite environment setup.
Author of the blog here, curious what a better alternative would be in this context. The channel has to be passed around for the producer and consumer to interface with each other. Are there better patterns for this?
Hi there, I’m Kaushik, one of the co-founders of PeerDB. PeerDB doesn’t handle schema changes today.
For CDC, change stream does give us events in case of schema changes, we would have to replay them on the destination. Schema changes on the destination are not supported, the general recommendation is to build new tables / views and let PeerDB manage the destination table.
For streaming the results of a query, as long as the query itself can execute (say a few columns were added or untouched columns were edited) mirror job will continue to execute. In case this is not the case, there will be some manual intervention needed to account for the schema changes.
Thanks for the question, this is a requested feature and on our roadmap.
[1] https://github.com/PeerDB-io/peerdb