This is the approach we took while building our Internal Developer Platform: watches (via client-go informers with client-side caching) to sync data into a Postgres database as JSONB. Changes are tracked using JSON patches and Kubernetes events. To avoid a watch on every resource kind, we handle this by performing incremental object fetches for the objects involved in watched events.
Getting this to perform well required several optimizations at both the Go and Postgres levels. On the Go side, we use prioritized work queues, event de-duplication, and even switched to Rust for efficient JSON diffs. For Postgres, we leverage materialized views and trigger-based optimistic locking
Flanksource | Founding Backend (Go), Frontend (React), SRE (Kubernetes) and Staff/Principal | Full Time or Contract | Remote First
Flanksource is building a test-driven incident response platform that enables teams to communicate effectively through real-time system models, reducing the broken telephone, "Please re-test" feedback loops that plague organizations with multiple teams.
SEEKING FREELANCER | flanksource | SRE | REMOTE first
flanksource is a niche consultancy focusing exclusively on Kubernetes and the Cloud Native ecosystem. We help companies navigate the CNCF landscape by evaluating and integrating technology into an infrastructure continuous delivery pipeline, tailored to each customer's particular environment and stack.
We are looking for thinkers with a bias for action, who love the Kubernetes community as much as we do.
Benefits:
- Remote first culture - work from anywhere in the world
- Flexible working hours - Choose both when and how much you work, we accommodate full-time employees/contractors, part-timers and weekend warriors
- Paid contribution to open-source projects in and around the kubernetes eco-system
- Annual development budget to spend on conferences, books, courses, etc..
Flanksource is a niche consultancy focusing exclusively on Kubernetes and the Cloud Native ecosystem. We help companies navigate the CNCF landscape by evaluating and integrating technology into an infrastructure continuous delivery pipeline, tailored to each customer's particular environment and stack.
We are looking for long-term part-time engineers to increase diversity and fill out our on-call schedules.
Getting this to perform well required several optimizations at both the Go and Postgres levels. On the Go side, we use prioritized work queues, event de-duplication, and even switched to Rust for efficient JSON diffs. For Postgres, we leverage materialized views and trigger-based optimistic locking