Got it, yea seems this is more valuable in a big data context or when actively pressed up against scale limitations of a DB deployment. Funny on the the last piece as well - appreciate the thoughts here!
For the folks you worked with in the past that were excellent, were they full time DBAs in terms of their title? Or were they folks that were in other departments (i.e. devops) that also had expertise in good DB design in scaling?
It might also depend on the size of the company the particular need? Evolving a monolith-like DB already in production to a better structured state might be much to ask of an individual.
I think some of the administrative pain can come from having a 'sandbox' or 'qa' tier beyond dev (which in this case would be customer facing) before production.
If data separation isn't required/essential, seems as though a a separate environment is actually unnecessary and feature gating could be used. But if data separation from other customers is required, then new infrastructure should be introduced via separate CICD if I follow correctly?
Would this break down if you had 10 customers that would prefer the slow channel of release, but each had their own ever so slightly different desired release pace or training requirements?
Was the beta channel that you had in the pipeline for prod? i.e. prod would have no chance of getting ahead of beta because prod was always deployed based on what was last on the beta release?
Hot or bug fixing is an interesting topic because you could end up with diverging histories if you're not careful. Did you try to maintain `master` as the single source of truth or did you make separate release and deploy branches for each environment?
Thanks for the reply - yea these are great points to consider and raised a few of them. Ideally rollouts benefit all customers, and the concern is that there is some possibility that something might not jive with them and they want the ability to do testing ahead of time.
Seems like putting as many things that might risk this under settings/configurable as possible? This does extend development to have to gate everything in this way though.
Would you have the staging environment share data with production or have its own infrastructure? or use feature gating? One customer may not care that it shares the same DB, but another might.
How would this work at scale? say if there were say 10 customers wanting approval, you'd have to have separate code release branches/versions for each customer? I imagine this could get error prone.
I suppose to your point on charging additional fees for this you can afford any sort of workflow.
Most of these seem like fairly weak reasons at this point.
Some degree of vendor lock-in is pretty much inevitable no matter what solution you go with.
Limited programming languages may have been an issue a few years ago, but with the support of Java, Go, Python, Javascript to name a few - most of the core user bases are covered.
Additionally, after the introduction of API gateway, most CRUD applications, including those that require async tasks, can be served by the lambda and serverless.
After having used serverless for a while, the biggest turn off for using it (still) seems to be the performance issues. Provisioned concurrency, i.e leaving function containers running, is really just a bandaid and runs counter-intuitive to the original motivation of serverless to begin with.