Most places I land there's deference to the status quo and those that came before. People assume that efforts of that past were well considered and that good work was done.
But ultimately when pressed, there's generally no evidence of the above, and when there is evidence, one can generally demonstrate the flaws in the approach, so I press on anyway and it hasn't really bitten me yet.
Of course, in many of these projects there's now deference to decisions I've made. Some of those decisions were well made, with evidence and supporting work, to make them challengeable should the understanding changes. Some of them are gut checks because they didn't matter to me at the time.
Either way the take away is good, but often practices exist simply to frustrate.
I actually don't have a problem with runners very often, but I must admit I use a combination of public and private the leans private. Are you seeing elevated errors on actions?
Whilst that's crazy, github hasn't migrated to azure yet, so it's probably not exclusively an azure problem and we've seen the same problems with amazon too ( and several other large orgs ).
I worked on a small part of one of these back in around 2013 ( specifically managing beds ).
You were talking about a team of 5 cranking this out in about 2-3 months with some longer term part time involvement, with an annual cost of less than 1m and those people mostly all dellivering several product lines ( so actual cost is half or a quater ).
Our old jenkins hosts were largely forever instances with forever credentials that were just waiting to take down the org.
Modern pipelines are orchestrates that run ephemeral execution environments with ephemeral credentials that can significantly decrease the impact and timescales of getting pwned.
They're not perfect, but you can get pretty good posture by applying expertise to the subject. The problem, like always, is this expertise is neither valued nor rewarded.
Reproducible builds shows that, within a specific configuration, the code produced the binary, regardless of who signed or published it.
Indeed, this could mitigate an attacker replacing the binary with something that's not produced from the code, but it does not mitigate the tool chain or code itself containing the exploit, creating a malicious binary.
I wrote a scheduler for VMs a long time ago. k8s is basically just the same thing but for containers.
It really confuses me how someone can argue for cloud providers over a decent open solution without realising their argument is simply they don't want to be managing the thing.
And that's fine, most teams shouldn't be neck deep in managing a platform. But that doesn't make the solution bad.
I took over tech for a POS company some years ago. They were a .net shop with about 80 developers, less than 200 concurrent connections, 6 figures spend cloud, and 0 nines uptime with a super traditional setup.
Point being, it's not the tools the causes the probem.
The deployment files / structure were mostly equivalent with the main differences being I can't shell into ECS and I lose kubectl in favour of looking at the AWS GUI ( which for me is a loss, for others maybe not ).
The main difference is k8s has a lot of optionality, and folks get analysis paralysis with all the potential there. You quickly hit this in k8s when you have to actually need the addon to get cloudwatch logs.
This is also where k8s has sharp edges. Since amazon takes care of the rest of the infrastructure for you in ECS, you don't really need to worry about contention and starving node resources resulting in killing your logging daemon, which you could technically do in k8s.
However, you'll note that this is a vendor choice. EKS Auto Mode does away with most of the addons you need to run yourself, simplifying k8s, moving it significantly closer to a vendor supported solution.
My app is fairly simple node process with some side car worker processes. k8s enables me to deploy it 30 times for 30 PRs, trivially, in a standard way, with standard cleanup.
Can I do that without k8s? Yes. To the same standard with the same amount of effort? Probably not. Here, I'd argue the k8s APIs and interfaces are better than trying to do this on AWS ( or your preferred cloud provider ).
Where things get complicated is k8s itself is borderline cloud provider software. So teams who were previously good using a managed service are now owning more of the stack, and these random devops heros aren't necessarily making good decisions everywhere.
So you really have three obvious use cases:
a) You're doing something interesting with the k8s APIs, that aren't easy to do on a cloud provider. Essentially, you're a power user.
b) You want a cloud abstraction layer because you're multi-cloud or you want a lock-in bargaining chip.
c) You want cloud semantics without being on a cloud provider.
However, if you're a single developer with a single machine, or a very small team and you're happy working through contended static environments, you can pretty much just put a process on a box and call it done. k8s is overkill here, though not as much as people claim until the devops heros start their work.
Nobody really tries to get technical people to do the work.
Like cool, it's a great idea and would potentially produce positive results if done well, but the roles pay half the engineering roles, and the interviews are stacked towards compliance frameworks.
There's very little ability to fix a large public company when HR is involved
1. Users - your users table may not benefit by being ordered by created_at ( or uuid7 ) index because whether or not you need to query that data is tied to the users activity rather than when they first on-boarded.
2 Orders - The majority of your queries on recent orders or historical reporting type query which should benefit for a created_at ( or uuidv7 ) index.
Obviously the argument is then you're leaking data in the key, but my personal take is this is over stated. You might not want to tell people how old a User is, but you're pretty much always going to tell them how old an Order is.
Most places I land there's deference to the status quo and those that came before. People assume that efforts of that past were well considered and that good work was done.
But ultimately when pressed, there's generally no evidence of the above, and when there is evidence, one can generally demonstrate the flaws in the approach, so I press on anyway and it hasn't really bitten me yet.
Of course, in many of these projects there's now deference to decisions I've made. Some of those decisions were well made, with evidence and supporting work, to make them challengeable should the understanding changes. Some of them are gut checks because they didn't matter to me at the time.
Either way the take away is good, but often practices exist simply to frustrate.