I find it interesting that many people seem to conflate the complexity of managing infrastructure and services with K8s.
K8s is complex because managing distributed services is. Not using it doesn't mean it goes away. The complexity migrates and ends up being bundled up in a separate tool or a runbook process or some script.
It's hard to maintain because the tools and apis are different from what some engineering teams are accustomed to using. Building an in-house tool gives them a warm fuzzy feeling and comfort that they can handle problems when they appear due to familiarity with their own code and design choices.
It's a fair trade off. I do wonder how much of the time spent doing this exercise could have been spent on K8S training.
I do feel that the K8S community do downplay how much a PITA k8s configuration can be and that the perceived robustness of cloud-managed K8S isn't up to scratch for something this complex.
Your standards aren't too high but I think you must realise that this is a cultural problem with little hope of changing. Even if the push comes from the CTO, it will take years for change to happen, it will require new hires and bringing new blood into the engineering leadership.
If you do want to take the challenge (which I strongly discourage you from) you'd need to collect data to build your case, quantify the time and human cost from issue/jira to code landing in prod to number of incidents/bugs. The instrumentation to do this will be a fairly chunky piece of devops work. Frame the data in light of your competitor's ability to iterate their products and so on. When it's collected and presented it can be quite compelling and people will listen.
It's only at this point you'll be able to present the problem to management in way that they understand. You know and I know that this is a cultural problem first, then a process problem and lastly a technology problem. The amount of work to effect this kind of organisational change, even in a small engineering company is immense. I don't know your motivations are for staying, if it's the domain or the money but if this something that bothers you then this is the best piece of advice I can give you:
I'm referring to having the same testing, deployment,packaging,versioning policies etc being consistently applied across projects within the same repository not deploying, testing and releasing together.
It's the drift and inconsistencies across these concerns across projects that makes deployment and operations less predictable.
Without knowing more about their architecture it is difficult to comment beyond the conclusion Alexandra Noonan came to, stated at the beginning of the article. It looks like to me that the architectural assumptions were changing too quickly due to the demands of a fast growing business. Having all their code in a single repository means that they can control dependencies, versioning and deployment centrally, it gives them central control of their software development lifecycle. I can't see how they could not have had the same benefits of the monolith if their microservices existed in a single repo to begin with and had the appropriate tooling to enforce testing, versioning, deployment across all services in the repo. I guess this is the whole monorepo debate and tooling.
This article for me is more about the complexity of managing a large team across different sites where the architecture needs to change rapidly when modularity is absent. They did get a measurable benefit around performance, though. I wonder if Alexandra will comment on the challenges of running a team in an environment of this complexity?
K8s is complex because managing distributed services is. Not using it doesn't mean it goes away. The complexity migrates and ends up being bundled up in a separate tool or a runbook process or some script.
It's hard to maintain because the tools and apis are different from what some engineering teams are accustomed to using. Building an in-house tool gives them a warm fuzzy feeling and comfort that they can handle problems when they appear due to familiarity with their own code and design choices.
It's a fair trade off. I do wonder how much of the time spent doing this exercise could have been spent on K8S training.
I do feel that the K8S community do downplay how much a PITA k8s configuration can be and that the perceived robustness of cloud-managed K8S isn't up to scratch for something this complex.