DevOps, SRE, and Platform Engineering(iximiuz.com)
iximiuz.com
DevOps, SRE, and Platform Engineering
https://iximiuz.com/en/posts/devops-sre-and-platform-engineering/
99 comments
I have seen organisation where platform teams build tools, frameworks and nobody uses them. Few places where it makes sense is when an organisation is large and has corporate and regulatory policies that applications and people have to adhere to.
As platform team member, you really have to know what to build, doing interviews, figuring out what developers and SREs really want, filter through what they are asking for, control through not-invented here syndrome, make build-vs-buy decision, create the community around the product you build within the organisation etc. You cannot just start building a PaaS (most common thing I have seen them build). It is actually quite complex if done right. Having said that, lot of organisation the definition is quite vague, political too and sometimes the boundaries are not clear.
As platform team member, you really have to know what to build, doing interviews, figuring out what developers and SREs really want, filter through what they are asking for, control through not-invented here syndrome, make build-vs-buy decision, create the community around the product you build within the organisation etc. You cannot just start building a PaaS (most common thing I have seen them build). It is actually quite complex if done right. Having said that, lot of organisation the definition is quite vague, political too and sometimes the boundaries are not clear.
Our systems team is veering into the first sentence hard. Tried to make a whole "one size fits all" CI/CD CLI tool that is inflexible and obfuscates the process.
Nobody asked for it and everyone is content rolling their own pipelines by hand, yet they've sunk a ton of time into it.
Nobody asked for it and everyone is content rolling their own pipelines by hand, yet they've sunk a ton of time into it.
You are not alone my friends. I have seen this happen over and over again.
I do a lot of this kind of work on the consulting side, and the interview process is critical to establishing buy-in from the essential stakeholders. You have to understand the pain points because without that, you won’t build the right thing for the way the company operates.
The most common hurdle is simply that most developers buried in enterprise software dev have no familiarity with the cloud or foundational DevOps tech like infra as code. So we typically end up building some form of deployment mechanism cobbled together with packer and terraform, but the real value in bringing in outside help is that we handle training and change management necessary to actually drive adoption.
“Build it and they will come” is a terrible design philosophy. You can’t just stand up a platform team and expect it to go well: a more transformational mindset inclusive of org design and strategic outlook is required, and this type of DevOps / SRE transformation is no less of a radical change than HR or finance transformation programs.
The most common hurdle is simply that most developers buried in enterprise software dev have no familiarity with the cloud or foundational DevOps tech like infra as code. So we typically end up building some form of deployment mechanism cobbled together with packer and terraform, but the real value in bringing in outside help is that we handle training and change management necessary to actually drive adoption.
“Build it and they will come” is a terrible design philosophy. You can’t just stand up a platform team and expect it to go well: a more transformational mindset inclusive of org design and strategic outlook is required, and this type of DevOps / SRE transformation is no less of a radical change than HR or finance transformation programs.
As a platform engineer, that's quite a cynical take you have there. What I've found works best is to write tools that take away complexity so that teams don't all need to reinvent the wheel. Then make it so well-documented and easy to use that teams don't need to be _forced_ to use the platform tooling, they'll just use it because it's the most convenient thing to do.
> What I've found works best is to write tools that take away complexity so that teams don't all need to reinvent the wheel.
The problem is complexity is variable. For developers who never really learn how their code actually executes, or haven't had to be responsible for tuning and such, this is great, but for those of us who want to know where the rubber meets the road, these tools are kid-gloves. How can we fully optimize and support our code if we can't see the compute on which it runs? What other "benefits" will the system bring that we have to account for?
> Then make it so well-documented and easy to use that teams don't need to be _forced_ to use the platform tooling, they'll just use it because it's the most convenient thing to do.
Very optimistic. In my experience with this kind of thing with homegrown platforms, documentation was a distant last, because it always is, and we weren't left with other choices anyway. I would love to pick the best tool because it is convenient, but that, too, is variable. Maybe I know Kubernetes inside and out - that's plenty convenient for me, then, and will be better documented and discussed than any homegrown platform because the world can contribute to it.
The problem is complexity is variable. For developers who never really learn how their code actually executes, or haven't had to be responsible for tuning and such, this is great, but for those of us who want to know where the rubber meets the road, these tools are kid-gloves. How can we fully optimize and support our code if we can't see the compute on which it runs? What other "benefits" will the system bring that we have to account for?
> Then make it so well-documented and easy to use that teams don't need to be _forced_ to use the platform tooling, they'll just use it because it's the most convenient thing to do.
Very optimistic. In my experience with this kind of thing with homegrown platforms, documentation was a distant last, because it always is, and we weren't left with other choices anyway. I would love to pick the best tool because it is convenient, but that, too, is variable. Maybe I know Kubernetes inside and out - that's plenty convenient for me, then, and will be better documented and discussed than any homegrown platform because the world can contribute to it.
They appear to all be cynical takes. :)
I think one way of reading the comment above is that it's the effect of business incentives / Conway's law on various approaches to staffing the non-developer parts of your organization:
- DevOps is incentivized to be an extension of the dev team and get stuff into production, i.e., in the average company hiring "DevOps," the engineers operating production report up into existing dev management. Therefore you get "scripts and hacks" because you're never given the mandate to do anything better, and you're evaluated on how much you can manage to ship things to prod without building separate specialties, not how well you ship it to prod.
- SRE is incentivized to demonstrate good SRE practices and their value to the business. Incident response is a lot more visible than incident prevention. Picking an observabiity vendor who can get you nice graphs is a lot more visible than spending months making the right graphs. You're evaluated on how much you improve "reliability," which means you need to figure out how to measure it (and display it) in the first place, but it's not clear your measurement is what the broader business would measure.
- Platform engineering is incentivized to build up a well-defined platform and have their own products/services that are used internally. They are a development team of their own. So their value is most visible when those products exist and are widely used, and a little more cynically, when those products aren't unmodified vendor or OSS products. You're evaluated on the number of people using your product and how infrequently the rest of the company says "I'm just going to run this myself, give me an AWS account."
Any of these disciplines can be done well with strong management support, by which I mean when management actually cares about the long-term problem of running production well and can make informed decisions about how to staff that problem. (For myself, for what it's worth, I'm in a platform engineering organization but my current role is 80% to make it easier for people to use OSS and to reduce our deltas and 20% incident prevention.) But when incentives get disconnected from the business and the inertia is to keep delivering what people thought you should have delivered in the past, the disciplines break down in different ways.
I think one way of reading the comment above is that it's the effect of business incentives / Conway's law on various approaches to staffing the non-developer parts of your organization:
- DevOps is incentivized to be an extension of the dev team and get stuff into production, i.e., in the average company hiring "DevOps," the engineers operating production report up into existing dev management. Therefore you get "scripts and hacks" because you're never given the mandate to do anything better, and you're evaluated on how much you can manage to ship things to prod without building separate specialties, not how well you ship it to prod.
- SRE is incentivized to demonstrate good SRE practices and their value to the business. Incident response is a lot more visible than incident prevention. Picking an observabiity vendor who can get you nice graphs is a lot more visible than spending months making the right graphs. You're evaluated on how much you improve "reliability," which means you need to figure out how to measure it (and display it) in the first place, but it's not clear your measurement is what the broader business would measure.
- Platform engineering is incentivized to build up a well-defined platform and have their own products/services that are used internally. They are a development team of their own. So their value is most visible when those products exist and are widely used, and a little more cynically, when those products aren't unmodified vendor or OSS products. You're evaluated on the number of people using your product and how infrequently the rest of the company says "I'm just going to run this myself, give me an AWS account."
Any of these disciplines can be done well with strong management support, by which I mean when management actually cares about the long-term problem of running production well and can make informed decisions about how to staff that problem. (For myself, for what it's worth, I'm in a platform engineering organization but my current role is 80% to make it easier for people to use OSS and to reduce our deltas and 20% incident prevention.) But when incentives get disconnected from the business and the inertia is to keep delivering what people thought you should have delivered in the past, the disciplines break down in different ways.
That's a wonderful idea in theory but I've never seen it work out that way in practice.
I think it's just genuinely too difficult for most companies to do well and it's too easy to fall into the same old well trodden traps.
Documentation is something everybody professes to keep up to date, clear and complete, for instance, but IME it never truly is.
I think it's just genuinely too difficult for most companies to do well and it's too easy to fall into the same old well trodden traps.
Documentation is something everybody professes to keep up to date, clear and complete, for instance, but IME it never truly is.
I mostly agree with you, an average dev does not know and do not care how things are done. But if platform team or any tooling is blocking you from delivering your features frequently, it becomes an issue. They are either under-staffed, or generalizing in favor of greater good comes at a cost of dropping your specific requirements.
In my experience, the end result is having a team of kubernetes fanatics reinventing the wheels themselves and forcing everyone to use their custom ego filling tools instead of paying $100 per month to an already existing solution.
Most developers just need something like heroku, tsuru, appengine, or openstack. But because kubefans want to have fun too and try to be as cool as Google or Facebook we end up having to deal with tons of custom Golang tooling with fancy names, half assed documentation and grumpy looks when you ask for help because you don't understand the 38 steps guide to deploy a service.
Most developers just need something like heroku, tsuru, appengine, or openstack. But because kubefans want to have fun too and try to be as cool as Google or Facebook we end up having to deal with tons of custom Golang tooling with fancy names, half assed documentation and grumpy looks when you ask for help because you don't understand the 38 steps guide to deploy a service.
We're not here to compete for most cynical take.
$100 is not really a price point for any of the things you mentioned, and the way PaaS charge is often restrictive of development model (want microservices? we charge by the service. want a monolith? sorry, we only have instances up to this size. need to scale in smaller increments than "one machine"? too bad.) When I helped build my first platform, it was a migration from Beanstalk to Kubernetes (1.8 at the time).
Not only did it save 70%+ costs, it allowed developers to cut services as large as they wanted, and scale them as fast or slow as they wanted. Metrics and logging were nothing they had to worry about on the infra layer either.
I wouldn't build these things again, OAM / KubeVela is a nice compromise if you want one.
Also, I wouldn't give a dev direct access to Openstack the same way I woudn't give them access to Kubernetes itself. They have too many footguns and too much complexity, either you end up with half-baked deployments or half the teams time spent on doing things the right way.
$100 is not really a price point for any of the things you mentioned, and the way PaaS charge is often restrictive of development model (want microservices? we charge by the service. want a monolith? sorry, we only have instances up to this size. need to scale in smaller increments than "one machine"? too bad.) When I helped build my first platform, it was a migration from Beanstalk to Kubernetes (1.8 at the time).
Not only did it save 70%+ costs, it allowed developers to cut services as large as they wanted, and scale them as fast or slow as they wanted. Metrics and logging were nothing they had to worry about on the infra layer either.
I wouldn't build these things again, OAM / KubeVela is a nice compromise if you want one.
Also, I wouldn't give a dev direct access to Openstack the same way I woudn't give them access to Kubernetes itself. They have too many footguns and too much complexity, either you end up with half-baked deployments or half the teams time spent on doing things the right way.
Of course it is not $100, but it is a lot less than the salaries of a team of infrastructure engineers most of the times
Anecdotally, when my entire team started leaving over the span of 5 months or so after a new CEO attempted to force a move to every service provider he had previously worked with (GCP over AwS, Cloudflare over Akamai, MS Teams over Slack), the non-cloud costs for the platform jumped to about 6 times the teams combined salary when they decided to hire specialist contractors, and even they had a hard time matching the raw performance of that cluster.
The typical case of feature teams not being allowed to refactor / reduce technical debt was extreme. Some frontend requests would result in hundreds of backend requests for no good reason. We've always been ignored by management, so cluster was on the verge of coming apart at the (mostly "spending half the cpu time in conntrack") seams for years. So close, it'd sometimes just die for a minute, probably because someone decided to use HTTP way too liberally (network is free, right?). I had a pcap file from a sample of 10 machines, and a 30 second sample was so large, it was impossible to analyze with any real-time tools.
No managed Kubernetes could match this out of the box. Adding a mesh to do more directed requests for this to be fixed failed even when we tried it. As far as I know they're still using the platform as is (in maintenance mode, with the contractor assisting) and fired that CEO pretty soon after.
Is this a bit of a special case? Sure! Does it still happen a lot, no matter your infrastructure? Count on it.
The typical case of feature teams not being allowed to refactor / reduce technical debt was extreme. Some frontend requests would result in hundreds of backend requests for no good reason. We've always been ignored by management, so cluster was on the verge of coming apart at the (mostly "spending half the cpu time in conntrack") seams for years. So close, it'd sometimes just die for a minute, probably because someone decided to use HTTP way too liberally (network is free, right?). I had a pcap file from a sample of 10 machines, and a 30 second sample was so large, it was impossible to analyze with any real-time tools.
No managed Kubernetes could match this out of the box. Adding a mesh to do more directed requests for this to be fixed failed even when we tried it. As far as I know they're still using the platform as is (in maintenance mode, with the contractor assisting) and fired that CEO pretty soon after.
Is this a bit of a special case? Sure! Does it still happen a lot, no matter your infrastructure? Count on it.
I don't know why would you need contractors, maybe that is what went wrong?
A company I worked for years ago moved our custom built platform to AppEngine Flex and it was the best thing we could have done. We embaraced DevOps as it should be done (not with "devops people" but with developers understanding their production environment). No need for contractors. Just the documentations and an easy to use, well tested and battle proof platform.
Similar experience many, many years ago with Tsuru, which is a kind of smaller self hosted heroku, although this was a company of about 30 enhineers
A company I worked for years ago moved our custom built platform to AppEngine Flex and it was the best thing we could have done. We embaraced DevOps as it should be done (not with "devops people" but with developers understanding their production environment). No need for contractors. Just the documentations and an easy to use, well tested and battle proof platform.
Similar experience many, many years ago with Tsuru, which is a kind of smaller self hosted heroku, although this was a company of about 30 enhineers
Scale, in terms of diversity of use-cases rather than requests per second, seems to be a very important factor in choosing custom, semi-custom or ready-made.
Kubernetes being "a platform for building platforms" still rings true to me, and it is extremely good at that.
One major issue was that the culture wasn't great. SREs without teeth and Platform Engineers that fix the shortcomings of development teams with glue more than anything.
As for why contractors, this was the IT division of a big retail business that refused to pay competitive salaries in solidarity with their other workers, but never fired anyone for underperforming (or not performing at all), so the people that actually get work done tend to leave pretty fast. They tried to compensate this with contractors. Some departments were fine, and I loved the team (high standards, created lots of value for teams), but it wasn't a great long term workplace.
Kubernetes being "a platform for building platforms" still rings true to me, and it is extremely good at that.
One major issue was that the culture wasn't great. SREs without teeth and Platform Engineers that fix the shortcomings of development teams with glue more than anything.
As for why contractors, this was the IT division of a big retail business that refused to pay competitive salaries in solidarity with their other workers, but never fired anyone for underperforming (or not performing at all), so the people that actually get work done tend to leave pretty fast. They tried to compensate this with contractors. Some departments were fine, and I loved the team (high standards, created lots of value for teams), but it wasn't a great long term workplace.
> Most developers just need something like heroku, tsuru, appengine, or openstack.
A lot of companies don't operate at Google or Facebook scale but many companies operate at a scale which quickly outgrows the solutions you mentioned. Maybe you just haven't worked at places with larger scale to see the benefits things like Kubernetes provides?
A lot of companies don't operate at Google or Facebook scale but many companies operate at a scale which quickly outgrows the solutions you mentioned. Maybe you just haven't worked at places with larger scale to see the benefits things like Kubernetes provides?
The key is in what that scale is. Everyone thinks they're already at that scale, or soon will be. Most are not and won't for a very long time. I've worked for large companies (> 500 engineers) where building your own custom thing might make sense. But I've also worked for companies with less than 60 engineers where about 10 of them where infrastructure engineers building all this craziness we're talking about.
Put into the equation all these salaries and the opportunity cost and you have a pretty juicy budget for appengine or similar.
Put into the equation all these salaries and the opportunity cost and you have a pretty juicy budget for appengine or similar.
> - Platform: We made this, we know better than you, you cannot use anything else. If you need new thing/improvement, create a ticket we will get back to you in 6 months to say we cannot do it due to other company wide very important initatives.
I think it is due to cynical takes like this that the industry is evolving towards “Use whatever you want, so long as you are open to being on-call for it yourself. Or you can use this Platform maintained by other people.”
Nothing sucks as much as being woken up by pages because some new, shiny, and untested piece of technology was used in production and broke in an unknown way.
I think it is due to cynical takes like this that the industry is evolving towards “Use whatever you want, so long as you are open to being on-call for it yourself. Or you can use this Platform maintained by other people.”
Nothing sucks as much as being woken up by pages because some new, shiny, and untested piece of technology was used in production and broke in an unknown way.
What's cynical about it? If the platform team is on the way it becomes a problem. And going to be on-call is way to go if you want to have actual ownership, or it becomes to same dev and ops people split.
There is not an exact line. Platform team can easily abstract (i.e) Kubernetes deployment infrastructure, but not everything else that you might be needing.
There is not an exact line. Platform team can easily abstract (i.e) Kubernetes deployment infrastructure, but not everything else that you might be needing.
Then the team who made their own platform hire DevOps and SRE to janitor what they created because it's spiraling out of control, and now you have another platform! It's the circle of IT life.
How did SRE become such a non-skilled job almost like a CISSP auditor is for security?
Originally it was supposed to be a SWE who could also do operations, but there seems to have been a quick race to the bottom to remove all the SWE tasks and turn them into pure operations policy wonks.
Originally it was supposed to be a SWE who could also do operations, but there seems to have been a quick race to the bottom to remove all the SWE tasks and turn them into pure operations policy wonks.
SRE is a highly skilled job. It is actually difficult. Doing a root cause analysis or investigating an infrastructure blackhole is a very rare skill. You have to be aware of the end-to-end landscape, or atleast communicate well with the stakeholders who own the domain and resolve issues, manage expectations, huge pressure dealing with time critical issues.
Having seen all of roles, being SRE was the only time when a slack message sounds would trigger anxiety in me.
Having seen all of roles, being SRE was the only time when a slack message sounds would trigger anxiety in me.
I mean I used to be a tier 4 oncall at Amazon and had global root access, managed the configuration management infrastructure and had enable access on all the networking equipment. I'm criticizing the SRE role from a background of having a very high level of understanding of difficult operational roles. The selling point to me 10 years ago was that it would break down the wall so that people like me could also have access to the source code and be SWEs (instead of having to strace a black box all the time), but it seems like the job has become significantly dumbed down from the systems side, and SREs tend to grow out of the software teams and are the people responsible for the pager, metrics, SLOs, hacking up turing computable YAML, memorizing the whole CNCF landscape, and yelling at Kubernetes.
SRE is just a name for a group that Google created to fit its own purposes. It was never supposed to be some panacea. SRE is just a collection of "production support" responsibilities with authority over a product owner and some software development.
> SREs tend to grow out of the software teams and are the people responsible for the pager, metrics, SLOs, hacking up turing computable YAML, memorizing the whole CNCF landscape, and yelling at Kubernetes.
Yeah, that's exactly what they were created for...
> SREs tend to grow out of the software teams and are the people responsible for the pager, metrics, SLOs, hacking up turing computable YAML, memorizing the whole CNCF landscape, and yelling at Kubernetes.
Yeah, that's exactly what they were created for...
It was always like that outside of select few like Google. Most companies’ infra is not complex enough to warrant anything beyond sysadmin skills. Fwiw google sres also had two distinct subroles within sre one of which is not software focused
>Insists on creating dashboards and putting them on TVs
You didn't have to call me out like that.
You didn't have to call me out like that.
You are mistaking your personal experience with various companies and positions to the actual industry standard. I've worked on the same roles as you did and my experience is completely different and very positive.
I explicitly mentioned it's my personal experience. I'm normally very happy with what I do. Not just the enterprise environment I'm in.
One should not disclose that their comment is based on their personal experience when it is obvious this was the case.
My personal experience agrees 100% with what he has said :)
My personal experience agrees 100% with what he has said :)
Are good ideas just destined to be corrupted? Some examples:
Royce: "These are the flaws in the 'Waterfall model'" -> everybody looks at his diagram and uses it [1]
Agile: "Individuals and interactions over processes and tools" -> SAFe [2]
DevOps: "Assign development and operations the same business goals so they work together, rather than against each other" -> "DevOps was a cultural shift giving development teams more control over shipping code to production" (this article)
[1]: https://en.wikipedia.org/wiki/Waterfall_model#History
[2]: https://scaledagile.com/safe-50/
Royce: "These are the flaws in the 'Waterfall model'" -> everybody looks at his diagram and uses it [1]
Agile: "Individuals and interactions over processes and tools" -> SAFe [2]
DevOps: "Assign development and operations the same business goals so they work together, rather than against each other" -> "DevOps was a cultural shift giving development teams more control over shipping code to production" (this article)
[1]: https://en.wikipedia.org/wiki/Waterfall_model#History
[2]: https://scaledagile.com/safe-50/
Are these really good ideas if they can't survive the collision with the real world? While I don't doubt that there are lots of smart people behind them, knowing what you should ideally do and knowing what would be the best compromise, in reality, are two very different sets of skills.
For example, agile was never going to work according to the rules, because "individuals and interactions over processes and tools" are not how most companies work.
There's probably a parallel to draw with programming languages. A few people will use Haskell or Lisp or Idris and be at the bleeding edge, but most people are satisfied with Java that sometimes cherrypicks good features.
Both of those seem to be rooted in aversion to risk, or (inclusive) managers making most of the decisions. I'd say that's because developers tend to have different values from other people, often valuing stability a bit less and "the right solution" a bit more.
For example, agile was never going to work according to the rules, because "individuals and interactions over processes and tools" are not how most companies work.
There's probably a parallel to draw with programming languages. A few people will use Haskell or Lisp or Idris and be at the bleeding edge, but most people are satisfied with Java that sometimes cherrypicks good features.
Both of those seem to be rooted in aversion to risk, or (inclusive) managers making most of the decisions. I'd say that's because developers tend to have different values from other people, often valuing stability a bit less and "the right solution" a bit more.
> I'd say that's because developers tend to have different values from other people, often valuing stability a bit less and "the right solution" a bit more.
You're hinting at what the original DevOps idea was trying to solve. The traditional view is that Developers value shipping features while Ops engineers value stability/uptime. It should be easy to see why these differing values cause friction between these two groups.
The way DevOps was trying to solve this was getting _management_ to assign incentives/values/goals that would apply to _ both_ groups. An example might be increasing user engagement.
Aligning these two groups should be the main takeaway. The reason why The Phoenix Project book was so successful was it perfectly described this adversarial relationship. I'd certainly seen the same thing playing out at several companies I'd worked for.
You're hinting at what the original DevOps idea was trying to solve. The traditional view is that Developers value shipping features while Ops engineers value stability/uptime. It should be easy to see why these differing values cause friction between these two groups.
The way DevOps was trying to solve this was getting _management_ to assign incentives/values/goals that would apply to _ both_ groups. An example might be increasing user engagement.
Aligning these two groups should be the main takeaway. The reason why The Phoenix Project book was so successful was it perfectly described this adversarial relationship. I'd certainly seen the same thing playing out at several companies I'd worked for.
Yeah, pretty much. But don't despair, this is how the industry progresses.
First, elite practitioners identify flaws in the standard practices of some corner of the industry and figure out a way to ameliorate them. They give it a name so they can talk about it, write about it, organize conferences etc. They're mainly interested in disseminating this good idea.
The idea becomes popular, and a cottage industry springs up. Some of the original proponents become consultants, but there are others who were early adopters, and many more who are jumping on the trend. You also get an ecosystem of open source and commercial tools. This starts to warp the original ideas, as they stop being about purely about improving results and incorporate a bias for generating more consulting or tool revenue.
This drives wider adoption. Companies start looking for people with experience in the new thing, and people start putting it on their resume. It gets applied in situations it wasn't meant for, sometimes on purpose and sometimes by accident. You start to see "no true Scotsman" discussions where detractors claim they tried the thing and it didn't work and proponents claim they weren't doing it right. There's also cargo culting, where people who don't really understand the ideas try to implement the practices with varying degrees of success.
Eventually things get watered down to the point where the terminology starts to just mean "good" or "cool". The actual practitioners stop using it because it's no longer useful for communication and it falls out of favour.
But. All is not lost! The original ideas were good, and they've been widely disseminated. They get taught to young people joining the industry, and they start to be part of "how we do things." They don't need special names to distinguish them from "mainstream" practices because they are the mainstream. In fact, young people sneer at them because duh, without knowing what they were a reaction to.
You can see this with Agile, Lean and DevOps. On the business side, "disruption". The cycles are slower and maybe less intense in more CS and pure coding domains, but you find it there too with scripting languages, object oriented languages, TDD. Functional programming is leaving the consulting phase and seeing wider adoption.
This is probably not how we'd like to think things work. But they do work.
First, elite practitioners identify flaws in the standard practices of some corner of the industry and figure out a way to ameliorate them. They give it a name so they can talk about it, write about it, organize conferences etc. They're mainly interested in disseminating this good idea.
The idea becomes popular, and a cottage industry springs up. Some of the original proponents become consultants, but there are others who were early adopters, and many more who are jumping on the trend. You also get an ecosystem of open source and commercial tools. This starts to warp the original ideas, as they stop being about purely about improving results and incorporate a bias for generating more consulting or tool revenue.
This drives wider adoption. Companies start looking for people with experience in the new thing, and people start putting it on their resume. It gets applied in situations it wasn't meant for, sometimes on purpose and sometimes by accident. You start to see "no true Scotsman" discussions where detractors claim they tried the thing and it didn't work and proponents claim they weren't doing it right. There's also cargo culting, where people who don't really understand the ideas try to implement the practices with varying degrees of success.
Eventually things get watered down to the point where the terminology starts to just mean "good" or "cool". The actual practitioners stop using it because it's no longer useful for communication and it falls out of favour.
But. All is not lost! The original ideas were good, and they've been widely disseminated. They get taught to young people joining the industry, and they start to be part of "how we do things." They don't need special names to distinguish them from "mainstream" practices because they are the mainstream. In fact, young people sneer at them because duh, without knowing what they were a reaction to.
You can see this with Agile, Lean and DevOps. On the business side, "disruption". The cycles are slower and maybe less intense in more CS and pure coding domains, but you find it there too with scripting languages, object oriented languages, TDD. Functional programming is leaving the consulting phase and seeing wider adoption.
This is probably not how we'd like to think things work. But they do work.
I see DevOps as a culture shift that is working towards a culture where developers can easily build, deploy, monitor and alert on their code in production. It is a direct counter-attack to the pattern of developers throwing their code over the wall and hoping that a sys admin will deploy it. You need expertise in tooling and processes in order to achieve the culture shift, that is where these 3 positions come in.
I see each role as an evolution where you go from DevOps -> SRE -> Platform Engineer. Automating the infrastructure and tooling is the catalyst. You can have any of these roles in your company based on how far along the path your company is, how on board management is and how interested your development teams are in growing/learning these practices.
DevOps writes scripts/code to automate CI/CD pipelines, creation of infrastructure, deployments. Once automated you move onto..
SRE writes scripts/code monitoring, alerting, SLOs/SLAs, incident processes. Once automated you move onto..
Platform Engineering where you write the tooling to make all of the above codified. Tooling could be anything from cli tools to kubernetes operators.
I say this as someone that has spent the last 3 years automating my way towards platform engineering.
I see each role as an evolution where you go from DevOps -> SRE -> Platform Engineer. Automating the infrastructure and tooling is the catalyst. You can have any of these roles in your company based on how far along the path your company is, how on board management is and how interested your development teams are in growing/learning these practices.
DevOps writes scripts/code to automate CI/CD pipelines, creation of infrastructure, deployments. Once automated you move onto..
SRE writes scripts/code monitoring, alerting, SLOs/SLAs, incident processes. Once automated you move onto..
Platform Engineering where you write the tooling to make all of the above codified. Tooling could be anything from cli tools to kubernetes operators.
I say this as someone that has spent the last 3 years automating my way towards platform engineering.
I've been listening to the "DevOps is a culture, not a role" mantras on HN and elsewhere for years now. Meanwhile, I've been working precisely as a DevOps Engineer/SRE and doubling my income every year or two.
The job is what you and your colleagues make it out to be. You are there to fill a certain need and a knowledge gap within the team. Figure out what that is and you are golden.
The job is what you and your colleagues make it out to be. You are there to fill a certain need and a knowledge gap within the team. Figure out what that is and you are golden.
You having a successful career is not really relevant to the discussion.
I do agree though that the most important part of a professional life is to fill the gap within your team. But that's another discussion.
I do agree though that the most important part of a professional life is to fill the gap within your team. But that's another discussion.
Totally agree. In my experience it seems engineering titles and the associated role/duty are wildly inconsistent between companies. The company size and industry have a huge impact.
For some reason "DevOps Engineer" is much more polarizing than "FullStack Engineer" or "Senior Developer"
For some reason "DevOps Engineer" is much more polarizing than "FullStack Engineer" or "Senior Developer"
How do you double your income yearly?
Heh I may have exaggerated a bit to make a point ;) realistically its more like every 2 years or so, and partially because I started from a junior role in an enterprise environment and worked my way up to independent consulting.
start with very low income and only have a small number of years in industry.
15k -> 30k -> 60k -> 120k
15k -> 30k -> 60k -> 120k
For the first couple of years it's possible. I'm not sure it's going to work past year 3 or maybe 4.
Part time junior -> full time junior -> senior -> lead over 5 or 6 years.
Also if you have stock at a successful company in the past couple of years things have gone vertical.
Also if you have stock at a successful company in the past couple of years things have gone vertical.
Generally in my experience, a company that's looking for a DevOps engineer does not know what DevOps is.
DevOps is great, but it's not a title. It's beautifully summarized in "The DevOps Handbook" and it's fictional counterpart "The Pheonix Project". It's about culture.
Some people might feel different and want to re-evaluate due to so many company adapting it in a similar vein many companies want to adapt agile. But make no mistake, you can't define it then, since then the context of the company defines the word. Thus sadly the word is slowly loosing its meaning.
This is likely due to lots of management people wanting to adapt the buzzword without understanding it. This since it has have had such a great success in many companies that actually know what it is, have taken the DevOps culture to heart and implemented processes in line with it.
DevOps is great, but it's not a title. It's beautifully summarized in "The DevOps Handbook" and it's fictional counterpart "The Pheonix Project". It's about culture.
Some people might feel different and want to re-evaluate due to so many company adapting it in a similar vein many companies want to adapt agile. But make no mistake, you can't define it then, since then the context of the company defines the word. Thus sadly the word is slowly loosing its meaning.
This is likely due to lots of management people wanting to adapt the buzzword without understanding it. This since it has have had such a great success in many companies that actually know what it is, have taken the DevOps culture to heart and implemented processes in line with it.
Whoever gets too hung up on this must've never had to hire people to do what the DevOps culture they so want to implement needs.
If your plan is to just hire software engineers, you're going to have confused applicants, if your plan is to hire sysadmins, they probably will have a different skillset than you need. Most real companies hire DevOps engineers because that's the way to get the people with the skillsets you need.
Being so purist about this sounds to me like useless debate, but maybe you have better suggestions.
At my current place we stopped hiring DevOps and started hiring SREs + software engineers for specific internal teams, but for a few years there between 2010-2020, good luck getting people with DevOps skills without actually advertising DevOps roles.
If your plan is to just hire software engineers, you're going to have confused applicants, if your plan is to hire sysadmins, they probably will have a different skillset than you need. Most real companies hire DevOps engineers because that's the way to get the people with the skillsets you need.
Being so purist about this sounds to me like useless debate, but maybe you have better suggestions.
At my current place we stopped hiring DevOps and started hiring SREs + software engineers for specific internal teams, but for a few years there between 2010-2020, good luck getting people with DevOps skills without actually advertising DevOps roles.
You can certainly advertise a DevOps culture in a listing that isn't for a DevOps engineer, it's not mutually exclusive.
What's DevOps skills in your opinion?
What's DevOps skills in your opinion?
I think that it's less important to make the distinction nowadays because systems engineers use developer tools and modern practices. 25 years ago when I was a "Junior Linux systems administrator" my peers kept their vast collection of shell scripts on a file server if you were lucky - some hoarded their own "craft" and kept them on their personal workstations. Scripts got copied onto machines and executed. There was no CI/CD because in most places, systems folk didn't use version control. The best we had back then was a PXE boot server + kickstart files + custom packages. Married to virtualization, this gave you a pretty flexible way to roll out big changes without having to run around with CDs and then manually install and configure software. If you were really lucky, some bright spark put the package source files in CVS, and maybe even triggered RPM automatically, so newly built or rebuilt servers were in an almost ready state from the start. But system state would begin to drift immediately, and you could almost guarantee that the Apache config in one box wouldn't match the next. While frequent automatic server rebuilds were the gold standard, few shops actually implemented that. That meant that systems engineers needed to be able to fix anything. Manually. This required intimate familiarity with the content of system files for GRUB, PAM, dhcpd, init scripts, if-scripts, LDAP, NFS, Apache, SendMail... In fact, the ability to manually hack on those things were the skillset for a systems engineer. Look at the RedHat 6 RHCE exam curriculum!
If you could write complex shell or perl scripts, and you knew how to dig into performance metrics on the fly, you were a god. (Yeah, capturing and displaying system metrics was in its infancy, and most shops simply didn't know how to do it)
While some systems were built to be "horizontally scalable", you were lucky if you worked with systems that weren't SPOFs. You were happy with redundancy via active/passive failover (and that was probably a manual affair). Maybe if you were really lucky, you worked for a SaaS that served a monolithic website via a fleet of webservers behind an F5 loadbalancer. "Wooooah!". But you still set up those F5 VIPs and pools manually.
The modern systems engineer rarely has to do anything like this, even though the underlying technology is similar. These days everything you care about is in git, and is mostly testable on git push. Modern companies, including tiny 4-person SaaS startups, can have infrastructure-as-code, configuration-as-code, packer configs, dockerfiles, container orchestration, CI/CD, metrics collection, USE/RED metric graphs for every system with alerting, pager, incident handling runbooks...
At some point over the last 25 years, the term Systems Engineer could mean that you're an expert in the old ways or the new ways. When you advertised for a DevOps engineer, it left no doubt that you were looking for the modern variety and expected that experience and skillset.
If you could write complex shell or perl scripts, and you knew how to dig into performance metrics on the fly, you were a god. (Yeah, capturing and displaying system metrics was in its infancy, and most shops simply didn't know how to do it)
While some systems were built to be "horizontally scalable", you were lucky if you worked with systems that weren't SPOFs. You were happy with redundancy via active/passive failover (and that was probably a manual affair). Maybe if you were really lucky, you worked for a SaaS that served a monolithic website via a fleet of webservers behind an F5 loadbalancer. "Wooooah!". But you still set up those F5 VIPs and pools manually.
The modern systems engineer rarely has to do anything like this, even though the underlying technology is similar. These days everything you care about is in git, and is mostly testable on git push. Modern companies, including tiny 4-person SaaS startups, can have infrastructure-as-code, configuration-as-code, packer configs, dockerfiles, container orchestration, CI/CD, metrics collection, USE/RED metric graphs for every system with alerting, pager, incident handling runbooks...
At some point over the last 25 years, the term Systems Engineer could mean that you're an expert in the old ways or the new ways. When you advertised for a DevOps engineer, it left no doubt that you were looking for the modern variety and expected that experience and skillset.
I supposed instead of saying DevOps has lost its meaning it has been co-opted from it's original meaning to something else. Problem is just that it isn't well defined. I like a definition like this, but as an earlier comment talked about a DevOps role is about facilitating the DevOps culture values, which is quite the different thing.
On another note, how amazing it is to define our systems so we can more systematically iterate on them and improve. I love it when developers get integrated into the process as well.
The general consensus where i work is that everyone does DevOps, you develop the app, you make sure you deliver the app, if there's any issues you debug. You ain't done until your ticket is in production, and you can't point to other to solve it for you, only help you.
Where does the DevOps engineer fit in there?
On another note, how amazing it is to define our systems so we can more systematically iterate on them and improve. I love it when developers get integrated into the process as well.
The general consensus where i work is that everyone does DevOps, you develop the app, you make sure you deliver the app, if there's any issues you debug. You ain't done until your ticket is in production, and you can't point to other to solve it for you, only help you.
Where does the DevOps engineer fit in there?
> where does the devops engineer exist there
Improving pipelines. Dealing with downtime on the CI. Improving performance of the CDN. Writing better tooling for the devs to deliver value even faster.
Improving pipelines. Dealing with downtime on the CI. Improving performance of the CDN. Writing better tooling for the devs to deliver value even faster.
You can also advertise for zookeeper or chef, but if you want people doing X, a good bet is to advertise for X.
I asked for alternative suggestions and offered what we use. If you have suggestions of alternative titles you've successfully used to hire without slowing things down I'm all ears.
> What's DevOps skills in your opinion?
The most important skill is knowing to not fall for the bait that is this question so that you can actually work.
I asked for alternative suggestions and offered what we use. If you have suggestions of alternative titles you've successfully used to hire without slowing things down I'm all ears.
> What's DevOps skills in your opinion?
The most important skill is knowing to not fall for the bait that is this question so that you can actually work.
But if you want Software Engineers that will take up DevOps skills and responsibilities, surely you'd want to advertise this DevOps culture instead of a role? Advertising this as culture is likely to attract people that want to work in a place without a DevOps/SE stratification.
I think the only reason to advertise for it as a role, is when your tooling is too complex and you've given up looking for Software Engineers that could do it on the side. Which is fair enough, I guess, though providing a Golden Path nowadays is probably more popularly done by hiring Platform Engineers.
I think the only reason to advertise for it as a role, is when your tooling is too complex and you've given up looking for Software Engineers that could do it on the side. Which is fair enough, I guess, though providing a Golden Path nowadays is probably more popularly done by hiring Platform Engineers.
I'm all ears when you're on brake.
Generally, in my experience, an individual who says, "DevOps is a culture, not a title", does not know what DevOps is. It can be both, and exclusive, at the same time.
https://link.medium.com/tC5p4nfYCib
https://link.medium.com/tC5p4nfYCib
I think there's a point to the blog article itself, but i think you miss a vital point in the conclusion. The way a DevOps position that is described in the blog post is a valid position that could have DevOps in the title. But that's not usually how the DevOps role is used. This can be easily seen in many listings of the job title.
The analogy with agile is pretty good here. Sure you could have an Agile Engineer, that works primarily with making sure the agile process is actually used and help facilitating that. Normally that would not be a position itself though, especially not an engineering one.
The analogy with agile is pretty good here. Sure you could have an Agile Engineer, that works primarily with making sure the agile process is actually used and help facilitating that. Normally that would not be a position itself though, especially not an engineering one.
> you miss a vital point in the conclusion
Never had someone tell me I dont understand my own conclusion to my own article before. Thats a first.
Never had someone tell me I dont understand my own conclusion to my own article before. Thats a first.
You can certainly misrepresent your own article. But in this case you're arguing different things as far as I interpret your text.
My argument, in both cases, is that devops can be a title, and that the blanket statement "Devops isnt a role", is incorrect.
Sure, maybe some job postings are still wrong, but that doesnt mean they are all wrong.
"DevOps is great, but it's not a title."
If you are out there interviewing for a DevOps job and you hear things like this: run. Every meeting you have with this person will be a living hell. Endless debates on semantics and definitions while your project is due today.
If you are out there interviewing for a DevOps job and you hear things like this: run. Every meeting you have with this person will be a living hell. Endless debates on semantics and definitions while your project is due today.
So you think the issue is semantics and not understanding? Being due today and having 5 developers thinking the end goal is different and working against each other is way worse then spending some time building consensus and work toward the same goal
In many way the adoption of DevOps has many similarities with the larger adoption of agile development. It just looses it original meaning in a world of little reflection and just surface level knowledge.
Those who can manage to surpass that though can get great productivity from it though.
Those who can manage to surpass that though can get great productivity from it though.
I wrote a blog post about this relatively recently.
I was incredibly frustrated with everyone I spoke to having a different definition of DevOps. For many it’s just sysadmins rebadged. For others they think they sysadmins couldn’t code so it’s an evolution. For yet more they think it’s feature developers who manage ops.
Nobody has a formal definition that fits, the term only exists to obfuscate responsibility.
http://blog.dijit.sh/devops-confusion-and-frustration
I was incredibly frustrated with everyone I spoke to having a different definition of DevOps. For many it’s just sysadmins rebadged. For others they think they sysadmins couldn’t code so it’s an evolution. For yet more they think it’s feature developers who manage ops.
Nobody has a formal definition that fits, the term only exists to obfuscate responsibility.
http://blog.dijit.sh/devops-confusion-and-frustration
The term "DevOps" was originally coined to mean organizing teams so that development and operations were merged. That is, development teams were responsible for writing, deploying and operating services. It was a reaction to the then-common system where development and ops were separate.
But it was quickly coopted to be "ops, but cooler" because these sysadmins write code too. And then even that definition started to get coopted by people using it for other things. These days it can mean anything. Frankly, I see it as a small red flag. Anyone who uses it is basically jumping on yesterday's trendy jargon, and that's not a good sign.
But it was quickly coopted to be "ops, but cooler" because these sysadmins write code too. And then even that definition started to get coopted by people using it for other things. These days it can mean anything. Frankly, I see it as a small red flag. Anyone who uses it is basically jumping on yesterday's trendy jargon, and that's not a good sign.
I also recently wrote a blog post about this, trying to sum up why different definitions exist.
https://link.medium.com/tC5p4nfYCib
I think it's good that you took the time to write such a detailed post, certainly better than mine in a lot of ways.
I agree that Development should be mostly self-service and that the goal of any "Ops" person is to get out of the way. "Automate yourself out of a job" is a common sysadmin mantra.
However I don't agree with your other points, there's some over exaggeration when you say "if devops doesn't exist then thats a red flag", if devops can mean anything then it's a useless term and absence of it shouldn't be seen as a red flag.
>Shoud DevOps engineer be a role? Yes. Absolutely.
Small typo here, and of course I disagree with the statement and preceding paragraph.
I think "Platform Engineer" is closer to what your definition of DevOps is.
The point of my post is that everyone has their own definition, which is why I'm frustrated at the term.
> Infrastructure Engineer
Regarding this... This is actually my job title right now, but it exists outside of computing: "Infrastructure Engineer" being the kind of person who makes bridges and roads and so on.
I agree that Development should be mostly self-service and that the goal of any "Ops" person is to get out of the way. "Automate yourself out of a job" is a common sysadmin mantra.
However I don't agree with your other points, there's some over exaggeration when you say "if devops doesn't exist then thats a red flag", if devops can mean anything then it's a useless term and absence of it shouldn't be seen as a red flag.
>Shoud DevOps engineer be a role? Yes. Absolutely.
Small typo here, and of course I disagree with the statement and preceding paragraph.
I think "Platform Engineer" is closer to what your definition of DevOps is.
The point of my post is that everyone has their own definition, which is why I'm frustrated at the term.
> Infrastructure Engineer
Regarding this... This is actually my job title right now, but it exists outside of computing: "Infrastructure Engineer" being the kind of person who makes bridges and roads and so on.
> "Infrastructure Engineer" being the kind of person who makes bridges and roads and so on.
Reminds me of an old old joke.
There are two kinds of Engineers. Mechanical Engineers who build weapons, and Civil Engineers who build targets.
Reminds me of an old old joke.
There are two kinds of Engineers. Mechanical Engineers who build weapons, and Civil Engineers who build targets.
This is a very timely post for me. I was just writing a DevOps job listing, but it felt wrong. We don’t have a lot of code, but we do have a lot of AWS infrastructure. It sounds like we should be looking for a Platform Engineer more than a DevOps Engineer. Of course the titles are less important than the work, but I think a different title will help focus the search better. So, thanks for posting!
"The person who fiddles with our stuff on AWS"
I've held multiple titles doing this. Titles doesn't matter, because nobody respects what the titles implies.
I've held multiple titles doing this. Titles doesn't matter, because nobody respects what the titles implies.
If you are looking for someone who is very versitile in AWS and will also work on proposing architectural decisions:
Look for CloudOps, we often also have multiple certificates from Azure/AWS/DigitalOcean/Alibaba etc.
We not only keep infrastructure in code, but also make sure you are not ripped off by cloud provider and your architecture makes sense..
Look for CloudOps, we often also have multiple certificates from Azure/AWS/DigitalOcean/Alibaba etc.
We not only keep infrastructure in code, but also make sure you are not ripped off by cloud provider and your architecture makes sense..
You make a good point about cloud cost optimization. That is not a core DevOps function, but it is something we need. Yet another reason to refocus our search.
I see more and more "platform engineer" in job listing, for what require the same skills as infrastructure/hosting/system engineer (regarding SaaS, at least).
Maybe the next buzzword will be "platform engineer".
Maybe the next buzzword will be "platform engineer".
Sort of off-topic, though I'm looking for a role similar to what you described. If you're open to chatting about it, please do shoot me an email: notsure_12 at protonmail.com
Infrastructure Engineer is also a common title for this role.
DevOps is not a title. That’s like looking for “agile engineer”. What it sounds like you need is a systems eng / sysadmin
DevOps Engineer is absolutely a title nowadays. Refusing it because you personally don’t agree with it doesn’t change anything; it’s what searched by both candidates as well as companies.
EDIT: Great explanation from one of the commenters https://link.medium.com/tC5p4nfYCib
EDIT: Great explanation from one of the commenters https://link.medium.com/tC5p4nfYCib
Lol that is a ridiculous explanation. Most companies’ idea of “developer operations” is “someone needs to babysit that dinky old jenkins”, those that are serious about it certainly don’t call it devops in their job reqs
Again, you’re gatekeeping because of your personal believe. Check how the market looks like outside of your bubble.
Industry has moved on and if you need somebody to manage direction of your infrastructure (yes, be it just Jenkins or whole AWS) you will search for DevOps Engineer. There is no serious tier.
Industry has moved on and if you need somebody to manage direction of your infrastructure (yes, be it just Jenkins or whole AWS) you will search for DevOps Engineer. There is no serious tier.
> Industry has moved on and if you need somebody to manage direction of your infrastructure
If you needed dedicated people so dev teams can just throw it over the fence to them that already existed and was just called sysadmins/ops. No need to invent a new term for that. More like some companies didn’t get the memo that devops != sysadmin, not even a title at all.
> There is no serious tier.
Well who’s in a bubble really?
If you needed dedicated people so dev teams can just throw it over the fence to them that already existed and was just called sysadmins/ops. No need to invent a new term for that. More like some companies didn’t get the memo that devops != sysadmin, not even a title at all.
> There is no serious tier.
Well who’s in a bubble really?
I understand what you are saying. I have read blog posts that argue DevOps may be someone’s responsibilities, but it is not a title. I’ve read others that say DevOps is a method to managing code delivery, but not a title. I’ve read job descriptions and CVs that list DevOps as a title.
My conclusion is DevOps is a newly made up word. The technology community is still trying to agree on a definition. Like all words, that definition will change over time.
I think it is okay to have your favorite definition, but I would be careful inferring too much about someone’s “seriousness” or competence based on their choice of favorite definition.
My conclusion is DevOps is a newly made up word. The technology community is still trying to agree on a definition. Like all words, that definition will change over time.
I think it is okay to have your favorite definition, but I would be careful inferring too much about someone’s “seriousness” or competence based on their choice of favorite definition.
All words are made up. What really ticks me off is the term got blatantly hijacked by unwitting orgs to mean almost polar opposite of what the original intended (same seems to have happened to Agile). So now competent orgs have to jump through the hoops to market related roles effectively
Well, thought I’d share my personal experience and the focus areas of our platform engineering team, since the DevOps/SRE definitions are more-or-less correct.
I’ve seen Platform Engineering take various different forms, but it’s current incarnation is split into 2 teams:
1. Developer Productivity. Ownership of quality, efficiency, and developer tools. A recent example project was improving the quality of our test data capabilities to make manual testing easier/more consistent.
2. Unified Client Platform. This team owns our design system and is the primary owner of our effort to unify on React.js/React Native, though a lot of the heavy lifting is distributed around the team.
(I mostly wanted to weigh in since at a mid-sized company at least, we’re not interested in building our own PaaS… k8s or Lambda is fine for that. Instead, we want to ensure developer onboarding is painless and anything specific to us is easily understood - the bulk of the surface area should be best-in-class 3rd party tools or ideally popular OSS frameworks.)
I’ve seen Platform Engineering take various different forms, but it’s current incarnation is split into 2 teams:
1. Developer Productivity. Ownership of quality, efficiency, and developer tools. A recent example project was improving the quality of our test data capabilities to make manual testing easier/more consistent.
2. Unified Client Platform. This team owns our design system and is the primary owner of our effort to unify on React.js/React Native, though a lot of the heavy lifting is distributed around the team.
(I mostly wanted to weigh in since at a mid-sized company at least, we’re not interested in building our own PaaS… k8s or Lambda is fine for that. Instead, we want to ensure developer onboarding is painless and anything specific to us is easily understood - the bulk of the surface area should be best-in-class 3rd party tools or ideally popular OSS frameworks.)
Usually a bunch of wildly different teams are called "DevOps"
But I found a nice resource to reference where we have 7 "DevOps" team anti-patterns and 9 valid team patterns that people call "DevOps":
https://web.devopstopologies.com/
But I found a nice resource to reference where we have 7 "DevOps" team anti-patterns and 9 valid team patterns that people call "DevOps":
https://web.devopstopologies.com/
DevOps is a culture, where there is a mindset of "build it, run it, own it". SRE is a job title. A new job title for what used to called System Administrators, except Google called them SRE's so everyone else joined in.
[deleted]
"Theory"
DevOps: Make the developers work with infrastructure better, deploy faster, have less errors!
SRE: Make the products more reliable, get failures down to the minimum, have less toil!
Platform: Be more efficient by standardizing technology across a large organization!
"Reality"
DevOps: A sysadmin that writes Terraform
SRE: A sysadmin that writes Terraform and knows when the site goes down and if it's important
Platform: A couple of sysadmins forcing everyone to use one set of technology in one way without ever talking to anyone about what to build or how to use it
DevOps: Make the developers work with infrastructure better, deploy faster, have less errors!
SRE: Make the products more reliable, get failures down to the minimum, have less toil!
Platform: Be more efficient by standardizing technology across a large organization!
"Reality"
DevOps: A sysadmin that writes Terraform
SRE: A sysadmin that writes Terraform and knows when the site goes down and if it's important
Platform: A couple of sysadmins forcing everyone to use one set of technology in one way without ever talking to anyone about what to build or how to use it
What resources are there that go inepth into platform engineering?
I currently see myself in a position where I'm responsible for direction of a platform.
What best practices are there around common things like urn, config management, infra provisioning, and other common components.
I have my best practices, which I could enforce on users. But how do I know what is most important to focus on.
I currently see myself in a position where I'm responsible for direction of a platform.
What best practices are there around common things like urn, config management, infra provisioning, and other common components.
I have my best practices, which I could enforce on users. But how do I know what is most important to focus on.
I wrote down my experiences with building delivery platforms, which answers some of this. Might be interesting https://blog.brujordet.no/post/devops/building-a-delivery-pl...
But with regards to purely technical stuff like what tools to choose and how to use them you’ll need experience, either your own or someone elses. There’s really no replacement in my opinion. I had loads of experienced people around me previously, but now that I work more or less alone in the field I rely a lot on community slack channels for the various tools and debugging. It’s amazing to be stuck trying to figure out how to best use Loki for logging, jump on their slack and get advice from the Lead Engineer working on the tool. Or getting input on what tools to use from 50 different people in different companies through /r/devops on reddit.
But with regards to purely technical stuff like what tools to choose and how to use them you’ll need experience, either your own or someone elses. There’s really no replacement in my opinion. I had loads of experienced people around me previously, but now that I work more or less alone in the field I rely a lot on community slack channels for the various tools and debugging. It’s amazing to be stuck trying to figure out how to best use Loki for logging, jump on their slack and get advice from the Lead Engineer working on the tool. Or getting input on what tools to use from 50 different people in different companies through /r/devops on reddit.
Started out 10 years ago as IT Administrator. Transitioning from platform (Company A) to DevOps (Company B) soon.
I'm convinced that DevOps, for better or worse, is where the money is / will be. Platform is still quite close to the classic Sysadmin from 10-20 years ago. Sure, there's some IaC and pipelines and automation, but depending on the company, it might as well be 2007 and you're wrangling bash scripts on individual hosts.
I work on a Platform team, but I'm wondering whether we should rename it. We do provide CI/CD to the rest of the company, and we provide an internal development platform.
Our biggest focus, though, is providing the platform upon which our customers build their apps. We are a Multi-Tenant SaaS development platform. So we offer the APIs and run "The Platform" in production, which includes a database, a task system, a programming language, etc...
Our biggest focus, though, is providing the platform upon which our customers build their apps. We are a Multi-Tenant SaaS development platform. So we offer the APIs and run "The Platform" in production, which includes a database, a task system, a programming language, etc...
Platform teams have become my bread and butter. They come with an interesting set of challenges.
One of the biggest of those, and the reason it can be a challenging position to be in, is that the person managing the team, coming up with the vision and roadmap, and even the architecture direction are often the same person. Few people around you are experts in building a good vision, strategy, or roadmap for a platform organization. It’s a skill set heavily overlapping with product strategy but also completely divergent. If you’re in this position, you’ll be doing several of these things poorly at any given point in time. (My current employer sits on the opposite side and has made sure we have a fully engaged product partner to do this well, which is a breath of fresh air. It's a much different working relationship than with a product partner in user-facing teams)
It’s a high leverage position and can provide a center of excellence for the rest of the organization to emulate. You tend to be very visible among developers in the org if you’re doing it well, and you drive a lot of cross-cutting architectural goals.
On the team building side, hiring people that thrive doing platform work can also be tough! It’s lower visibility, but provides a unique set of technical challenges and an opportunity for building technical breadth you find in few other roles.
Early on in your career can be a great time to explore this sort of work. Repetition is low, and you build a lot of tech chops through the work. Try to be somewhere where it seems like a first class citizen in the organization ecosystem with engagement from product, too.
And for what it's worth, I don't think your general goal should be to attempt to build over buy. Helping teams adopt and successfully use paid-for tools is a big part of the gig.
One of the biggest of those, and the reason it can be a challenging position to be in, is that the person managing the team, coming up with the vision and roadmap, and even the architecture direction are often the same person. Few people around you are experts in building a good vision, strategy, or roadmap for a platform organization. It’s a skill set heavily overlapping with product strategy but also completely divergent. If you’re in this position, you’ll be doing several of these things poorly at any given point in time. (My current employer sits on the opposite side and has made sure we have a fully engaged product partner to do this well, which is a breath of fresh air. It's a much different working relationship than with a product partner in user-facing teams)
It’s a high leverage position and can provide a center of excellence for the rest of the organization to emulate. You tend to be very visible among developers in the org if you’re doing it well, and you drive a lot of cross-cutting architectural goals.
On the team building side, hiring people that thrive doing platform work can also be tough! It’s lower visibility, but provides a unique set of technical challenges and an opportunity for building technical breadth you find in few other roles.
Early on in your career can be a great time to explore this sort of work. Repetition is low, and you build a lot of tech chops through the work. Try to be somewhere where it seems like a first class citizen in the organization ecosystem with engagement from product, too.
And for what it's worth, I don't think your general goal should be to attempt to build over buy. Helping teams adopt and successfully use paid-for tools is a big part of the gig.
DevOps and Agile: The two worst things ever to happen to software development (and developers).
As a developer, I love the idea of operating, maintaning, deploying and monitoring the services I work on, as long as I'm using s proven platform such as appengine, tsuru, openstack, heroku, etc.
I don't want to waste my life understanding your custom reinvented wheel Golang tooling around kubernetes and 50 steps guides to do anything.
I don't want to waste my life understanding your custom reinvented wheel Golang tooling around kubernetes and 50 steps guides to do anything.
- DevOps: Hired to do everything not involved in the feature development of main business. Range varies from Terraform to maintaining Jira, GSuite, Jenkins, ETL, BizOps. Maintains scripts and hacks everywhere.
- SRE: Orders teams to define SLOs, focuses only after incident procedures, not before. Insists on creating dashboards and putting them on TVs (when we had offices). Most eventually become vendor fans.
- Platform: We made this, we know better than you, you cannot use anything else. If you need new thing/improvement, create a ticket we will get back to you in 6 months to say we cannot do it due to other company wide very important initatives.
I love not being a regular developer and work on platform side, but its a very irregular space and very hard to hire.