Three Ways of DevOps(ermetic.com)
ermetic.com
Three Ways of DevOps
https://ermetic.com/whats-new/blog/the-three-ways-of-devops/
23 comments
DevOps consultants have existed for years, but they only talk to upper management, like the Six Sigmas before them.
This article doesn't really explain The Three Ways that well. I recommend going to the source: https://itrevolution.com/the-three-ways-principles-underpinn... Keep in mind this is only one of several ways of explaining what DevOps is about, as the whole concept evolved over several years.
Microservices were old and busted 5 years ago, but like Scrum, the industry pays no notice to broken patterns, only new ones.
"Learning" is definitely a higher-order pattern that you have to practice to get good at. Whether it's machine learning, distributed tracing, sliding metric alarms, etc they aren't easy to implement correctly but they can level up the efficiency of dealing with big systems. You're right that simpler systems are best, until a specialized team can work on more complex ones without getting run into the ground.
This article doesn't really explain The Three Ways that well. I recommend going to the source: https://itrevolution.com/the-three-ways-principles-underpinn... Keep in mind this is only one of several ways of explaining what DevOps is about, as the whole concept evolved over several years.
Microservices were old and busted 5 years ago, but like Scrum, the industry pays no notice to broken patterns, only new ones.
"Learning" is definitely a higher-order pattern that you have to practice to get good at. Whether it's machine learning, distributed tracing, sliding metric alarms, etc they aren't easy to implement correctly but they can level up the efficiency of dealing with big systems. You're right that simpler systems are best, until a specialized team can work on more complex ones without getting run into the ground.
It's an advertisement for a company's product on their corporate blog.
Yes, its an invention by Gene Kim the author of The Phoenix Project and a couple of other books on the subject. Self profused expert in all things DevOps and inventor of the worst way to give names to common practices.
First Way - is whatever practice, I never can remember. Second Way - is another practice, no idea which one. Third Way - is the third practice, naturally, equally eludes me.
First Way - is whatever practice, I never can remember. Second Way - is another practice, no idea which one. Third Way - is the third practice, naturally, equally eludes me.
> This article triggers my Scrum™️ PTSD. The Way of DevOps? Am I going to have to fight off DevOps™️ certified consultants for the next five years now?
I feel you. Agile / DevOps certified consultants are an oxymoron. Outside of evangelization and some advice, they have no clue of the project and as such are just outsourced bosses. The core concepts are not that hard to grasp: teams are responsible to deliver value in small batches to the client so they can get quick feedback from the client. Having to employ them should be seen as a failure and as a symptom of a dysfunctional organization where the teams are not trusted by the higher ranks to have a say in how they organize themselves and thus they hire expansive people who gave them a sale pitch that corresponds initial vision. tl;dr: agile comes from within, people telling you how to work are just other way to say "boss".
> By the way, I would recommend not trying to make your systems clever and “learn”, as the Third Way™️ suggests. Keep them dumb and understandable. The last thing a critical service needs is mysterious, unpredictable, unexplainable behavior. This point alone is a huge red flag for me, brace yourselves for five years of over-engineering as people blindly latch on to advice like this.
If that happens the problem is not the people who try to develop improvements, it's the system that is not resilient to changes. These "mysterious, unpredictable, unexplainable" behaviors rarely manifest themselves all at once just by magic. They are usually caused by a comorbidity of problems that has accumulated due to a lack of will to fix them promptly or the lack of tools to catch and identify them early.
Blaming your developers because they attempted to improve the system is just downright counterproductive and demoralizing because you will end up with a Tech Lead™️ playing police because they think they must protect the System like the Secret Services must protect VIPs from threats that could come up in all directions.
Instead ask yourself those questions: How many times have you heard "we don't have time to fix this"? How many sheets of paper would you use to explain the multitasking architecture of your system? Can you test your system on a single thread so you can make sure your sequential logic is sound? How much time does it take to run all the tests? How much does your development environment differs from production? Can you easily toggle a feature to test it?
The problem is not the "3rd way". There is a reason why they have numbered it from 1 to 3, it's because you can't achieve the third without the second and the first before that. In other words you can't learn acrobatics without learning how to walk first.
> I suppose this is the next round of silver bullets after “microservices”.
Fads are universally bad yes, but as with most prejudices they always have a kernel of truth.
I feel you. Agile / DevOps certified consultants are an oxymoron. Outside of evangelization and some advice, they have no clue of the project and as such are just outsourced bosses. The core concepts are not that hard to grasp: teams are responsible to deliver value in small batches to the client so they can get quick feedback from the client. Having to employ them should be seen as a failure and as a symptom of a dysfunctional organization where the teams are not trusted by the higher ranks to have a say in how they organize themselves and thus they hire expansive people who gave them a sale pitch that corresponds initial vision. tl;dr: agile comes from within, people telling you how to work are just other way to say "boss".
> By the way, I would recommend not trying to make your systems clever and “learn”, as the Third Way™️ suggests. Keep them dumb and understandable. The last thing a critical service needs is mysterious, unpredictable, unexplainable behavior. This point alone is a huge red flag for me, brace yourselves for five years of over-engineering as people blindly latch on to advice like this.
If that happens the problem is not the people who try to develop improvements, it's the system that is not resilient to changes. These "mysterious, unpredictable, unexplainable" behaviors rarely manifest themselves all at once just by magic. They are usually caused by a comorbidity of problems that has accumulated due to a lack of will to fix them promptly or the lack of tools to catch and identify them early.
Blaming your developers because they attempted to improve the system is just downright counterproductive and demoralizing because you will end up with a Tech Lead™️ playing police because they think they must protect the System like the Secret Services must protect VIPs from threats that could come up in all directions.
Instead ask yourself those questions: How many times have you heard "we don't have time to fix this"? How many sheets of paper would you use to explain the multitasking architecture of your system? Can you test your system on a single thread so you can make sure your sequential logic is sound? How much time does it take to run all the tests? How much does your development environment differs from production? Can you easily toggle a feature to test it?
The problem is not the "3rd way". There is a reason why they have numbered it from 1 to 3, it's because you can't achieve the third without the second and the first before that. In other words you can't learn acrobatics without learning how to walk first.
> I suppose this is the next round of silver bullets after “microservices”.
Fads are universally bad yes, but as with most prejudices they always have a kernel of truth.
I don't know what you're talking about WRT the 3rd way. I agree that developers should continuously improve their systems.
I was replying to the descriptions in the article:
> The third way of DevOps is continuous learning. [...] Sometimes this means introducing artificial intelligence and machine learning into your systems or tooling, to ensure that it is continuously learning.
> The third way involves continuous learning. If our automated monitoring system, itself, can learn continuously from the traffic, access, and behavior of our cloud activity and users, then that respects the third way. A system that can train itself, learn when to remove permissions, when to replace permissions, and when to block access, not only respects the third way, it makes for a fantastic least privilege security tool.
If your company specializes in this, and you have ML experts who know how to design, maintain, interpret, and monitor the models you are using in production, and who carry pagers so you can page them at 2 AM when something breaks and you can't understand what the model is doing...then sure. If you are the 99.999% of other companies, then this is a terrible idea and you should let me know before the interview so I can avoid working at your company (unless you're hiring me to remove it, because it inevitably caused more problems than it solved).
I was replying to the descriptions in the article:
> The third way of DevOps is continuous learning. [...] Sometimes this means introducing artificial intelligence and machine learning into your systems or tooling, to ensure that it is continuously learning.
> The third way involves continuous learning. If our automated monitoring system, itself, can learn continuously from the traffic, access, and behavior of our cloud activity and users, then that respects the third way. A system that can train itself, learn when to remove permissions, when to replace permissions, and when to block access, not only respects the third way, it makes for a fantastic least privilege security tool.
If your company specializes in this, and you have ML experts who know how to design, maintain, interpret, and monitor the models you are using in production, and who carry pagers so you can page them at 2 AM when something breaks and you can't understand what the model is doing...then sure. If you are the 99.999% of other companies, then this is a terrible idea and you should let me know before the interview so I can avoid working at your company (unless you're hiring me to remove it, because it inevitably caused more problems than it solved).
For those unfamiliar with “Three Ways”, I was just introduced to the concept in the book “The Phoenix Project” which was in reference to the book “The Goal”.
Excited to read this article and after briefly skimming, didn’t see any immediate reference so figured I’d leave webcrumbs here.
The Goal: https://a.co/dUDdGQ6 Phoenix Project: https://www.google.com/search?q=phonix+prijext&ie=UTF-8&oe=U...
Excited to read this article and after briefly skimming, didn’t see any immediate reference so figured I’d leave webcrumbs here.
The Goal: https://a.co/dUDdGQ6 Phoenix Project: https://www.google.com/search?q=phonix+prijext&ie=UTF-8&oe=U...
I stopped reading The Phoenix Project about halfway through... just read The Goal - it's better written and was the original source for the core ideas. Or, said another way, if you've already read any of Goldratt's books, you know the core ideas of the Phoenix Project. Are there other books that could more directly speak to implementation details, sure, probably.
I didn't read The Goal but one of the key point of the Phoenix Project is that the same principles that make a factory efficient (or inefficient) are exactly the same for software and IT.
That is general hand-wavy-way to say that things can be improved, in general.
The Goal actually specifies which misconceptions we have in many cases that leads to solutions that seem reasonable but are completely wrong! Without spoiling the whole book, but do look for the explanation in there for "statistical fluctuations" and "dependant events".
And then the Goal make you the reader come up with a solution that fits the factory environment (with the stat.fluctuations and dependant events) like a glove. A solution that creates a sort of a miracle for that time and place.
Other Goldratt books give more example, and more teachings, on how to look at practices everyone is doing wrong and appraising those using common sense. A favorite saying by Goldratt is that "common sense is not very common".
So just "solving problems" in an IT environment in most cases will lead to the wrong solutions, just because many people assume that solution X is the best solution without even looking at the problem THEY have. (GitFlow I am looking at you!)
The Goal actually specifies which misconceptions we have in many cases that leads to solutions that seem reasonable but are completely wrong! Without spoiling the whole book, but do look for the explanation in there for "statistical fluctuations" and "dependant events".
And then the Goal make you the reader come up with a solution that fits the factory environment (with the stat.fluctuations and dependant events) like a glove. A solution that creates a sort of a miracle for that time and place.
Other Goldratt books give more example, and more teachings, on how to look at practices everyone is doing wrong and appraising those using common sense. A favorite saying by Goldratt is that "common sense is not very common".
So just "solving problems" in an IT environment in most cases will lead to the wrong solutions, just because many people assume that solution X is the best solution without even looking at the problem THEY have. (GitFlow I am looking at you!)
The Phoenix Project was great, I’d recommend its sibling book The Dev Ops Handbook, and the more recent The Unicorn Project (the same narrative story focused on the dev side).
[deleted]
Thanks for the recommendation will check it out
If anyone thinks that The Phoenix Project is horrible, then you have seen nothing before you read The Unicorn Project. This new book is a whole new level of disgusting.
Goldratt's books are pure gems, most of them I have read several times over the years just to remind myself each time that I have not understood something properly the last time I read it.
Goldratt's books are pure gems, most of them I have read several times over the years just to remind myself each time that I have not understood something properly the last time I read it.
Seems to be a pretty vitriolic viewpoint could you please provide some examples of why you think that?
The author lost me at "In order to do DevOps, You Must Follow the Three Ways of DevOps". Software engineers have the tendency to come up with semi religious movements, I have seen good companies going down the drain because of strict adherence to their own made up guidelines.
> If it could automate removing unused permissions, tell us if someone tries to use a removed permission, and shows us reports of both, that would certainly fit well into the second way.
Automatic changes are ticking time-bombs which rarely have any place in production. And this particular example would break production very quickly. For example, a system that performs daily and monthly reports will break when the automated system removes the permissions it uses only once a month.
Automatic changes are ticking time-bombs which rarely have any place in production. And this particular example would break production very quickly. For example, a system that performs daily and monthly reports will break when the automated system removes the permissions it uses only once a month.
If you grant no permissions ever, by default... you don't even have to worry about anything other than the permissions you gave. The challenges to doing this are three fold
1. Operating Systems like Linux, Windows, etc... aren't designed to do this, and can't without a design change.
2. Applications need to be reworked a bit so that they are given file handles instead of telling the OS what files to open. It can looks the same to the user if done correctly.
3. The adoption curve is long on this one.... but systems like Genode are getting there... I look forward to using it sometime in the next year for experimenting.
The post’s argument is illogical.
It doesn’t follow from wanting to speed things up for other teams that you can avoid some kinds of testing in the pipeline by having monitoring. That makes no sense.
It also says a full test suite in the pipeline is not needed because there should be monitoring in production.
Monitoring doesn’t catch all bugs, and letting things fail at a later stage isn’t the right path for a CMM level 5 application that lives and/or serious investment depend on.
Granted, while application monitoring puts additional ongoing load on the application, which slows it down, going against the desired goal of speed, it can help to have application monitoring that may identify slow, insecure, or error prone queries. However, that shouldn’t have to run constantly unless it has low overhead. Also, if problems are just reported after deployment and aren’t a gate to deployment, then you’ll keep pushing crap code into production.
So, what I’m reading here is that Dev Ops, as defined, values getting things through the pipeline quickly, so that when bugs, security, and performance problems are found, those aren’t a Dev Ops problem, and those production issues can be handled by developers, because they’ll have fast feedback from monitoring that maybe they didn’t kill someone yet?
It doesn’t follow from wanting to speed things up for other teams that you can avoid some kinds of testing in the pipeline by having monitoring. That makes no sense.
It also says a full test suite in the pipeline is not needed because there should be monitoring in production.
Monitoring doesn’t catch all bugs, and letting things fail at a later stage isn’t the right path for a CMM level 5 application that lives and/or serious investment depend on.
Granted, while application monitoring puts additional ongoing load on the application, which slows it down, going against the desired goal of speed, it can help to have application monitoring that may identify slow, insecure, or error prone queries. However, that shouldn’t have to run constantly unless it has low overhead. Also, if problems are just reported after deployment and aren’t a gate to deployment, then you’ll keep pushing crap code into production.
So, what I’m reading here is that Dev Ops, as defined, values getting things through the pipeline quickly, so that when bugs, security, and performance problems are found, those aren’t a Dev Ops problem, and those production issues can be handled by developers, because they’ll have fast feedback from monitoring that maybe they didn’t kill someone yet?
The first and most important way of devops is be lazy.
o if you can use something premade, use it
o if you can buy in the service, buy it in (with in reason, make your own buy/build checklist)
o if you have to automate it, do it, then opensource it
o If you have made it, document it, so you don't have to answer questions.
o if you can use something premade, use it
o if you can buy in the service, buy it in (with in reason, make your own buy/build checklist)
o if you have to automate it, do it, then opensource it
o If you have made it, document it, so you don't have to answer questions.
> If you have made it, document it, so you don't have to answer questions.
Dont' communicate one-to-one, but always on an (internal) public medium, preferrably more persistent than chat.
- Need to explain something to someone? Write it a wiki and send them the link.
- Have a great idea that needs brainstorming? Document it as a proposal and open a pullrequest to discus and have it merged into architecture documentation.
- People asking questions about your code in a pullrequest? Don't explain in the PR why you did it, make your code and comments speak for itself so questions are not raised in the first place.
Dont' communicate one-to-one, but always on an (internal) public medium, preferrably more persistent than chat.
- Need to explain something to someone? Write it a wiki and send them the link.
- Have a great idea that needs brainstorming? Document it as a proposal and open a pullrequest to discus and have it merged into architecture documentation.
- People asking questions about your code in a pullrequest? Don't explain in the PR why you did it, make your code and comments speak for itself so questions are not raised in the first place.
The post makes a huge leap which doesn't make sense to me. They claim that it makes sense to not test the use of security privileges in CI, and instead route "reports" to a separate security team, who will then somehow know whether this app should or should not be using the key. And then even block access when the app isn't known to use the privileges. Shouldn't this just be a declarative configuration alongside the app, and then allow the security team to audit which apps are given what privileges? That seems like an actual "least privilege" implementation. This seems more aligned with the first two Ways.
[deleted]
By the way, I would recommend not trying to make your systems clever and “learn”, as the Third Way™️ suggests. Keep them dumb and understandable. The last thing a critical service needs is mysterious, unpredictable, unexplainable behavior. This point alone is a huge red flag for me, brace yourselves for five years of over-engineering as people blindly latch on to advice like this.
I suppose this is the next round of silver bullets after “microservices”.