DevOps didn’t exist when I started as a developer(circleci.com)
circleci.com
DevOps didn’t exist when I started as a developer
https://circleci.com/blog/devops-did-not-exist/
175 comments
When I hear systems administration I assume it's pets not cattle, and that you're administering individual named servers with bash scripts rather than administering a mesos cluster with chef or puppet or whatever.
I'm not sure where I picked up this impression from.
I'm not sure where I picked up this impression from.
It comes from bright, talented SWEs who do not, bless their hearts, understand that IT Operations is an entire profession unto its own, and not merely the grunt-work they are forced to do while working as interns or juniors.
I know that "DevOps" is one of those terms that can mean whatever the user wants it to mean (like "agile" and "REST" and so on), but at least with DevOps, I've mostly only seen 2 types of common definitions:
1) "DevOps teams" are just rebranded operations teams
2) "DevOps teams" are teams that are responsible for both development and operations for their app
I think the first one is the one you're frustrated with, and I agree it's a completely useless label when used like that, but it's not definitely not the only way it's used.
1) "DevOps teams" are just rebranded operations teams
2) "DevOps teams" are teams that are responsible for both development and operations for their app
I think the first one is the one you're frustrated with, and I agree it's a completely useless label when used like that, but it's not definitely not the only way it's used.
Being a "Build-Release Engineer", essentially the sysadmin of the development tool chain, has existed since at least the 90's.
But the roots can be traced to the 70's (the PDP-11 had the compile-link process).
If you were really smart and could program in Assembler they called you a "Systems Programmer".
In the 80's, the PC revolution fostered debates over how much centralized control IT should have.
There was a long-standing debate in computing as to where the software should live (that is on a server or at the client level).
This debate was settled when Google invested heavily in making the browser a powerful and stable platform (think about Javascript in the 90's vs. today).
So the modern situation is server=cloud and client=browser. This creates a demand for people who understand how software is built/deployed/maintained in this environment.
So we slap the Devops label on it and pretend we understand what's going on.
But it's really a debate about how long the current status quo will last before it's subsumed by something else.
But the roots can be traced to the 70's (the PDP-11 had the compile-link process).
If you were really smart and could program in Assembler they called you a "Systems Programmer".
In the 80's, the PC revolution fostered debates over how much centralized control IT should have.
There was a long-standing debate in computing as to where the software should live (that is on a server or at the client level).
This debate was settled when Google invested heavily in making the browser a powerful and stable platform (think about Javascript in the 90's vs. today).
So the modern situation is server=cloud and client=browser. This creates a demand for people who understand how software is built/deployed/maintained in this environment.
So we slap the Devops label on it and pretend we understand what's going on.
But it's really a debate about how long the current status quo will last before it's subsumed by something else.
So I should probably talk to HR about changing my title from Systems Administrator to something more in vogue?
In all seriousness, you will see a 50-90% salary increase if you do, albeit at your next job site, perhaps.
(Speaking pithily as a sysadmin-turned-DevOps-turned-SRE doing vaguely the same work for far more money...)
(Speaking pithily as a sysadmin-turned-DevOps-turned-SRE doing vaguely the same work for far more money...)
DevOps is the intersection of three things.
1. Developers learning that running code is not the same thing as reliable code. Certain things must be put in at design time to allow for operations.
2. Operations people supporting development by formalizing/streamlining the deployment process to have changes occur faster, safer, and more often.
3. Aligning goals and attitudes in such a way that prevents conflict between teams. Development doesn't get to shift the costs of outage prone unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all changes.
In the bad old days, it was common for developers to act with little care for operational outcomes. The responsibility was far removed, and bugfixes didn't win many accolades. Managers want wish fulfillment and pushed developers to toss new features into the mix as fast as possible, resulting in tremendous pressure against ops teams to deploy despite increasingly crufty and debt laden code bases causing unpredictable problems. Ergo, ops teams who blocked all changes for trivial issues because they were judged only based on outages, not feature delivery.
1. Developers learning that running code is not the same thing as reliable code. Certain things must be put in at design time to allow for operations.
2. Operations people supporting development by formalizing/streamlining the deployment process to have changes occur faster, safer, and more often.
3. Aligning goals and attitudes in such a way that prevents conflict between teams. Development doesn't get to shift the costs of outage prone unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all changes.
In the bad old days, it was common for developers to act with little care for operational outcomes. The responsibility was far removed, and bugfixes didn't win many accolades. Managers want wish fulfillment and pushed developers to toss new features into the mix as fast as possible, resulting in tremendous pressure against ops teams to deploy despite increasingly crufty and debt laden code bases causing unpredictable problems. Ergo, ops teams who blocked all changes for trivial issues because they were judged only based on outages, not feature delivery.
Yes, devops is not one person, but team is devops. It comes from standards, that developers should not have access to production data, so you need ops. But on the other hand you don't want to have dev and ops team people who throw shit over the fence.
So basically it comes to having devs, test, ops, sec, persons in one team with common goal of delivering featuers.
Because most friction was on boarders like dev | test | ops | security, all those people being on role specific teams would do what is important for their team. So test people would blame devs, sec people would blame ops, test or devs. When you are in one team that has common goal of delivering feature X and you are measured by delivering feature X, everyone is willing to take pragmatic approach instead of "code has to be perfect", "tests have to be perfect", "security has to be perfect".
I read somewhere "magic happens if you seat testers and developers in the same room", I belive it was "Project Phoenix" book. But I despise from the start having those devs vs test people or test people beeing happy about finding bugs. They should be sad because what matters is delivering feature, so they should be happy when they don't find anything. That said, thay should do all they can to find bugs, because we have common goal: "working software".
So basically it comes to having devs, test, ops, sec, persons in one team with common goal of delivering featuers.
Because most friction was on boarders like dev | test | ops | security, all those people being on role specific teams would do what is important for their team. So test people would blame devs, sec people would blame ops, test or devs. When you are in one team that has common goal of delivering feature X and you are measured by delivering feature X, everyone is willing to take pragmatic approach instead of "code has to be perfect", "tests have to be perfect", "security has to be perfect".
I read somewhere "magic happens if you seat testers and developers in the same room", I belive it was "Project Phoenix" book. But I despise from the start having those devs vs test people or test people beeing happy about finding bugs. They should be sad because what matters is delivering feature, so they should be happy when they don't find anything. That said, thay should do all they can to find bugs, because we have common goal: "working software".
eternalban(4)
This article was frustrating to me. When it started out in the mid-90s (which is when I started my career also), I thought our experiences would be more aligned. However, I've never worked with an operations team.
In all the places I've worked I've been expected to do everything from setting up servers (originally physical servers, later cloud servers), hardening them, installing software, optimizing the software, installing and optimizing the database, creating database schemas and related objects, writing untold number of sql queries, writing server side code, writing front end code, etc.
I was hoping that the article would explain what devops really means today and how I can jump on the devops wagon to hopefully make my job of doing all of the above easier.
In all the places I've worked I've been expected to do everything from setting up servers (originally physical servers, later cloud servers), hardening them, installing software, optimizing the software, installing and optimizing the database, creating database schemas and related objects, writing untold number of sql queries, writing server side code, writing front end code, etc.
I was hoping that the article would explain what devops really means today and how I can jump on the devops wagon to hopefully make my job of doing all of the above easier.
> make my job of doing all of the above easier.
That’s the dream… the reality in my experience is that you have to describe, in detail, the exact steps that you would perform if you were doing it yourself so that somebody else can perform them, verbatim. But you have to describe those steps, exactly, without any access to the target platform where they’ll be performed.
That’s the dream… the reality in my experience is that you have to describe, in detail, the exact steps that you would perform if you were doing it yourself so that somebody else can perform them, verbatim. But you have to describe those steps, exactly, without any access to the target platform where they’ll be performed.
Does it get any better?
I feel like I'm on thin ice whenever I deploy. Granted, I now try to make sure the SAs have to do as little as possible (because now I "understand" the infrastructure, I can build a postinstall that handles all the environment configuration I am allowed under an FID), it still feels nerve wracking to know that a single fuck up means I have to spend another 4-5 hours trying to get tickets to deploy.
Fuck this.
I feel like I'm on thin ice whenever I deploy. Granted, I now try to make sure the SAs have to do as little as possible (because now I "understand" the infrastructure, I can build a postinstall that handles all the environment configuration I am allowed under an FID), it still feels nerve wracking to know that a single fuck up means I have to spend another 4-5 hours trying to get tickets to deploy.
Fuck this.
And its also written from a very recent perspective, arguably I was formally working in devops in the 80's
I was hired as development team member partially as I was a sysadmin on the hardware we were using, our team both wrote the code but also the entire JCL infrastructure that managed the compilation linking, deployment and the running of code across the cluster of 16 or 17 systems.
On of my friends, an IBM guy from the operations side was shocked that we as developers where allowed to deploy and run our own code.
I was hired as development team member partially as I was a sysadmin on the hardware we were using, our team both wrote the code but also the entire JCL infrastructure that managed the compilation linking, deployment and the running of code across the cluster of 16 or 17 systems.
On of my friends, an IBM guy from the operations side was shocked that we as developers where allowed to deploy and run our own code.
My background is more like yours. For me it’s likely a difference in size and type of companies worked for.
> I was hoping that the article would explain what devops really means today and how I can jump on the devops wagon to hopefully make my job of doing all of the above easier.
1. Use AWS Fargate for all of your backend services. Keep the architecture simple enough that complicated service discovery issues etc don’t come up. If you need coordination between services, do it through Redis or similar.
2. Use RDS unless you really need to save money or use unavailable extensions.
3. Use terraform for initially provisioning the above
4. Set up CI/CD such that merges to master automatically update your services. (I like CircleCI’s aws-ecr and aws-ecs orbs for this.)
Pretty simple recipe, but it means no setting up servers, no hardening servers, no installing or optimizing software (other than by adding it to the Dockerfile), no installing or optimizing the database.
This recommendation reflects what modern devops means to me; opinions differ. To me it means:
- Infrastructure as code (terraform rather than clicking buttons in the AWS console then later forgetting which buttons you clicked)
- Immutable infrastructure (aka cattle not pets). Never SSHing into a server again.
- Automated testing and deployment cleanly integrated with existing dev workflow
Obviously there’s a scale at which you have to do something more complex, but I’d say that’s the scale at which you previously would have had an operations team.
Getting rid of the “writing server side code” and “writing front end code” parts is beyond the scope of devops, but you can skip a lot of the “writing server side code” part by using PostgREST. In exchange you may have to write an even-more-untold number of SQL queries, depending on your current practices.
Edit: Someone helpfully pointed out that I forgot to mention anything about logging or monitoring, which is a pretty glaring omission. On that front I strongly recommend Honeycomb. To set it up with Fargate you may need to run it in a sidecar container, but it’s fairly straightforward.
> I was hoping that the article would explain what devops really means today and how I can jump on the devops wagon to hopefully make my job of doing all of the above easier.
1. Use AWS Fargate for all of your backend services. Keep the architecture simple enough that complicated service discovery issues etc don’t come up. If you need coordination between services, do it through Redis or similar.
2. Use RDS unless you really need to save money or use unavailable extensions.
3. Use terraform for initially provisioning the above
4. Set up CI/CD such that merges to master automatically update your services. (I like CircleCI’s aws-ecr and aws-ecs orbs for this.)
Pretty simple recipe, but it means no setting up servers, no hardening servers, no installing or optimizing software (other than by adding it to the Dockerfile), no installing or optimizing the database.
This recommendation reflects what modern devops means to me; opinions differ. To me it means:
- Infrastructure as code (terraform rather than clicking buttons in the AWS console then later forgetting which buttons you clicked)
- Immutable infrastructure (aka cattle not pets). Never SSHing into a server again.
- Automated testing and deployment cleanly integrated with existing dev workflow
Obviously there’s a scale at which you have to do something more complex, but I’d say that’s the scale at which you previously would have had an operations team.
Getting rid of the “writing server side code” and “writing front end code” parts is beyond the scope of devops, but you can skip a lot of the “writing server side code” part by using PostgREST. In exchange you may have to write an even-more-untold number of SQL queries, depending on your current practices.
Edit: Someone helpfully pointed out that I forgot to mention anything about logging or monitoring, which is a pretty glaring omission. On that front I strongly recommend Honeycomb. To set it up with Fargate you may need to run it in a sidecar container, but it’s fairly straightforward.
The article is lost on me too. ~15 years professional experience as a software developer and I never had to deal with an "Operations" team. My understanding is that it's the people that would be doing Amazon's job if you were trying to not use AWS or some other cloud service?
"Devops" means not fighting with those people??? haha
"Devops" means not fighting with those people??? haha
[deleted]
I take it you’ve never worked in any of the Fortune 500, then.
I work for a Fortune 150, but I’ve never worked with a operations, devops, etc. team.
Who monitors production? Who provisions routers, wireless access points, VPN servers?
Just because you are “throwing it over a wall” doesn’t mean the team on the other side doesn’t exist.
Just because you are “throwing it over a wall” doesn’t mean the team on the other side doesn’t exist.
I've worked for a Fortune 500 company for the past 5 years, still nope.
You configure the Cisco routers, then?
The company doubtlessly has a CEO, too, even if you are so compartmentalized as to never speak with her.
The company doubtlessly has a CEO, too, even if you are so compartmentalized as to never speak with her.
I think the mismatch is that I never developed websites, think games, media tools, developer tools. Anytime there was a dependency on a web component I was using a cloud service, external or internal. Internal cloud services were operated by different BU's like a separate business.
Software was distributed via boxed CD or 3rd party web stores. I've never looked into the eyes of someone that configures routers or even heard them mentioned in all my years. They were always essentially in a different company.
That makes sense. It is modern in the sense of being structured as cloud services, yet the proverbial walls between Dev and Ops were tall and sturdy, as is the tradition.
Personally I think DevOps is like religion. It means whatever someone wants it to mean. Some companies think DevOps means having an automated pipeline for building, testing, and shipping code. Other companies think it means micro-services. Others think it means making developers do DBA / SysAdmin work.
All of these things are fine for companies to do. How you run your org is on you. But I wish companies would go deeper than "DevOps" when putting up job posting requirements.
That's like saying they do "security". There's a lot of different ways that can be interpreted. I (as an applicant) only know a handful of those interpretations really well, so it's important that companies clarify these terms in their job descriptions. This way they have the right candidates applying.
All of these things are fine for companies to do. How you run your org is on you. But I wish companies would go deeper than "DevOps" when putting up job posting requirements.
That's like saying they do "security". There's a lot of different ways that can be interpreted. I (as an applicant) only know a handful of those interpretations really well, so it's important that companies clarify these terms in their job descriptions. This way they have the right candidates applying.
> Personally I think DevOps is like religion
As long as no-one is knocking on my door asking if I have five minutes to discuss Agile development methodologies.
I could reasonably defend most of what people attach to the "DevOps" buzzword as sane practices most of us were already doing before the hoopla.
Agile (with a capital A) is the absolute worst thing that has ever happened to the software industry. Kill off that cult and you can DevOps my work with containerized OWASP Gitflows until the end of days for all I care.
Snark aside, Dave Thomas makes a compelling argument for trying to choose terms that are as hard as possible for others to co-opt for selling snake oil (anecdotally, I encounter people on a weekly basis who are self-described Agile or DevOps experts, but have no technical background whatsoever).
As long as no-one is knocking on my door asking if I have five minutes to discuss Agile development methodologies.
I could reasonably defend most of what people attach to the "DevOps" buzzword as sane practices most of us were already doing before the hoopla.
Agile (with a capital A) is the absolute worst thing that has ever happened to the software industry. Kill off that cult and you can DevOps my work with containerized OWASP Gitflows until the end of days for all I care.
Snark aside, Dave Thomas makes a compelling argument for trying to choose terms that are as hard as possible for others to co-opt for selling snake oil (anecdotally, I encounter people on a weekly basis who are self-described Agile or DevOps experts, but have no technical background whatsoever).
I once worked at a place full of Agile cultists, including a bunch of "certified" "coaches." They spent about 1/4th of the time in planning meetings, retros, grooming sessions, sending emails about updating percentage complete on jira tickets, etc.
They made some of the dumbest technical decisions I have ever seen, were perpetually rewriting things, changing core APIs, breaking other parts of the system. The stuff barely worked, and this was after 5 years and dozens and dozens of developers.
The place was so dysfunctional, you could not even create your own branch in source control. You had to request it from the IT group that maintained the "enterprise" SCM server. The IT groups were doing their own form of Agile, so these requests could take weeks.
Truly awful.
They made some of the dumbest technical decisions I have ever seen, were perpetually rewriting things, changing core APIs, breaking other parts of the system. The stuff barely worked, and this was after 5 years and dozens and dozens of developers.
The place was so dysfunctional, you could not even create your own branch in source control. You had to request it from the IT group that maintained the "enterprise" SCM server. The IT groups were doing their own form of Agile, so these requests could take weeks.
Truly awful.
Heh centralized teams to request something as simple as a branch to commit code... or to deploy a feature, and not at scale or anything, where “services” are a simple CRUD API with like 4 people who are customers.
Bloated IT organizations. Reminds me of working in the energy industry. Made up jobs, unmotivated workers, and incompetent management including the CTO. C level exec magazines says to literally eat shit? Here, let’s eat warm turds everyone.
Bloated IT organizations. Reminds me of working in the energy industry. Made up jobs, unmotivated workers, and incompetent management including the CTO. C level exec magazines says to literally eat shit? Here, let’s eat warm turds everyone.
I've worked in several places that each had their own unique snowflake take on "Agile".
With Agile being constant, the experiences were wildly different between them.
The key variable to whether they were dysfunctional or not wasn't Agile. It was the quality of the people in management.
With Agile being constant, the experiences were wildly different between them.
The key variable to whether they were dysfunctional or not wasn't Agile. It was the quality of the people in management.
My pet peeve with agile is it is so rigid at most places. Our team had a bug a while back. We all failed to understand something until it was getting QA'd. We told the PO that we'd need at least part of a sprint to fix it and asked if we could get that prioritized in the next sprint. You would have thought we'd asked to kill someone. We wasted a couple hours debating why it couldn't be finished in this the last day of the sprint. WTF?
This is literally the opposite of the agile manifesto and the original intent and the origins from Lean manufacturing.
Take a look at original agile folks who discuss this extensively on twitter.
Look at John Cutler's incredible blog posts: https://medium.com/@johnpcutler
Take a look at original agile folks who discuss this extensively on twitter.
Look at John Cutler's incredible blog posts: https://medium.com/@johnpcutler
[deleted]
The "Agile" most people complain about is the same thing we had before "Agile" came. You've got 2 broad camps: people who really want to do adhoc development and would like an industry accepted word to call doing whatever-the-heck-I-feel-like. The other side are the people who, having risen to a position of influence are desperately trying to add value, even though they don't actually do anything that results in code being written.
In the former camp, the people tend to like "Agile" because of the doctrine of writing code over doing planning. Often you will find PGMs/PMs who will write a story like "Make the site awesome!" with no other details. They usually want an estimate of how long that's going to take, but will refuse to add any more details -- because that's over-complicating the issue. They really just want to take credit for anything good that happened and punish people for anything bad that happened, all while doing absolutely no work. Usually they are actually deluded enough to think that their role is vital.
In the latter camp, usually there is a feeling that the programmers are stupid and this is why everything is in a complete mess. In order to fix things, they want to take control over everything and throw things over the wall at the programmers. Often the people in these camps used to be programmers (or may still think of themselves as programmers) and have always thought, "If only I could make everyone do what I want, then I can fix all the problems". So they spend their days dictating what everyone will do. It gets even more fun when similar people band together to form elite troops of specifiers -- that's where you get the endless meetings.
We've had this since programming in groups began. This is nothing to do with Agile. It wasn't any better at all before Agile. After Agile was introduced, nothing changed at all except now these guys have another name to incorrectly describe what they are doing.
My biggest problem with Agile is that it says nothing at all about how to accomplish the stated goals. That's why it is so attractive to people who don't have a useful methodology for producing software.
Do I want to continuously deliver software? Tick. Do I welcome changes to requirements? Tick. Do I want to deliver software frequently? Tick. Do I want business people working with software developers? Tick. Do I want motivated developers? Tick. Do I want to have face-to-face conversations? Tick. Do I want working software? Tick. Do I want to sustain a constant pace indefinitely? Tick. Do I want excellent software and design? Tick. Do I want simplicity? Tick. Do I want self organising teams (as long as "self" means myself)? Tick. Do I want the team to tune itself to get better? Tick.
Wow! I'm Agile! Whoo hoo! How do I accomplish the above? To quote someone I worked with in a previous job: "Most of this stuff is common sense. We're all the best in the industry and so this should be second nature to us. If anyone can't do these things then they really don't belong on this team". Now that's a strategy!
Mark my words: within the next 10 years there will be a new word for people to rally behind. At first it will make sense and some people will do cool things. But then the people who want to do nothing and the people who want to force others to do their bidding will grab the word and crush any meaning it ever had.
In the former camp, the people tend to like "Agile" because of the doctrine of writing code over doing planning. Often you will find PGMs/PMs who will write a story like "Make the site awesome!" with no other details. They usually want an estimate of how long that's going to take, but will refuse to add any more details -- because that's over-complicating the issue. They really just want to take credit for anything good that happened and punish people for anything bad that happened, all while doing absolutely no work. Usually they are actually deluded enough to think that their role is vital.
In the latter camp, usually there is a feeling that the programmers are stupid and this is why everything is in a complete mess. In order to fix things, they want to take control over everything and throw things over the wall at the programmers. Often the people in these camps used to be programmers (or may still think of themselves as programmers) and have always thought, "If only I could make everyone do what I want, then I can fix all the problems". So they spend their days dictating what everyone will do. It gets even more fun when similar people band together to form elite troops of specifiers -- that's where you get the endless meetings.
We've had this since programming in groups began. This is nothing to do with Agile. It wasn't any better at all before Agile. After Agile was introduced, nothing changed at all except now these guys have another name to incorrectly describe what they are doing.
My biggest problem with Agile is that it says nothing at all about how to accomplish the stated goals. That's why it is so attractive to people who don't have a useful methodology for producing software.
Do I want to continuously deliver software? Tick. Do I welcome changes to requirements? Tick. Do I want to deliver software frequently? Tick. Do I want business people working with software developers? Tick. Do I want motivated developers? Tick. Do I want to have face-to-face conversations? Tick. Do I want working software? Tick. Do I want to sustain a constant pace indefinitely? Tick. Do I want excellent software and design? Tick. Do I want simplicity? Tick. Do I want self organising teams (as long as "self" means myself)? Tick. Do I want the team to tune itself to get better? Tick.
Wow! I'm Agile! Whoo hoo! How do I accomplish the above? To quote someone I worked with in a previous job: "Most of this stuff is common sense. We're all the best in the industry and so this should be second nature to us. If anyone can't do these things then they really don't belong on this team". Now that's a strategy!
Mark my words: within the next 10 years there will be a new word for people to rally behind. At first it will make sense and some people will do cool things. But then the people who want to do nothing and the people who want to force others to do their bidding will grab the word and crush any meaning it ever had.
It’s not really common sense, and the agile I know of says quite a lot about how to accomplish goals, if your goal is “delivering quality software products/services”.
There also is bullshit Agile of course, which you describe well, but that’s no different from any old software process that’s more bureaucracy than content.
My point is that the cynicism ignores actual differences in practice and process that can have an order of magnitude impact on performance, and its not always “obvious” for smart people to adopt such practices.
Consider Lean manufacturing: the Japanese thought it was just a common sense extrapolation of Henry Ford’s system of organization. It still took decades for the US auto manufacturers (including Ford!) to wrap their heads around it.
There were plenty of crooks and charlatans in that field too. It didn’t negate the importance of what Toyota had figured out.
There also is bullshit Agile of course, which you describe well, but that’s no different from any old software process that’s more bureaucracy than content.
My point is that the cynicism ignores actual differences in practice and process that can have an order of magnitude impact on performance, and its not always “obvious” for smart people to adopt such practices.
Consider Lean manufacturing: the Japanese thought it was just a common sense extrapolation of Henry Ford’s system of organization. It still took decades for the US auto manufacturers (including Ford!) to wrap their heads around it.
There were plenty of crooks and charlatans in that field too. It didn’t negate the importance of what Toyota had figured out.
Brutal. Accurately brutal.
What’s your preferred methodology to Agile? Waterfall?
I quite like this as a good resource:
https://agilemanifesto.org/.
Whatever gets the work done and doesn't cause unnecessary burdens while adding no measurable value is good with me at this point.
I'd make do with anything but Agile, which needs to be shot. Out of a cannon. Into the Sun.
https://agilemanifesto.org/.
Whatever gets the work done and doesn't cause unnecessary burdens while adding no measurable value is good with me at this point.
I'd make do with anything but Agile, which needs to be shot. Out of a cannon. Into the Sun.
Genuinely curious, what is it that you don’t like about Agile? I mean, the concept, the processes? The lack of something or too much of something?
Not the parent, but I agree fully with their point - agile is good; but there is a difference between "agile" and "Agile".
Especially in the enterprise world, agile has been overran by worthless certifications and so-called "Agile" consultants/experts, who broadly fall into 1 of 2 groups: project managers looking to cash in, and technically inept "senior developers" looking to cash in by turning consultant.
Every one of which has their own incredibly dogmatic view of what it means to be Agile. And all of whom are skilled bullshit artists, adept at gaining the ear of upper management, proficienct at indoctrinating them into "the one true way"TM. Every one proficient at making developers miserable - all while charging exorbitant fees, and leaving just in time so management is convinced the failures are down to the heathen devs not following The Agile Way.
Standups where you literally have to stand up (simultaneously, even if spread across 3 continents and numerous time zones) and invariably go on for an hour, constant complaints (and of course meetings) about stories not finished before the sprint ends, endless meetings about the line on the bastard burndown chart, backlog refinements that last days or even weeks (invariably followed by story refinements that last days), telling the devs "decisions are up to the team... but you will choose my way", meaningless identikit retrospectives when the devs just want out so they can actually ship something...
Honestly, I'm not sure whether these "consultants" actually believe the dogmatic, quasi-religious snake-oil BS they are selling, or they know full well, but know just as well how much money they make from it.
All I know, is these people give agile a bad reputation, and bring absolute misery to many a developer.
Especially in the enterprise world, agile has been overran by worthless certifications and so-called "Agile" consultants/experts, who broadly fall into 1 of 2 groups: project managers looking to cash in, and technically inept "senior developers" looking to cash in by turning consultant.
Every one of which has their own incredibly dogmatic view of what it means to be Agile. And all of whom are skilled bullshit artists, adept at gaining the ear of upper management, proficienct at indoctrinating them into "the one true way"TM. Every one proficient at making developers miserable - all while charging exorbitant fees, and leaving just in time so management is convinced the failures are down to the heathen devs not following The Agile Way.
Standups where you literally have to stand up (simultaneously, even if spread across 3 continents and numerous time zones) and invariably go on for an hour, constant complaints (and of course meetings) about stories not finished before the sprint ends, endless meetings about the line on the bastard burndown chart, backlog refinements that last days or even weeks (invariably followed by story refinements that last days), telling the devs "decisions are up to the team... but you will choose my way", meaningless identikit retrospectives when the devs just want out so they can actually ship something...
Honestly, I'm not sure whether these "consultants" actually believe the dogmatic, quasi-religious snake-oil BS they are selling, or they know full well, but know just as well how much money they make from it.
All I know, is these people give agile a bad reputation, and bring absolute misery to many a developer.
GordonS gets it.
This is true. Not long ago (about six months) a project I was working on was assigned a scrum-master (tip: You can just be the person saying "any blockers?" and "next". You don't need to be master of anything.)
Genuinely nice person, it's anecdotal, but just making it clear I'm not bashing them. Regardless, after a stand-up in the very first week they just volunteered that they had no technical background whatsoever, and couldn't write a line of code. They were at the same time actively making architecture decisions outside of the stand-ups and relaying this to the P/O.
It was a nightmare.
The above is right, Agile (with a capital A) has become little more than a way for non-technical people to have lucrative careers in software by destroying the otherwise promising careers of technical people in software.
This is true. Not long ago (about six months) a project I was working on was assigned a scrum-master (tip: You can just be the person saying "any blockers?" and "next". You don't need to be master of anything.)
Genuinely nice person, it's anecdotal, but just making it clear I'm not bashing them. Regardless, after a stand-up in the very first week they just volunteered that they had no technical background whatsoever, and couldn't write a line of code. They were at the same time actively making architecture decisions outside of the stand-ups and relaying this to the P/O.
It was a nightmare.
The above is right, Agile (with a capital A) has become little more than a way for non-technical people to have lucrative careers in software by destroying the otherwise promising careers of technical people in software.
Appreciate you explaining this point further, I didn’t realise there was a difference between ‘agile’ vs ‘Agile’ and the only other methodology I had heard of was waterfall which also seems to be universally disliked.
"Agile" is just an instance of "agile" which is an abstract concept. Hence you can't actually do "agile". Therefore every form of agile you see is "Agile" and they're quote "doing agile wrong".
Read the manifesto. It literally says handful of almost completely meaningless, unimplementable things.
Any attempt to actually take that and turn it into something is destined to be sharply divorced from reality.
But the real fun comes when you do "white water rafting" style development. This is when there is all the ceremony of "Agile" (read: a "fast paced environment") with all the planning of waterfall and absolutely everything is top priority. You and your coworkers are constantly thrown around trying not to smash into rocks or capsize the boat, meanwhile being pulled violent in every direction. Somehow you continue to make your way down the river eventually with fewer and fewer co-workers as she casualties mount.
Fun times.
The actual methodology is completely irrelevant. Working under terrible management is terrible. Working under terrific management is terrific.
Read the manifesto. It literally says handful of almost completely meaningless, unimplementable things.
Any attempt to actually take that and turn it into something is destined to be sharply divorced from reality.
But the real fun comes when you do "white water rafting" style development. This is when there is all the ceremony of "Agile" (read: a "fast paced environment") with all the planning of waterfall and absolutely everything is top priority. You and your coworkers are constantly thrown around trying not to smash into rocks or capsize the boat, meanwhile being pulled violent in every direction. Somehow you continue to make your way down the river eventually with fewer and fewer co-workers as she casualties mount.
Fun times.
The actual methodology is completely irrelevant. Working under terrible management is terrible. Working under terrific management is terrific.
You should read the original waterfall paper as it is basically agile in spirit: http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970...
Waterfall had its place in military and aerospace, where the requirements and API were well-defined, and to a lesser extent in life-critical medical systems development.
But in even those cases, it was largely replaced by Spiral, which is similar to agile and pre-dates it.
Edit: Spiral is like going down the Waterfall two or more times. Similar to agile, but formally documented, with management and customer sign-off up the line, and with a much longer cycle time.
But in even those cases, it was largely replaced by Spiral, which is similar to agile and pre-dates it.
Edit: Spiral is like going down the Waterfall two or more times. Similar to agile, but formally documented, with management and customer sign-off up the line, and with a much longer cycle time.
My experience of that which has been called 'agile' is that it's an excuse by programmers not to do the boring stuff, so they can got straight into the coding.
Not to plan, or even think too much, definitely not to document (sometimes not write anything down at all), often not to test except in a token form and sometimes not even that.
Is that agile? That's how 'agile' has been done where I work anyway. Maybe I've been unlucky but 'agile' like that is a loathsome, unprofessional practice that can't produce good results.
Not to plan, or even think too much, definitely not to document (sometimes not write anything down at all), often not to test except in a token form and sometimes not even that.
Is that agile? That's how 'agile' has been done where I work anyway. Maybe I've been unlucky but 'agile' like that is a loathsome, unprofessional practice that can't produce good results.
Yes, I understand. While I’ve never only attributed that to Agile alone, but also this sort of brogrammer-code-is-the-real-product, it sometimes feels like a maturing industry sort of rolled back into its teenage years. Diagrams? Solution models (up front thinking)? Document for future reference? Nah, we don’t want to do that.
It’s sad. Just recently had an experience like this with developers I manage. I do loathe the brogrammer culture more than anything...
It’s sad. Just recently had an experience like this with developers I manage. I do loathe the brogrammer culture more than anything...
A significant chunk of software is a dead simple one off and does not need much in the way of design or maintenance. At the other end you have software that risks human lives and or billions in damages.
Picking the right process for where each project fits on that continuum is surprisingly difficult. Cowboy coding can be anything from a horrific idea to the most efficient setup, unfortunately people are rarely flexible enough to know the difference.
Picking the right process for where each project fits on that continuum is surprisingly difficult. Cowboy coding can be anything from a horrific idea to the most efficient setup, unfortunately people are rarely flexible enough to know the difference.
That’s why I always push the message that we choose the right amount of bureaucracy for the task at hand. Simple change / feature? No need to make a lot of documents for that? Fluffy idea that several people needs to work on? We need some mocks, diagrams and take requirements from client.
I think that’s a healthy way of doing it, but sadly many almost have to be pushed to do it, even with bigger things.
I think that’s a healthy way of doing it, but sadly many almost have to be pushed to do it, even with bigger things.
> feels like a maturing industry sort of rolled back into its teenage years
Beautiful description! That is so on the nail.
Beautiful description! That is so on the nail.
There are cult-like tendencies in part of the agile community. I don't see it in the teams I have worked in, but in forums on LinkedIn, by those that see their career as agile specialists, experts in coaching the true way to perform agile software development. Many of them see themselves as catalysts, and that they with clever questioning will make teams grow. They don't themselves want to have a delivery responsibility, but they want to stand on the sidelines and "coach".
That I can understand and I’ve seen that too. And now Scaled Agile Framework is coming in hot as the enterprise version.
I’m actually proud of the parts of agile we work with, which primarily enables us to react changing needs and priorities quickly (unlike heavy waterfall).
After 22+ years in the industry I’ve come to realize that any new concept or concept that gets traction (CQRS, Event storming, AI) will produce a religious following. It’s kinda sad, because causes as lot of noise and friction.
I’m actually proud of the parts of agile we work with, which primarily enables us to react changing needs and priorities quickly (unlike heavy waterfall).
After 22+ years in the industry I’ve come to realize that any new concept or concept that gets traction (CQRS, Event storming, AI) will produce a religious following. It’s kinda sad, because causes as lot of noise and friction.
Shape Up sounds interesting, particularly for remote-only teams in non-massive companies. Haven't tried it but it sounds interesting.
https://basecamp.com/shapeup
https://basecamp.com/shapeup
agile, lowercase a, is not bad.
Right, I didn’t understand there was a difference. Thanks for that, I’ll look into it.
It is not a formal difference, but agile has evolved into an industry itself, with certification ladders and gurus. So on one hand you have the one page agile maniefesto, on the other people who make their living on preaching how teams should work, shouting from the sidelines.
my bar for "DevOps" on my (SaaS) team is that the engineers responsible for the platform the product runs on are involved in the discussions about the product we build.
everything that comes after that: pipelines, testing, release engineering, migrations, post-mortems, whatever -- it's all a result of your infrastructure engineers having a stake in the thing your company is building. I have yet to see a definition of DevOps that invoked some sort of tech stack that made any sense to me. the processes and tools emerge from a cooperation between stakeholders and a shared responsibility for the delivered product.
everything that comes after that: pipelines, testing, release engineering, migrations, post-mortems, whatever -- it's all a result of your infrastructure engineers having a stake in the thing your company is building. I have yet to see a definition of DevOps that invoked some sort of tech stack that made any sense to me. the processes and tools emerge from a cooperation between stakeholders and a shared responsibility for the delivered product.
DevOps is the channel in slack I check when my code didn't update. That the directs me to somewhere else where I just sort of randomly click things until it works or gets worse.
But yeah I'm with you on the wild definitions, it can be anything from some overnight IT grunt, to something far more respected and involved depending on how the company did things.
But yeah I'm with you on the wild definitions, it can be anything from some overnight IT grunt, to something far more respected and involved depending on how the company did things.
> But I wish companies would go deeper than "DevOps" when putting up job posting requirements.
I am not sure this problem is limited to DevOps. Once I applied for a job thinking I understood the job description. During the interview, I had to find out that they put only half the job in the description (the good parts).
Afterward, I took a look on the job description again asking myself if I misinterpreted the description but came to the conclusion that they just didn't want to write about the downsides of the job (not even in management speak like 'You love a challenge?').
I am not sure this problem is limited to DevOps. Once I applied for a job thinking I understood the job description. During the interview, I had to find out that they put only half the job in the description (the good parts).
Afterward, I took a look on the job description again asking myself if I misinterpreted the description but came to the conclusion that they just didn't want to write about the downsides of the job (not even in management speak like 'You love a challenge?').
Firms don't want you to know it, but they are struggling to hire competent IT professionals to an extreme degree. Recruiters are being regularly ghosted (like they used to do to candidates) and turn over is increasing. That's why you see them resulting to tricks like this. One thing to keep in mind is that most managers working today cut their teeth when unemployment was at 10% or so, as such they really only know about bait and switch tactics and the stick. Carrots aren't something that really fits into their mental model.
In short, to anyone reading this. Now is the time to make your move.
In short, to anyone reading this. Now is the time to make your move.
> Recruiters are being regularly ghosted (like they used to do to candidates)
Not sure if this is the same thing (with "ghosted"), but I have 3 rock-bottom criteria before I'll respond to a recruiter's email:
* Send to the email on my resume, not my personal one.
* Show that they've read (or at least keyword-matched) my resume.
* Something in the job description should at least be vaguely enticing, such as a technology I have on my resume. The same thing can hit both points 2 and 3, the depending on how the email is structured.
In nearly 10 years, a grand total of one recruiter got all these points. Half of them failed on the first one.
Not sure if this is the same thing (with "ghosted"), but I have 3 rock-bottom criteria before I'll respond to a recruiter's email:
* Send to the email on my resume, not my personal one.
* Show that they've read (or at least keyword-matched) my resume.
* Something in the job description should at least be vaguely enticing, such as a technology I have on my resume. The same thing can hit both points 2 and 3, the depending on how the email is structured.
In nearly 10 years, a grand total of one recruiter got all these points. Half of them failed on the first one.
> Some companies think DevOps means having an automated pipeline for building, testing, and shipping code.
This seems to be the most common interpretation, but I largely agree with you on no hard and fast rule here.
This seems to be the most common interpretation, but I largely agree with you on no hard and fast rule here.
It's because it ultimately does mean that.
I mean, sure, if we're being pedantic about it, devops is about the silos coming closer together, cross learning, culture and what have you, etc.
But the outcome of that the cross learning is pretty much release engineering, that includes the above and the products giving more data for helping run them, and simplicity leading to being easier to fix, and do the aforementioned things.
I also think we should just get over it and probably just adopt SRE, Release Engineer and Product. Not because I <3 google, just because the split makes a tad bit more sense.
I mean, sure, if we're being pedantic about it, devops is about the silos coming closer together, cross learning, culture and what have you, etc.
But the outcome of that the cross learning is pretty much release engineering, that includes the above and the products giving more data for helping run them, and simplicity leading to being easier to fix, and do the aforementioned things.
I also think we should just get over it and probably just adopt SRE, Release Engineer and Product. Not because I <3 google, just because the split makes a tad bit more sense.
> Others think it means making developers do DBA / SysAdmin work.
I've been in that spot many times. In my last job, I did engineering (networking, server, etc) / support for 7 data centers. We started to support of 3rd party "cloud" products, and then they dropped the DCs they gave me the title of DevOps Engineer IV. When they did that, I didn't have any Dev or Ops responsibilities.
I've been in that spot many times. In my last job, I did engineering (networking, server, etc) / support for 7 data centers. We started to support of 3rd party "cloud" products, and then they dropped the DCs they gave me the title of DevOps Engineer IV. When they did that, I didn't have any Dev or Ops responsibilities.
People can’t agree what devops is, sure. But I feel like it’s extremely easy to tell what devops isn’t. If you don’t have any confidence in your releases before you’ve done extensive ad-hoc manual probing you aren’t doing devops. If your using oracle and Delphi even though none of the developers would chose it because “that’s what management decided” you are not doing devops.
The rest of the story is just about the tooling and management decisions that support getting out of those pits. You use micro services because it allows developers to use whichever tools they want as long as they solve the task. You use automated tests and deployment pipelines so the team and organizational procedure to be confident in releases shifts from becoming “talk to Tom and get his blessing after he’s spend a week probing everything” to “well if it passed all tests and deployed then it’s a go.
It’s not like the technical issues suddenly disappear. You need to set up your tests to a level where you are confident in the pipeline. You still need to spend the time and resources making sure your microservice infrastructure is working.
But at the end of it you’ve removed the power of the gits who where previously controlling the techs allowed and the releases, and your letting teams move forwards through proving their work rather than constant audiences with individuals who are gatekeepers because they spend 20 years in the same organization and feel that everything invented in the last decade is scary.
The rest of the story is just about the tooling and management decisions that support getting out of those pits. You use micro services because it allows developers to use whichever tools they want as long as they solve the task. You use automated tests and deployment pipelines so the team and organizational procedure to be confident in releases shifts from becoming “talk to Tom and get his blessing after he’s spend a week probing everything” to “well if it passed all tests and deployed then it’s a go.
It’s not like the technical issues suddenly disappear. You need to set up your tests to a level where you are confident in the pipeline. You still need to spend the time and resources making sure your microservice infrastructure is working.
But at the end of it you’ve removed the power of the gits who where previously controlling the techs allowed and the releases, and your letting teams move forwards through proving their work rather than constant audiences with individuals who are gatekeepers because they spend 20 years in the same organization and feel that everything invented in the last decade is scary.
> If your using oracle and Delphi even though none of the developers would chose it because “that’s what management decided” you are not doing devops.
What does that have to do with DevOps? Clearly that's not an ideal situation, but that has nothing to do with CI/CD pipelines, developers handling operations work, etc.
What does that have to do with DevOps? Clearly that's not an ideal situation, but that has nothing to do with CI/CD pipelines, developers handling operations work, etc.
I think they're driving at "management is dictating technical decisions"; many people view DevOps as a cultural shift (if not a mindset) that includes empowering developers.
Interestingly, your question of illustrative of the "DevOps has no clear, standard definition" problem.
Interestingly, your question of illustrative of the "DevOps has no clear, standard definition" problem.
That brings to mind another interesting point--even if you can tell what DevOps isn't, so much of a "DevOps" role at any particular company is dependent on the technologies that the company has adopted. I imagine DevOps looks wildly different at a company that deploys a monolith to an EC2 autoscaling group versus the company that deploys microservices to Kubernetes. Similarly, whether you use a monorepo or a multirepo. These sorts of concerns drive virtually everything about the DevOps role--the entire structure of the CI/CD pipeline, the strategies for fast rollbacks, the local developer tooling, how much time you spend at which layer (e.g., if you deploy to EC2, you're probably spending a good chunk of upfront time configuring log exfiltration and process management versus something like Fargate). And this says nothing about which cloud provider you're familiar with, or the cultural concerns (one company's DevOps might be chartered with driving cultural change while another's is taking strict orders from management).
Given how large the gap between any two given DevOps positions, I wonder how valuable it is to have "DevOps" job reqs or conferences or other "DevOps" things.
Given how large the gap between any two given DevOps positions, I wonder how valuable it is to have "DevOps" job reqs or conferences or other "DevOps" things.
Nailed it.
I also got a chance to work in such an organization in the 00's, where dev and ops closely aligned with each other's work to produce truly reliable, repeatable, fast, quality products, often. It was fantastic. Nobody would say "oh but you can't do that, that's not Infrastructure as Code!" They would say, "what can we all do to make this better?"
I also got a chance to work in such an organization in the 00's, where dev and ops closely aligned with each other's work to produce truly reliable, repeatable, fast, quality products, often. It was fantastic. Nobody would say "oh but you can't do that, that's not Infrastructure as Code!" They would say, "what can we all do to make this better?"
In the 90's and 00's I was responsible for maybe 20-30 servers, now I am responsible for somewhere around the 2000 mark..
While "oh but you can't do that, that's not Infrastructure as Code!" sounds lame, and your co-worker should learn to be a better communicator, things need to be reproducible when working with more complex systems and that calls for more rigorous review practices.
Please consider this: It's our pagers that go off at 3am when your code fails..
In my experience ops call the developers when something breaks overnight in any case.
In our organisation, developers have need nor clearance to access live customer data. This means they stay out of production and we have to deal with breakage.
Usually I can just revert to an earlier version and file a bug report, sometimes I have to cherry pick.
Having developers "fix" things in production under time-stress at 3am without proper code review sounds like extremely poor practice.
Usually I can just revert to an earlier version and file a bug report, sometimes I have to cherry pick.
Having developers "fix" things in production under time-stress at 3am without proper code review sounds like extremely poor practice.
So you have some one every shift who is familiar with every part of a complex system and the requisite programming languages.
And reverting part way through a monthy telco billing run might not be the best idea.
And in my case they also looked after online services and other systems.
And reverting part way through a monthy telco billing run might not be the best idea.
And in my case they also looked after online services and other systems.
Preventing calling the devs requires dev to front-load ops. Ops needs triage documents, remediation strategies, architectural and workflow diagrams, dependency charts, distributed tracing, intelligent logging, custom application metrics, and tests they can run during incidents to isolate causes.
The devs can do all of this on their own, teach ops how to use it, and then they'd only be called when it was a code issue. But as a dev, you probably don't know all of the above, so ops has to go to dev and be like, "hey y'all, if you don't want to be called, this is what we need."
And this is what DevOps is intended to fix: get everyone in a room, talk about problems, find solutions among everyone. If your org isn't doing this, you can start the change.
The devs can do all of this on their own, teach ops how to use it, and then they'd only be called when it was a code issue. But as a dev, you probably don't know all of the above, so ops has to go to dev and be like, "hey y'all, if you don't want to be called, this is what we need."
And this is what DevOps is intended to fix: get everyone in a room, talk about problems, find solutions among everyone. If your org isn't doing this, you can start the change.
Isn't it amazing that working on/as a team is seen as an outlier?
We need more of this.
We need more of this.
I found this line interesting, everything else considered:
DevOps is NOT…easily achieved nor implemented
Debatable, at least IMO. DevOps isn't easily achieved nor implemented if you're trying to implement ALL THE THINGS to say you did and check-off a series of "We did DevOps thing x" boxes as so many companies appear to want-at least from reading various DevOps-y job descriptions lately.
It is easier implemented if, like Gene Kim tells us in "The DevOps Handbook"-we start our DevOps transformations with a small, sympathetic team and iterate outward.
DevOps is NOT…easily achieved nor implemented
Debatable, at least IMO. DevOps isn't easily achieved nor implemented if you're trying to implement ALL THE THINGS to say you did and check-off a series of "We did DevOps thing x" boxes as so many companies appear to want-at least from reading various DevOps-y job descriptions lately.
It is easier implemented if, like Gene Kim tells us in "The DevOps Handbook"-we start our DevOps transformations with a small, sympathetic team and iterate outward.
In 1997, when my first code I wrote went live, they gave me a pager and said "Welcome to Ops." Each time the code in production had an abnormal end, I got an email. I was expected to fix it. Managers and directors discussed these metrics.
The change in job description from just "operations" to "developer operations" over the past decade is best captured by the growing number of systems that must be managed per person, often larger by a factor of 10 times or more (e.g. going from 10 systems per person to 100+).
I appreciate the self-reflective views on this "new" job role, but they often seem to miss the rather basic market forces at work. More systems, more automation, new skills and demands.
I appreciate the self-reflective views on this "new" job role, but they often seem to miss the rather basic market forces at work. More systems, more automation, new skills and demands.
Could web developers please stop using floating top bars? It breaks the UX of the "Page Down" button. It's annoying beyond imagination. Thank you.
I resisted. Management insisted.
Good thing it was a non public app instead of a blog
Good thing it was a non public app instead of a blog
Sidebar: does anyone else get the feeling that infrastructure as code is just cloud vendor lock-in by another name? Especially since the output of the code ends up being wild unstructured JSON/YAML files with no spec or discernible schema.
My favorite is how in the Microsoft toolchain you can build a CI pipeline in a visual UI on the right that automatically updates the YAML file on the left. “We know you don’t want YAML, but we also know your boss says you need infra-as-code...”
My favorite is how in the Microsoft toolchain you can build a CI pipeline in a visual UI on the right that automatically updates the YAML file on the left. “We know you don’t want YAML, but we also know your boss says you need infra-as-code...”
Infra-as-code is mainly just a method to force people to write things down, rather than the default of nobody writing anything down, and people just manually making changes that nobody else knows about, making it really difficult to manage changes. Even a bash script that creates an ec2 instance is infrastructure-as-code.
OTOH, I kind of hate Terraform now, because you can use it in a way that will surprise you and blow up, leaving a mess that you have to clean up manually. It's more "infrastructure as code" than Ansible, but Ansible will at least attempt to fix things that aren't "right". So everything that has a certain label is not by definition superior to all alternatives. (I actually hate both tools for different reasons, but they have their uses)
OTOH, I kind of hate Terraform now, because you can use it in a way that will surprise you and blow up, leaving a mess that you have to clean up manually. It's more "infrastructure as code" than Ansible, but Ansible will at least attempt to fix things that aren't "right". So everything that has a certain label is not by definition superior to all alternatives. (I actually hate both tools for different reasons, but they have their uses)
You might want to look into Terraform [1], which works across multiple cloud vendors, with pluggable open-source "providers" facilitating the communication with the backend cloud platform.
[1]: https://www.terraform.io/
[1]: https://www.terraform.io/
Terraform is great but if you want to change to a different cloud provider you pretty much have to rewrite everything
Terraform is okay, if you don't value things like "loops that aren't an awful hack", but in 2019 Pulumi is significantly better and their cloud-agnostic implementation actually kinda works okay.
We should make an effort to redefine how we use 'infrastructure-as-code', because right now Pulumi/Troposphere/Azure PowerShell and Ansible/Terraform/etc. are indistinguishable when viewed that way, despite the former being much more useful.
Infrastructure-as-verisonable-flat-configuration-files should not be thought of as infrastructure-as-code.
Infrastructure-as-verisonable-flat-configuration-files should not be thought of as infrastructure-as-code.
Pulumi would be perfect if the community edition was simply self hosted without support vs free with a single user only. At a stingy small org, I have no hope of ever using it instead of simply installing Terraform when the starter edition excludes secrets management and the API.
It's a bit rocky and I've been filing bugs to help, but S3 state support is strictly no worse than Terraform's and supports secrets just fine.
Haven't heard of Pulumi before your comment but will be sure to check it out now :)
> Terraform is great but if you want to change to a different cloud provider you pretty much have to rewrite everything
While true, the process is closer to a variable name refactor than to a complete re-write. With Terraform, you have the stack described in code and the resource names translate cleanly across cloud vendors. Certainly more so than a word doc describing infrastructure setup steps.
While true, the process is closer to a variable name refactor than to a complete re-write. With Terraform, you have the stack described in code and the resource names translate cleanly across cloud vendors. Certainly more so than a word doc describing infrastructure setup steps.
Yeah? If you build on top of popular open infra that is hosted by a variety of providers, it shouldn't be too hard. You can build on top of K8s, Postgres, and S3-alike APIs, then use Terraform to move across AWS, GoogleCloud and DO without having to rewrite a thing.
You still need to have pretty intimate knowledge of your stack provider though. Terraform is a way to formally express your infrastructure via code, but you still need to know how all of your components need to be configured and wired together. It doesn't give you any of that.
You’re using a cloud provider; you’re already locked in. But you’re also locked in like you’re locked in by your OS or your programming language or your ISP, which is to say: the whole point of cloud providers is to abstract away a certain set of things. You can look at it as lock-in, or as using a platform that enables you to solve the problems you care about. You can look at your OS as lock-in, or a tool that enables you to solve the problems you care about. If your goal is to provide cloud infrastructure quickly and easily and without building a data center, then lock-in doesn’t really matter. And if it does, there are more expensive, more appropriate options.
Lock-In is defined as, "an arrangement according to which a person or company is obliged to deal only with a specific company."
If you mean in terms on Kubernetes, the idea is identical JSON would work on AWS, GCP, Azure, Digital Ocean. If you want to go cloudless, you can run k8s on your own infrastructure. And if you want to run on your own infra, the need for configuration doesn't go away. Given how open the specification is for the JSON documents, I find it difficulty in characterizing it as lock-in. The configuration JSON is highly reusable in variety of contexts, with the examples I gave above.
How is the opaque visual UI better? IaC doesn't unlock you from a vendor but at least you have a reproducible config file you can check in and version.
I tried Terraform this weekend and it was awesome to use. Spinning up a new ec2 instance was very quick. Almost just as quick when you spin up multiple instances.
It supports "providers", which are interfaces to the different vendors (aws, foogle cloud, etc), and "provisioners", which are frameworks for configuring a running instance (chef? Docker? Zookeeper?)
It supports "providers", which are interfaces to the different vendors (aws, foogle cloud, etc), and "provisioners", which are frameworks for configuring a running instance (chef? Docker? Zookeeper?)
Plausibly, if what I want is "start a kubernetes cluster with an ingress and a database", that's portable to every cloud provider. However, with terraform, you have to write platform-specific code to do this that isn't portable between providers.
[deleted]
No, the skills of your people are the lock-in.
cant believe so many people still get it wrong.
devops means Developer/Operations.
means, the sysadmin/sysops/ops guy now has to know things about development to center his daily-doing around code (ideally the code dictates what is going on [Infrastructure as Code]).
a very simple example to this: back then the ops guy or sysadmin did everything by hand. today, he uses code to get things done. bash vs. a HTTP JSON API. manually by hand vs. ansible [markup language yaml], etc.
so many people just dont know or understand what devops means. but it is so easy.
it is the fusion of development and operations.
thus, while you need two people back then, now you only need one person to do the same job.
a devops is a sysadmin who ideally knows how to code.
a devops is a developer who evolved into a sysadmin or is doing both because he learned it.
devops is development/operations.
it has nothing to do with agile, waterfall or a "mindset".
devops means Developer/Operations.
means, the sysadmin/sysops/ops guy now has to know things about development to center his daily-doing around code (ideally the code dictates what is going on [Infrastructure as Code]).
a very simple example to this: back then the ops guy or sysadmin did everything by hand. today, he uses code to get things done. bash vs. a HTTP JSON API. manually by hand vs. ansible [markup language yaml], etc.
so many people just dont know or understand what devops means. but it is so easy.
it is the fusion of development and operations.
thus, while you need two people back then, now you only need one person to do the same job.
a devops is a sysadmin who ideally knows how to code.
a devops is a developer who evolved into a sysadmin or is doing both because he learned it.
devops is development/operations.
it has nothing to do with agile, waterfall or a "mindset".
CircleCI trusts 8 analytics companies with your source code and API tokens
https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecur...
https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecur...
It certainly did exist when I started my career a decade earlier than this guy. It was called the integration team, which was responsible for coordination of the release. In the 80ies.
It was also responsible for the build servers, the SCM, and checking and prioritizing all the different parts and fixes of the various dev teams, and checking back with the PM and testers. Only hackers did cowboy coding, companies preferred processes and best practices. That's it was called "Software Engineering" and not hacking. I heard Margaret Hamilton coined that term in the 60ies for the Apollo project, which she managed.
It was also responsible for the build servers, the SCM, and checking and prioritizing all the different parts and fixes of the various dev teams, and checking back with the PM and testers. Only hackers did cowboy coding, companies preferred processes and best practices. That's it was called "Software Engineering" and not hacking. I heard Margaret Hamilton coined that term in the 60ies for the Apollo project, which she managed.
Get 'em! Pretty sure it's just called "teamwork!"
I think the evolution is Waterfall to Agile to DevOps.
1. Waterfall is a cycle time of weeks
2. Agile is a cycle time of days
3. DevOps is the same cycle time as agile but delivering the the user instead of just showing a demo at the end of a sprint.
1. Waterfall is a cycle time of weeks
2. Agile is a cycle time of days
3. DevOps is the same cycle time as agile but delivering the the user instead of just showing a demo at the end of a sprint.
[The (or any) Internet didn't exist when I started as a developer]
DevOps is about how CTO organizes engineering.
Is there an Infra team that supports everyone, maybe a separate Ops/security/etc teams that supports everyone. Then a bunch of App/Service dev teams. This is not devops.
Or is there only* a bunch App/Service teams, each wholly self-supporting and independent. Each doing their own infra, dev, ops. Maybe some teams' "Service" is used by only other internal teams (point being you can break up scope as fine grained as you want). The key is teams are self-supporting, taking care of the entire life-cycle of an App/Service.
Most "devops" sort of isn't or fails cause it's not "deployed as an organizational system, top-down (from CTO) across the whole organization.
All the things people typically bike shed over re: devops don't matter. Are you independent and self-contained? Are you responsible for entire life-cycle of Product/Service/App/Etc? Does the buck stop with you? Then you are devops.
DevOps is about how CTO organizes engineering.
Is there an Infra team that supports everyone, maybe a separate Ops/security/etc teams that supports everyone. Then a bunch of App/Service dev teams. This is not devops.
Or is there only* a bunch App/Service teams, each wholly self-supporting and independent. Each doing their own infra, dev, ops. Maybe some teams' "Service" is used by only other internal teams (point being you can break up scope as fine grained as you want). The key is teams are self-supporting, taking care of the entire life-cycle of an App/Service.
Most "devops" sort of isn't or fails cause it's not "deployed as an organizational system, top-down (from CTO) across the whole organization.
All the things people typically bike shed over re: devops don't matter. Are you independent and self-contained? Are you responsible for entire life-cycle of Product/Service/App/Etc? Does the buck stop with you? Then you are devops.
> [The (or any) Internet didn't exist when I started as a developer]
Internet is widely considered to have started in 1969. Although I'd consider something between 1982-1986 as the years when it truly became an international network.
I've been in this business only for less than 25 years, you make me feel so young. :-)
During that time, internet has just always existed. Ah, those loong boolean logic web search (hxxp://altavista.digital.com/) queries during my early years.
Internet is widely considered to have started in 1969. Although I'd consider something between 1982-1986 as the years when it truly became an international network.
I've been in this business only for less than 25 years, you make me feel so young. :-)
During that time, internet has just always existed. Ah, those loong boolean logic web search (hxxp://altavista.digital.com/) queries during my early years.
> DevOps is NOT… a job title or role
In which case DevOps as it exists in the real world is roughly on a par with 'agile' practices in the real world. Nothing like they were originally conceived, and nothing like the wildly optimistic descriptions of them that are shared amongst practitioners.
In which case DevOps as it exists in the real world is roughly on a par with 'agile' practices in the real world. Nothing like they were originally conceived, and nothing like the wildly optimistic descriptions of them that are shared amongst practitioners.
Agile: We don't have a formalized process of developing software, just wing it.
DevOps: We don't have admins, so you are reponsible to run the systems you develop.
This is at least how I perceive it. But the second point is not the worst thing that could have happened. I hate to write configs as much as anyone, but it has given me a better perspective on how to design systems that actually... you know... run.
DevOps: We don't have admins, so you are reponsible to run the systems you develop.
This is at least how I perceive it. But the second point is not the worst thing that could have happened. I hate to write configs as much as anyone, but it has given me a better perspective on how to design systems that actually... you know... run.
Generally Ive found tight feedback loops to be incredibly productive. Whether its for product, release, design, development. Having the ability to make modifications with feedback from customers, systems, business partners and colleagues changes everything.
Tight feedback loops is precisely why I dislike the current devops culture.
For me, a tight feedback loop is cycling between repl/ide/compiler/tests as quickly as possible.
A loose feedback loop is pull requests, docker, CI, terraform. Don't get me wrong - I appreciate a fast, smooth 'release' pipeline. I just hate that it's now become the 'development' pipeline.
For me, a tight feedback loop is cycling between repl/ide/compiler/tests as quickly as possible.
A loose feedback loop is pull requests, docker, CI, terraform. Don't get me wrong - I appreciate a fast, smooth 'release' pipeline. I just hate that it's now become the 'development' pipeline.
Yeah, the faster you’re able to iterate, the faster you can improve.
Only if you make meaningful iterations. I see a lot of demand for iteration coming from above and suddenly you are in this cycle of fixed length “sprints” where you worry more about releasing on time instead of meaningful releases.
One could argue that it still doesn't exist today at most companies. I think whatever most people call DevOps is an evolution of Sys Admin role in the era of Cloud Computing.
The goal of DevOps is to get code as developed to testing, staging, production. The deeper objective should be transfer of context- what is being deployed? do we know how? does it work as tested? and the who. As much DRY to reduce surprises and increase precision all through dev to deploy cycle.
Some of the best open source projects have CI included- thats the right approach to DevOps -- its not an alien, bolt-on after the fact practice, its code that takes care of code.
Some of the best open source projects have CI included- thats the right approach to DevOps -- its not an alien, bolt-on after the fact practice, its code that takes care of code.
IMHO DevOps has been around for many years. Is has just evolved as tooling became better and eventually tooling to connect tooling became better. It became META.
When I started doing database work we developed and implemented. We learned to produce reports on what we'd implemented by querying system tables. We learned to use those reports to produce scripts that could recreate the database on the same or a different platform.
I miss Devops Borat.
https://twitter.com/devops_borat
https://twitter.com/devops_borat
DevOps didn't exist... fella, let me tell you something, when I started, color screens were not the norm and you kept your homemade double-sided 5 1/4" floppy disks ( thank-you paper hole-punch ) in a plastic flip-top container with the contents written on the label in black sharpie.
it wasn't his own thing and there wasn't the scale yet to have it was a proper position, but when you were moving connection strings in a config file and linking it in etc you were doing devops. when you were building a shell script to fetch pinned dependencies so that the environment was repeatable, you were doing devops. weekend you were automating the package build/install and putting these files in CVS, you were doing devops.
heck we had versioned openpkg scripts in 200something to quickly rotate servers during upgrades, that wasn't called with fancy names and wasn't a standalone role but the seed was there.
heck we had versioned openpkg scripts in 200something to quickly rotate servers during upgrades, that wasn't called with fancy names and wasn't a standalone role but the seed was there.
Well, I wouldn't exactly call Dev Ops a productive career path. Most senior engineers I know seem to think it's a surefire way to drive your career into a dead end.
Devops has been standard practice at every financial firm i ever went to. Back then it meant devs where a pager. Ops also wore a pager and would support us.
> “a developer with operational tendencies”
Hmmm, I like the ring to that, might put it on my linkedin profile.
On the article itself, can't say I disagree.
Hmmm, I like the ring to that, might put it on my linkedin profile.
On the article itself, can't say I disagree.
I like to comment DevOps from a security perspective, a trend I noticed in my day job.
Windows is/was often bashed for being insecure. Lots of that stems from the decades of development related to centralized management solutions. A default windows workstation in a domain setting will open a bunch of ports, a bunch of which can be used for command execution. The attack surface for this system includes, but is not limited to:
- Remote access with local admin users via tools such as SMBExec, wmiExec, DCOM, Psexec, Powershell remoting
- Remote access domain admin users access via the same
- Local/domain admin access via RDP
- Remote domain admin access via group policy
All these have had their own associated vulnerabilities over the years. Examples are SMB relay attacks, which enabled an attacker to abuse flaws in NetNTLM and obtain access to machines by relaying other people's credentials. And then we're not even talking about the 'real' exploits, Eternal Blue, Eternal Romance, Blue Keep, MS14-068, MS08-067, and on and on.
Pentesters, researchers and Microsoft have been hammering away the kinks for years now. The 'fixes' and root causes for each individual issue are well understood and each new domain functional level increases the security of a default windows Domain by leaps and bounds.
When you look at the Unix/Linux side you'd see that no such attack surface ever existed. You manage your systems over SSH, and this can still be bad, an easily guessable root password shared between Dev, testing and production is still a death sentence. But by default there were no tier0 systems in your network, apart from those of sysadmins.
But now with DevOps things are changing on that side. With Ansible, Puppet, Terraform, your various container management systems, the CI pipeline, jenkins and numerous development teams able to push both to infra repositories and your actual products this has changed:
You use an automated CI pipeline? Any system in the chain is a tier0 system.
Your developers are maintainer status or higher in your source repositories? Then they are domain admin or equivalent. They can disable protected branches, push a backdoor, and watch their attack propagate through the pipeline.
Did you make it inconvenient for your developers to access various build systems? Then they are sharing credentials to these systems over your company chat.
It seems, from what I've seen so far, that while the 'architecture' of modern mass centralized IT management and development is more secure. You can't relay an SSH key for example, like you can in NetNTLM. But the institutional knowledge isn't there yet. New attack surface has opened up, and infosec people have not yet completely caught up with the new 'eggs' in the basket, even if they are aware.
Windows is/was often bashed for being insecure. Lots of that stems from the decades of development related to centralized management solutions. A default windows workstation in a domain setting will open a bunch of ports, a bunch of which can be used for command execution. The attack surface for this system includes, but is not limited to:
- Remote access with local admin users via tools such as SMBExec, wmiExec, DCOM, Psexec, Powershell remoting
- Remote access domain admin users access via the same
- Local/domain admin access via RDP
- Remote domain admin access via group policy
All these have had their own associated vulnerabilities over the years. Examples are SMB relay attacks, which enabled an attacker to abuse flaws in NetNTLM and obtain access to machines by relaying other people's credentials. And then we're not even talking about the 'real' exploits, Eternal Blue, Eternal Romance, Blue Keep, MS14-068, MS08-067, and on and on.
Pentesters, researchers and Microsoft have been hammering away the kinks for years now. The 'fixes' and root causes for each individual issue are well understood and each new domain functional level increases the security of a default windows Domain by leaps and bounds.
When you look at the Unix/Linux side you'd see that no such attack surface ever existed. You manage your systems over SSH, and this can still be bad, an easily guessable root password shared between Dev, testing and production is still a death sentence. But by default there were no tier0 systems in your network, apart from those of sysadmins.
But now with DevOps things are changing on that side. With Ansible, Puppet, Terraform, your various container management systems, the CI pipeline, jenkins and numerous development teams able to push both to infra repositories and your actual products this has changed:
You use an automated CI pipeline? Any system in the chain is a tier0 system.
Your developers are maintainer status or higher in your source repositories? Then they are domain admin or equivalent. They can disable protected branches, push a backdoor, and watch their attack propagate through the pipeline.
Did you make it inconvenient for your developers to access various build systems? Then they are sharing credentials to these systems over your company chat.
It seems, from what I've seen so far, that while the 'architecture' of modern mass centralized IT management and development is more secure. You can't relay an SSH key for example, like you can in NetNTLM. But the institutional knowledge isn't there yet. New attack surface has opened up, and infosec people have not yet completely caught up with the new 'eggs' in the basket, even if they are aware.
Exactly!
I've talked that this idea of "devops" is terrifying. It turns a whole bunch of devs whose security mindsets are in "eh password1! is great" because of convenience. And it turns them into domain admins. Gates are good... Although that means in having "Dev" and "Sysad" as 2 different groups. And I as a sysad can and will actively say "NO" with a reason, because something didn't pass a sniff test.
And with the whole AWS push on everything, do you know what else it does? It gives the devs an unlimited line of credit that company has to pay for. Nowhere before did sysads, devs, or other non-manager and non-C-level have unfettered access to the company wallet. And right now, if you have IAM access in AWS, you do.
I've talked that this idea of "devops" is terrifying. It turns a whole bunch of devs whose security mindsets are in "eh password1! is great" because of convenience. And it turns them into domain admins. Gates are good... Although that means in having "Dev" and "Sysad" as 2 different groups. And I as a sysad can and will actively say "NO" with a reason, because something didn't pass a sniff test.
And with the whole AWS push on everything, do you know what else it does? It gives the devs an unlimited line of credit that company has to pay for. Nowhere before did sysads, devs, or other non-manager and non-C-level have unfettered access to the company wallet. And right now, if you have IAM access in AWS, you do.
I thought that DevOps means: "doing operations with a development mindset."
That captures the gist of it imho.
That captures the gist of it imho.
It didn't exist when I started either, but talk about a let-down of an article conclusion!
DevOps is pretty much not a technology but a mindset
[deleted]
It's a critical skill now
sigzero(1)
All that’s really happened is that developers have learned or taken on the responsibility to build and deploy to production, and also be knowledgeable and responsible for the development stack, hence “full stack developer”.
Unfortunately, it’s incredibly dangerous if everyone involved doesn’t know what they’re doing.
Then kubernites and docker turn up. To make things easy?
No. They’re making the infrastructure more difficult to manage so that the developers can deploy things easier.
Developers should not be deploying. Large businesses should not be doing very regular deploys to live - it’s simply too risky without a shitload of testing.
Everyone praises startups with a few million customers that are open about their fuckups/downtime, but they are almost always because of their shitty modern deployments and lack of testing.
Crikey, look at Monzo. “We use a shit database and fucked up scaling it, but sorry your transactions failed and you looked like a twat buying your coffee”.
Try and revolutionise industries, fine. But ....
Unfortunately, it’s incredibly dangerous if everyone involved doesn’t know what they’re doing.
Then kubernites and docker turn up. To make things easy?
No. They’re making the infrastructure more difficult to manage so that the developers can deploy things easier.
Developers should not be deploying. Large businesses should not be doing very regular deploys to live - it’s simply too risky without a shitload of testing.
Everyone praises startups with a few million customers that are open about their fuckups/downtime, but they are almost always because of their shitty modern deployments and lack of testing.
Crikey, look at Monzo. “We use a shit database and fucked up scaling it, but sorry your transactions failed and you looked like a twat buying your coffee”.
Try and revolutionise industries, fine. But ....
[deleted]
If you want a database that is arbitrarily horizontally scalable, there aren't really good open source options.
As a former employee it isn't clear to me that Monzo couldn't have used boring stuff for a while and switched to C* or some equivalent later when it was genuinely needed, but it's worked well enough for them that I'm not inclined to critique the decision post hoc.
As a former employee it isn't clear to me that Monzo couldn't have used boring stuff for a while and switched to C* or some equivalent later when it was genuinely needed, but it's worked well enough for them that I'm not inclined to critique the decision post hoc.
Their last outage worries me greatly.
I think there's a clear critique of Cassandra in there, in that it shouldn't even be possible to configure nodes to serve data they don't have.
Monzo... It's very hard to say. I think based on the contents of the blog post they acted reasonably and competently (yeah, the test rollout was a badly designed test, but that's easy to say post hoc). The only fair basis for critiquing Monzo would be if they had notably more averages than is 'typical', and we don't have the data or conceptual framework required to do that. As a customer, I can't think of risks that Monzo exposes you to that aren't also present with other banks. I bank with them ¯\ _(ツ)_/¯
Monzo... It's very hard to say. I think based on the contents of the blog post they acted reasonably and competently (yeah, the test rollout was a badly designed test, but that's easy to say post hoc). The only fair basis for critiquing Monzo would be if they had notably more averages than is 'typical', and we don't have the data or conceptual framework required to do that. As a customer, I can't think of risks that Monzo exposes you to that aren't also present with other banks. I bank with them ¯\ _(ツ)_/¯
When I mention "systems administration" to younger coworkers, they look at me like I'm a crusty old relic out of touch with modern engineering. People have no sense of history and little understanding of where the devops role came from and what it really means. I had a CTO tell me he hadn't heard the term "system administration" in ten years.
What were our devops people doing? Running web servers, managing networks, configuring DNS, managing backups, configuring cloud services. Absolutely none of it in support of the dev team I was on. The developers were responsible for managing their own CI/CD pipeline and deployments. The people called the "DevOps Team" were responsible for managing production. In this case, they were systems administrators and were not in any way a devops team, but the terminology is so skewed now that they were labeled as such, and I was labeled an out of touch old greybeard.
This isn't to demean devops in any way; it's a valuable role, an evolutionary step in software team organization, but by no means is it new and by no means is systems administration a dead role.