DevOops, some common anti-patterns(hackernoon.com)
hackernoon.com
DevOops, some common anti-patterns
https://hackernoon.com/devoops-some-common-anti-patterns-1850ac2f5074
32 comments
"Small organizations don't require dedicated ops teams anymore"
So who's gonna deploy the developers' applications and implement the necessary infrastructure for that application?
Who's gonna create, maintain, and administer the CI/CD pipelines which allow us to do testing, receive feedback loops, and rollback when code does not work?
Who's gonna be the guy, or team of people, who has knowledge of all the different services and architecture of all these IAAS? And then who's gonna know how to implement that?
So who's gonna deploy the developers' applications and implement the necessary infrastructure for that application?
Who's gonna create, maintain, and administer the CI/CD pipelines which allow us to do testing, receive feedback loops, and rollback when code does not work?
Who's gonna be the guy, or team of people, who has knowledge of all the different services and architecture of all these IAAS? And then who's gonna know how to implement that?
Whoever steps up in a team to do it. I've seen all of that built in small development teams and I've seen big companies making a mess of this. It's all about hiring the right people.
Except that all those IAAS solutions must be setup / manage by ops / devops, regular dev have no clue about architecture, infrastructure as code, security, backup, alerting, best practices ect ...
I have yet to see a dev with good ops skills. DevOps are not dev that moved into ops world it's quite the opposite. It's Ops people that moved into dev teams.
The other thing is also that dev usually don't want to do that, and when you force a dev to do ops things it gets very dangerous.
I have yet to see a dev with good ops skills. DevOps are not dev that moved into ops world it's quite the opposite. It's Ops people that moved into dev teams.
The other thing is also that dev usually don't want to do that, and when you force a dev to do ops things it gets very dangerous.
Out of curiosity, any tips for companies small enough to not warrant an ops position, but also so "lean" (read: broke) that we can't too heavily rely on cloud solutions?
Ie, I advocate self hosting a lot in my ~8dev team (in a real, 10 year old small company). I advocate this, because the last company I was at was the traditional startup and we blew tons of money on cloud solutions. Ie, if we needed to aggregate logs, we threw everything at papertrail, rather than managing it ourselves. This has given me a cost aversion, so I theoretically prefer self hosting when sane. Mostly for everything but actual VMs.
With that said, a lot of self hosting leads to a lot of infra. I agree that hiring devs who also know ops is a good way to go (and would fit us well), but in this scenario, do you still think it's the better/best?
edit: to be clear, I advocate using AWS/etc. I don't advocate using dozens of SaaS options to solve our modest needs at non-modest prices. Many SaaS solutions were designed for scales much larger than ours, self hosting seems most apt.
Ie, I advocate self hosting a lot in my ~8dev team (in a real, 10 year old small company). I advocate this, because the last company I was at was the traditional startup and we blew tons of money on cloud solutions. Ie, if we needed to aggregate logs, we threw everything at papertrail, rather than managing it ourselves. This has given me a cost aversion, so I theoretically prefer self hosting when sane. Mostly for everything but actual VMs.
With that said, a lot of self hosting leads to a lot of infra. I agree that hiring devs who also know ops is a good way to go (and would fit us well), but in this scenario, do you still think it's the better/best?
edit: to be clear, I advocate using AWS/etc. I don't advocate using dozens of SaaS options to solve our modest needs at non-modest prices. Many SaaS solutions were designed for scales much larger than ours, self hosting seems most apt.
Man, so many red-flags in what you posted.
So, full-disclosure: I come from Ops. These days I'm more an orchestrator/developer than a hardware/bios/cable monkey though. For context my last deploy was 2,500~ physical machines. So I'm not the kind of ops that the parent is talking about.
Now, if you don't have the money to do cloud, and you don't have the headcount/experience to do self-hosted properly; then you're _done_. You will accrue technical debt at an exponential rate. At my last job Ops on the production environment was done exclusively by developers.
When I arrived it was an absolute shit-show nightmare. Silent data corruption, weird dependencies on services which were not highly available, hand-built machines, next to "cattle" but which had multiple deployment scenarios (capistrano, chef, bash, PHP kludge) none of which were documented.
But the worst thing was definitely the silent data corruption.
I agree in part with the parent, Ops is a dirty word. In my current company there's a separate "Ops" entity which is a true horror show exactly as he describes. However having people trained on ops in your team is -absolutely essential- to doing devops correctly. Developers and Operations think differently. It's a gross generalisation and there are exceptions but by-in-large developers are focused on "how can I make this work" (even robustly), where operations are focused on "how can this break and what are the implications if it does" and those methodologies are intrinsically opposite to each other. A defensive programmer is a great thing but they are not only rare they are not a panacea.
--
Pick a side; do cloud or hire an ops.
Cloud is basically "ops as a service", giving you all the tools you need to get by efficiently without classically trained sysadmins. I really promise that you will not get very far without someone handling this side of things for you, whether it be the SRE's at google/AWS or someone you hire.
So, full-disclosure: I come from Ops. These days I'm more an orchestrator/developer than a hardware/bios/cable monkey though. For context my last deploy was 2,500~ physical machines. So I'm not the kind of ops that the parent is talking about.
Now, if you don't have the money to do cloud, and you don't have the headcount/experience to do self-hosted properly; then you're _done_. You will accrue technical debt at an exponential rate. At my last job Ops on the production environment was done exclusively by developers.
When I arrived it was an absolute shit-show nightmare. Silent data corruption, weird dependencies on services which were not highly available, hand-built machines, next to "cattle" but which had multiple deployment scenarios (capistrano, chef, bash, PHP kludge) none of which were documented.
But the worst thing was definitely the silent data corruption.
I agree in part with the parent, Ops is a dirty word. In my current company there's a separate "Ops" entity which is a true horror show exactly as he describes. However having people trained on ops in your team is -absolutely essential- to doing devops correctly. Developers and Operations think differently. It's a gross generalisation and there are exceptions but by-in-large developers are focused on "how can I make this work" (even robustly), where operations are focused on "how can this break and what are the implications if it does" and those methodologies are intrinsically opposite to each other. A defensive programmer is a great thing but they are not only rare they are not a panacea.
--
Pick a side; do cloud or hire an ops.
Cloud is basically "ops as a service", giving you all the tools you need to get by efficiently without classically trained sysadmins. I really promise that you will not get very far without someone handling this side of things for you, whether it be the SRE's at google/AWS or someone you hire.
I added an edit, do you still feel this comment is accurate after my edit? Ie, I'm not advocating physical machines. The cost + time ratios don't add up imo. Using AWS/etc is simply better, I believe.
I'm talking about not using dozens of SaaS options for small tooling, think CI, log agg, metrics, git, etc. Much of these are trivial to self host when your traffic is small and predictable. Thoughts?
I'm talking about not using dozens of SaaS options for small tooling, think CI, log agg, metrics, git, etc. Much of these are trivial to self host when your traffic is small and predictable. Thoughts?
(Not the OP) If cloud equates in your mind to expensive, then you were probably using the cloud wrong. Using AWS (for example) as a data-center and self hosting on EC2 can be pretty expensive compared to just getting a few dedicated boxes. The point of cloud architecture with tools like terraform are meant to be able to have resources be torn down and scaled down when they are unnecessary. You can't scale down bare metal, and you can't scale it up either automatically. If you have a consistent workload and know exactly what you need on any given day/time and for any given number of clients and can predict what you need - bare metal is honestly not a bad choice.
Edit: Just realized I didn't answer your initial question in the post: A cloud architect, who comes from an engineering background, is more along the lines of what you would need versus just a pure ops person. My last company was a small company (startup mentality) and as chief architect I was able to design our cloud architecture as well as write code on our projects when my developers were struggling to meet deadlines.
Edit: Just realized I didn't answer your initial question in the post: A cloud architect, who comes from an engineering background, is more along the lines of what you would need versus just a pure ops person. My last company was a small company (startup mentality) and as chief architect I was able to design our cloud architecture as well as write code on our projects when my developers were struggling to meet deadlines.
I think there's a miscommunication here, but I probably should have said SaaS rather than "cloud".
Ie, I don't advocate self managing hardware. That's a difficult problem, one more cheaply and easily solved by using AWS, GC, etc. Yet, there are dozens of SaaS options for so many little problem you could have, and they can bleed you dry. A balance is needed.
I mentioned in my original post that I advocated self hosting except for VMs. Ie, I promote usage of AWS/etc. I just don't think we can afford Papertrail, and the billion other SaaS pieces I used in my last company. Self hosting has been far more scalable and cheap, due to our modest needs.
Ie, I don't advocate self managing hardware. That's a difficult problem, one more cheaply and easily solved by using AWS, GC, etc. Yet, there are dozens of SaaS options for so many little problem you could have, and they can bleed you dry. A balance is needed.
I mentioned in my original post that I advocated self hosting except for VMs. Ie, I promote usage of AWS/etc. I just don't think we can afford Papertrail, and the billion other SaaS pieces I used in my last company. Self hosting has been far more scalable and cheap, due to our modest needs.
In the most successful of the companies of the sort you've been describing that I've been around SaaS was kept pretty minimal. This was less a policy than a consequence of just having *nix skills or a desire to learn. Also many SaaSes have relatively serviceable self-hosted FOSS counterparts.
Log management in particular is really a shell script/self hosted app kind of task, just share the money saved on Papertrail with the team and I'm sure they'll get on board.
Log management in particular is really a shell script/self hosted app kind of task, just share the money saved on Papertrail with the team and I'm sure they'll get on board.
So what the 8 devs really need is a cloud architect?
The point about "you build it, you run it" not being sustainable really hits home for me. On a large project, there's a fairly natural specialization along many axes, with "deep in the guts" development vs. "close to the users" ops as one of the more natural ones. It's definitely important for developers to get some direct experience with operating what they produce. That's why I took my current job instead of letting myself become one of those awful "senior architect" types divorced from reality. Still, being on call is stressful. That stress is even greater when it's into areas others know 10x better, or when it interrupts progress toward the longer-term outcomes on which developers are measured even in a "devops" culture.
Changing teams to get a break from being on call starts to seem less disruptive than staying, and I'm sure plenty of other people where I work hit that realization long before me. I think more of a "pair programming" approach would often produce superior results. Two close associates don't need to know every command-line flag and semicolon of each others' work to understand and appreciate each other's contributions to the challenges they share.
Changing teams to get a break from being on call starts to seem less disruptive than staying, and I'm sure plenty of other people where I work hit that realization long before me. I think more of a "pair programming" approach would often produce superior results. Two close associates don't need to know every command-line flag and semicolon of each others' work to understand and appreciate each other's contributions to the challenges they share.
I'm not much of a fan of devops. I would really like to have an ops person who really knows what they are doing, and has that as their prime concern, rather than having it be just one more concern tacked onto everything else that developers already are responsible for. Specialization is dead, we just have a bunch of generalists muddling along. Right now I'm wearing more hats than a hat rack, between front-end, back-end, database, operations, support, security, project management, testing, and sorta kinda managing a few junior engineers. I can do one or two of those things well at a time, the rest I'm just bailing water or it goes by the wayside entirely.
DevOPs is not one person doing everything, it's removing the walls between developers and operations. Ideally you have a "devops team" consisting of devs and ops.
But as I always say: "Devops is not a job title"[0] only cheap hiring managers think it is.
[0]: https://www.infoworld.com/article/3263812/devops/do-not-hire...
But as I always say: "Devops is not a job title"[0] only cheap hiring managers think it is.
[0]: https://www.infoworld.com/article/3263812/devops/do-not-hire...
Also How about using kubernetes when all you are running is a few web applications?
Is it though? Hosted k8s clusters make deploying a single app just as simple as heroku or elasticbeanstalk, but you still have the flexibility to deploy more. For a small team it's just as easy to setup a managed k8s cluster as it is any other PaaS - you go though a wizard on GKE or some other provider. And if you apps are simple then you don't need to use the complicated features of k8s itself. 1 deployment and 1 LoadBalanced service is simple enough for most Ruby, Python, C#, Java monolith apps to be off to the races.
I'm not trying to argue that it's a silver bullet but there's a bit of circle jerk over k8s complexity when it's not that hard.
I'm not trying to argue that it's a silver bullet but there's a bit of circle jerk over k8s complexity when it's not that hard.
> Thinking, you build it, you run it, will work in the long term
This is basically the "hit-by-a-bus" scenario and it applies to pretty much anything in which you have a single point of failure in terms of manpower, not just in-house software. e.g. What happens when the engineer who installed your open-source solutions leaves? Who is left to support updates, migrations, etc? The solution to this is of course is to staff correctly...so that you don't have a single engineer building out an entire code base or system that your company heavily relies on.
This is basically the "hit-by-a-bus" scenario and it applies to pretty much anything in which you have a single point of failure in terms of manpower, not just in-house software. e.g. What happens when the engineer who installed your open-source solutions leaves? Who is left to support updates, migrations, etc? The solution to this is of course is to staff correctly...so that you don't have a single engineer building out an entire code base or system that your company heavily relies on.
In a way, after 3-4 years, there isn't much difference whether you have to update your self-hosted service, or the SaaS you're using is finally removing the functionality you rely on after having deprecated it 2 years earlier.
The only difference is server resource management, which is not much effort if automated correctly, and security fixes which is not that much more.
The only difference is server resource management, which is not much effort if automated correctly, and security fixes which is not that much more.
Strikes close to home as current employer try to shifts to better delivery patterns. I find https://web.devopstopologies.com/ talks a bit more about organizational mistakes.
Long ago companies had testers, but then they turned them into developers also, and at the same time everyone became an ops person too. What could go wrong?
Double-take moment. I thought this was related a machine named "DevOops" on HackTheBox :P
"DevOops," very cleaver. I nearly missed it and almost wrote it off as a typo :)
It's a very common term amongst security engineers. There are entire series of talks related to security and vulnerabilities that are introduced by DevOoops.
It's like some companies had testers, but then they turned them into developers also, and at the same time everyone becomes an ops person. What could go wrong?
It's like some companies had testers, but then they turned them into developers also, and at the same time everyone becomes an ops person. What could go wrong?
> 5 — Implement Ansible, Puppet, Chef, Kubernetes, OpenShift, Terraform…
Irrespective of personal feelings on these individual projects, they're all aimed at automating something?
I don't get the point of this item?
What's wrong with them?
Irrespective of personal feelings on these individual projects, they're all aimed at automating something?
I don't get the point of this item?
What's wrong with them?
I think sometimes they are implemented as a solution looking for a problem. I've seen many times when the automated solution is much more complicated than the manual solution. Sure, it's more reliable, except that it breaks when you change something.
The tools are good, but use them when you need them and when they help, not just because someone told you it was best practice.
The tools are good, but use them when you need them and when they help, not just because someone told you it was best practice.
It is extremely easy to overload on new and shiny. Sometimes a web app can be scaled with a minimum of tool work. If something can be automated that is great, but probably not everything needs to be automated right away. This is a special case of You Probably Aren't Going To Need It.
Putting Puppet on top of an environment that is poorly designed will not solve that poor design.
I implemented Puppet to manage a hugely complex application/environment. Because the architects of that app didn't think of Puppet, they threw up roadblocks at every opportunity. Things like refusing the allow the install package to be simply changed to RPM, or allowing a large part of install logic to be moved from shell scripts to Puppet. Or allowing an automated push from the release branch of the Puppet repository to any of the Puppet environments.
A lot of stuff that was initially planned that would have greatly simplified install, configuration, rollbacks, verification, etc etc, was all refused because it had been invented by the ops team instead of the architects.
So we ended up making things worse because management wanted Puppet, but didn't want to lay down the law to the architects to update their stuff.
I implemented Puppet to manage a hugely complex application/environment. Because the architects of that app didn't think of Puppet, they threw up roadblocks at every opportunity. Things like refusing the allow the install package to be simply changed to RPM, or allowing a large part of install logic to be moved from shell scripts to Puppet. Or allowing an automated push from the release branch of the Puppet repository to any of the Puppet environments.
A lot of stuff that was initially planned that would have greatly simplified install, configuration, rollbacks, verification, etc etc, was all refused because it had been invented by the ops team instead of the architects.
So we ended up making things worse because management wanted Puppet, but didn't want to lay down the law to the architects to update their stuff.
I think the point here is not to focus on tools for the sake of tools but instead focus on delivering value to the organization. Quite often, improving a workflow can be done more effectively by improving how you use your existing tools rather than by adding or replacing tools. Fear of missing out can drive us away from considering our existing workflow and how it can be optimized.
The "anti-pattern" wasn't about the tools themselves, but on focusing excessively on tooling at the expense of other aspects of DevOps.
[deleted]
is there an alternative URL where I can read this article? It won't load on my end. Thx in advance.
Here a mirror: http://archive.is/jjBhY
I witnessed AWS creeping into Nokia ten years ago. This did not go down well with internal ops initially. They did not realize they were part of the problem instead of the solution. They were a huge cost bottleneck and also a source of huge time delays and pointless bureaucracy.
Ultimately a lot of what they did stopped being a thing. Resources they controlled (e.g. firewalls between data centers, storage clusters, vms, etc.) were replaced by IAAS. This turned out to be magnitudes faster, cheaper, and more reliable (e.g. storage cluster outages were a regular thing).
Small organizations don't require dedicated ops teams anymore. If (dev)ops is done done well, ops is just not a full time job anymore. However, letting development teams do their own ops requires hiring people with ops skills as well as developer skills: i.e. devops people. So, not an anti pattern and it is perfectly valid to make sure that all teams have enough ops skills in the team. Also, these days I would expect senior developers to be not completely helpless doing ops. I don't expect them to like it but I do expect some rudimentary skills.
So,hiring, people with devops skills is not an anti-pattern. I'd actually argue having an ops team at all is an anti-pattern unless you are big enough to justify needing one. And if you don't have one, not hiring people with relevant skills would be a big anti pattern as well.
I've been around long enough to have seen most of what ops used to be about automated away or replaced by IAAS style deployments or otherwise no longer relevant. IMHO this is an ongoing thing and there is still a lot of non trivial stuff involved. A lot of time is still wasted on what is fundamentally tedious, repetitive, and error prone business of reinventing the same wheels over and over. It still seems way too hard to push out bog standard web applications in a way that results in responsible deployments with all the boxes ticked regarding security, ease of use, logging, auditing, etc. For me devops is a necessary evil. The less time I can spend on it the better. But the reality is that it still requires skills and time.