My teammates create e-mail filters to send useless daily e-mail reports to the trash. I try to find out who controls the e-mail process, or the e-mail address, or something, so I can fix it... but I spend lots of time and it ends in vain. I couldn't find who "owns" the process, I couldn't find who controls the mailing list, and I couldn't get anyone to give me permission to change it even if I knew how.
Clearly the problem isn't just having the skill or permission to change something, it's also the friction involved in figuring out how the hell to do it. How do you lower friction? Documenting things, making it easy to find things, making it easy to get access to things. If you can come up with an internal system that combines all of that, you have a one-stop shop for fixing high-friction problems.
I think Wikis are highly underrated. They seem to encapsulate all those things. Anyone can edit (or revert edits), anyone can access it, anyone can find it (eventually). Somehow we need to tie all the rest of an organization into a Wiki.
So then shouldn't we stop writing software? If it's really impossible to make software that doesn't have tons of bugs, yet it's perfectly possible to make hardware without those bugs, shouldn't we be "making hardware" instead?
Actually, now that I think of it, that's not the problem. The problem is we keep changing the software. My laptop from 15 years ago still functions exactly the same way it used to. It hasn't disintegrated into a puddle of bits. You just can't use it to visit any "modern website" or run any "modern software". If we just stopped upgrading everything every 5 seconds we could keep using old technology.
It can work this way (that's how software patches have historically worked) but if you don't test it from the beginning, you will still find the odd case where that added step is broken, even though it seemed like it should have worked. The more you use that method, the more chances for breakage.
If you don't want to repeat the steps from the beginning, you could make a completely separate checklist to be followed on a given system that includes things like "make sure X package is installed", "make sure Y configuration is applied", so that the new checklist accounts for any inconsistencies. This is pretty common anyway as checklists are broken up into discrete purposes and mixed and matched.
That's basically what the do-nothing script is. The difference is that before you write any automation, you document all the steps in the script. Right there - when you've got it all written down, and no automation work has been done yet - that in itself is a very valuable piece of work. Now you can point anyone in the company to that script, and they can all accomplish the task without having to figure it out for themselves. You can now scale that process N times (N = the number of people in your company). Just writing down the steps has become a force-multiplier of repeatable work. Then as you begin automating each step, people automatically receive the benefit of that automation. Because the documented steps and the automation are in the exact same place, both will always be up-to-date.
What about the do-nothing script do you find is different than a checklist? To me the whole thing is already a checklist, there just aren't check-boxes.
> If you only install a webserver once in a blue moon, make a .txt checklist of the steps you followed.
This brings up a very important point about checklists that I don't think gets enough attention.
The problem happens when somebody "updates" that web server in-place. If they try to record what changes they made in the middle of the checklist, eventually when someone tries the whole checklist from the beginning, they'll find it's now broken; the steps aren't working as expected. This happens to me when I try to record changes in my VirtualBox configuration after I add a new system package or something; later I try to re-deploy my vbox, and it breaks.
So checklists should be considered immutable. Once you create them, don't assume they will work again if modified. Instead, if you make any change to the checklist, you must follow all the steps from beginning to end. This way you catch the unexpected problems and confirm the checklist still works for the next person.
In construction, if the foreman / lead whoever is always angry, people get fearful of speaking up about something, and then more mistakes get made because nobody wanted to point out the glaring flaw.
Soldiers also prefer trustworthiness over skill competency. Of course you want your brother-in-arms to do their job well, but it's more important that you can trust them with your life.
Technical skills are needed to work with a machine. People skills are needed to work with people.
You could call it an "E-I Script" for Efficiency Interest Script. Over time your costs are gradually lowered as each step is automated - like accruing interest in a savings account.
This is the way. I wish this were taught in computer science class, development bootcamps, operations team onboarding, anywhere there is a procedure that is even slightly complicated to automate. It is the absolute best solution there is.
* Documentation of the entire procedure is contained in one place. No need to go sifting through 20 different sources of documentation. This lowers the human emotional barrier to "just get it done", as people will always avoid things they aren't comfortable/familiar with, or don't have all the steps to. This central point of documentation also enables rapidly improving the process by letting people see all the steps in one place, which makes it easier to fix/collapse/remove steps.
* Automation in small pieces over time avoids the trap of "a project" where one or more engineers have to be dedicated to this one task for a long period of time. Most things shouldn't be automated unless there is demonstrably greater value in the cost of automating them than the cost of not doing so. Automating only the most valuable/costly pieces first gives immediate gains without sinking too much into the entire thing.
* One unified "method" to encapsulate any kind of process means your organization can ramp up on processes easier, reducing overall organizational cost.
* In the absence of any other similar process, you are guaranteed to save time and money.
I would say that the only potential downside is if someone decides to "engineer" this method, making it more and more and more complicated, until it loses its value. KISS is a requirement for it to be sustainable.
How is it not a fair comparison? They're machines. Just because we are currently building them in a way that is incredibly fragile and needs constant fixes, does not mean they have to be built that way.
Cars used to be built by hand, had tons of bugs, and were expensive. Then a man came along and found a way to produce them faster, cheaper, and with less bugs. That was pretty amazing for a time, but they still had plenty of bugs. And then some people from a culture of very fastidious craftsmen obsessed with quality began producing cars a little cheaper, and with far fewer bugs, and they lasted much longer. Then the whole world realized, "shit, our machines don't actually need to be so fragile," and they followed suit.
The lessons learned by those people in that culture were promoted around the world, and evolved to shape what we now call Lean and Agile. But the people using these new processes forgot the first lesson: we don't have to accept the status quo.
These kind of organizational problems happen everywhere, that doesn't bug me. What bugs me is when leadership knows about it and doesn't care. After low-level engineers stick their professional neck out to complain in internal town halls and through feedback forms, and leadership gives some bullshit answer that doesn't address or even acknowledge the problem. It would be less infuriating if they just said "I don't give a shit." It's the weasel words and pretending the problem doesn't exist that infuriates me. A lot of the time it doesn't even take much work at all to begin addressing the issue, like a working group for continuous improvement of highly-painful high-value processes. You don't even have to solve it. Just attempt to address it.
Why do your machines need monthly updates? Do you constantly update any other machine that you own? Lawnmower, car, oven, microwave, bicycle, watch, reciprocating saw, vaccum, garage door, TV?
Is Apple still the only company you can pay for both hardware and software support? Because all I want is to drop down a couple grand and never have to think about "computer maintenance" again. I maintain my car myself because it's so infrequent (pretty much just oil changes) but it feels like my computer maintenance is constant.
One of the reasons for that constant maintenance seems to be The Web. Remember when you didn't need 4 gigs of ram to browse the web? When you didn't need a high-power 3D graphics card to look at Google Maps? (bad example but WebGL is mandatory for some simple sites, and if your graphics sucks/doesn't do hardware acceleration...)
I don't remember ever having to upgrade my car every few years just to visit a new local business. At some point we need to admit that this constant tech churn isn't improving our lives, but it is enriching some billionaires.
"MangoDB is a proxy which uses PostgreSQL as a backend. The proxy translates MongoDB wire protocol commands into SQL queries, and use PostgreSQL as storage."
You don't have to support MongoDB, but you can support apps that were only written with Mongo as backend? That's awesome. I can't imagine it's production-ready yet but it's a great idea.
It's not a philosophical difference, it's just complexity. More complex systems are more prone to failure. If the security system is more complex to set up, it's more likely to fail. More code means more bugs, and more domain-specific knowledge leads to more potential for user error. So if you have 'one program to secure it all', it's almost guaranteed to be better than having to use many programs all in the right way. And it isn't even a defense-in-depth issue because all those layers added to container security are really just to avoid the much larger attack surface; getting rid of attack surfaces reduces what you need to defend.
Trying to secure a container via non-VM means is a painful slog. You can pretend containers give you security, and then one of the hundreds of different attack vectors provides a breakout. It's been demonstrated time and again, largely because Linux security is just shit and always has been.
I don't think there's ever been a year without a half dozen privesc holes in the Linux kernel. Linus is also belligerently anti-security because he thinks it always results in worse user outcomes. And containers were never created with security as a top priority, they're just an amalgamation of resource abstractions, so of course it works as well as anything else not designed with security in mind.
The hypervisor isolates guest kernel bugs from the host by nature of strictly controlling resource use from the lowest level. There are of course hypervisor bugs that allow breakouts, but they are a couple orders of magnitude rarer than the typical Linux privesc bug.
It's important to understand that containers are not a security device. Containers are a mechanism to separate resources used by processes. You should not assume any significant security benefits to containers, regardless of what anyone claims (even a kernel developer - maybe especially them....) because it all depends on Linux kernel security, which is pretty crap.
If you want security with containers, use Firecracker. It uses Micro VMs rather than just kernel-level restrictions, so even a Linux kernel security bug shouldn't be able to jump out to the host or other containers/Firecrackers.
Clearly the problem isn't just having the skill or permission to change something, it's also the friction involved in figuring out how the hell to do it. How do you lower friction? Documenting things, making it easy to find things, making it easy to get access to things. If you can come up with an internal system that combines all of that, you have a one-stop shop for fixing high-friction problems.
I think Wikis are highly underrated. They seem to encapsulate all those things. Anyone can edit (or revert edits), anyone can access it, anyone can find it (eventually). Somehow we need to tie all the rest of an organization into a Wiki.