Awesome! Kudos for owning your own data and infra!
I don't think systems administration is a lost art, people just wanted to do things easier for less money and staff, and no one can blame them or the industry as a whole. The art is still right there, linux hasn't changed that much in 20+ years, I still have my first discs. It also takes some experience to "feel" your way through systems administration problems, working from the bottom up of course, feeling like its a network problem still has a high rate of success when troubleshooting for me!
I try everyday to teach some of these skills to people I work with, I just call it devops, or SRE, or some cyber cloud support position someone makes up, they are all systems admins/engineers to me still. I enjoy watching people learn and apply that knowledge to future problems, getting to the root cause or close to it, and the satisfaction that comes from fixing the issue from start to finish on your own, looking things up is not cheating in systems administration!
I would say they don't mostly suck, everyone of them who I have worked with has opened up doors that wouldn't be open otherwise. Made my salary go way up too!
So far ive dealt with bullies all my career at every company and i observe then challenge, then challenge again, im 2/3 getting them to quit...no need for hr or threats just be smart and trap em they never know who ;)
facebook had me as someone who could fill their quota for 1st rounds etc I sent an email asking them not to do this twice now. It is quite funny though, because every time I bring up NYC being the only possible location for me, the recruiter usually grumbles says its "tough to get into NYC" and then usually I get that the engineering manager "doesn't like that many people" I wonder how many people have been rejected working for Facebook because one person in NY doesn't like them? I mean that person might be amazing, but still seems limiting. They do fine anyway though, most of the hard parts have already been done long ago at Facebook I would suppose.....
1) Carrying an entire 75ft tree in pieces from my backyard to my front yard.
2) 10 hours maybe
3)wood needed to be moved
4) when looking at the pile of wood I have to move
5) the resulting stack of wood in front of my house
6)yup, been stronger ever since those few days a few months ago
FAANG's often don't have paying customers, banks do.
Its not being stuck between a rock and a hard place, you just need strong management that understands that changing whole systems is never a good idea without clear benefit, other than "technical debt" which is such a bad term for reliable software. Software doesn't age, im not sure how software engineers don't understand this, or maybe they do and want to write more s/w.
so right, but it won't make you any money or get you any linkedin/twitter fans if you speak the truth. We have AI being tossed around like it isn't just database queries most of the time.
When someone says "ex googler" usually during the beginning of a call,I immediately multi-task, let them finish with their chest pounding and move on from them, trying to avoid them going forward. If you can't convince people with the technical merits of your plan, I guess you just try and name drop (company drop). Some people buy it, most don't and they sit in some enterprise architect role with maybe a few internal fans. If you want to waste tons of money, scale your company for billions of users when you have 2000, hell don't even scale just say you can and claim success before you even do it, it works there are tons of these guys and gals ripping apart companies for their perfect "designs" that worked at google.
(i never worked at google but did interview, argued with the interviewer and hung up, he thought computers worked differently than they do in reality and just spoke with this heir of authority, without having a spec of it )
You just didn't have anyone skilled enough to automate those things, or you couldn't manage that. If you are colo you aren't swapping disks/network your colo provider is. Im not advocating for doing it ALL yourself, making network cables does save money, but its impractical. Just because you couldn't manage the whole thing while having the ear of the CTO, just means you failed it doesn't mean it doesn't work for lots of other companies that make lots of money.
Can you please tell us why? What are you afraid of? I have seen servers up much longer than 2 days, still humming along no memory leaks no whatever you can think of that "rots" on a server that doesn't change. Maybe your app has a lot of memory leaks or is not well coded where it has to be restarted every few days, or as you say completely brought up on a new server why?
Servers go down in AWS too, they are a colo shop with an open source services arm to implement multi tenant services of which you are one tenant yet paying for that multi tenancy service.
You may not notice it having 1 but have a few thousand. Everything goes down in AWS just like it does on normal servers, no one likes to mention that though so often its blamed on something else like software that is running on the cloud that just went down hah.
My last company, went the dedicated route, had management that understood how much of a ripoff AWS/Azure/GCP was and hired people who know how a lot of apps are built and run, and didn't need to redevelop them to fit the architecture which costs a metric ton more with little benefit. Press releases are almost always complete BS from companies when it comes to tech, again not all but most, we saved x using some dubious calculation.
AWS makes some things easy sure, but at my house I have what would cost $15k a year and I spent ~1500 all-in including 10G switch(used) power use is a joke, even with expensive electricity where i am AWS still a ripoff. Same processors as AWS essentially as he was quoting, 10G to my home nodes that need it and bobs your uncle. And I do tensorflow things at home too, i bet that would cost 30k/year, i do it for $1500 BOUGHT and for another $500 for a 3080 or a few in my servers and id be saving tons. Yeah it can go down, but just buy two or a rack, its going to be cheaper than AWS hosted colo. I could scale this myself to a few thousand servers, ive seen it done wrong so many times.....
The truth is more like he mentioned in hist first paragraph, if you have a fan boy at the helm it doesn't matter if he's CTO he knows it all, Ive worked with a lot of CTO's lately they don't have a clue and I feel sorry for them, wasting so many company assets because underlings couldn't possibly know more, i love the x google/facebook guys saying basically scrap everything re-architect its funny and so wasteful but at least they are making more money than me!
One guy said he hates Jenkins, some teams have hundreds of jobs in jenkins that work just fine, he said it should all be redeveloped, some of the stuff has been working fine for 10+ years, not sure he has company interest in mind, he want' everything to be "serverless" its the same damn thing effectively genius.
Hire some sysadmins to run your show, the good ones cost a ton, and computers at their core haven't really changed much in 20+ years nor has the fundamental way the internet works. Good experienced sysadmins can save you a ton and its just as reliable. You just have to trust experience over advertising and real math over funny math.
I don't think systems administration is a lost art, people just wanted to do things easier for less money and staff, and no one can blame them or the industry as a whole. The art is still right there, linux hasn't changed that much in 20+ years, I still have my first discs. It also takes some experience to "feel" your way through systems administration problems, working from the bottom up of course, feeling like its a network problem still has a high rate of success when troubleshooting for me!
I try everyday to teach some of these skills to people I work with, I just call it devops, or SRE, or some cyber cloud support position someone makes up, they are all systems admins/engineers to me still. I enjoy watching people learn and apply that knowledge to future problems, getting to the root cause or close to it, and the satisfaction that comes from fixing the issue from start to finish on your own, looking things up is not cheating in systems administration!