Ask HN: If I Close My Data Centers, What About the People/Jobs Lost?
149 comments
so, uh, I'm not weighing in on the job stuff... but I suggest you check your numbers on how hosting in the public cloud is cheaper than your own data centers.
I mean, sure, automating things is great, and running your own hardware cloud-style is great for a lot of workloads... but that doesn't have a lot to do with the decision to rent rather than own your servers.
Cloud providers charge a lot compared to owning your own hardware, unless your load is extremely bursty.
The bandwidth prices are particularly eregrious. They are like what you would pay for bandwidth 10+ years ago.
I personally ran a small VPS company and managed to turn a (small) profit while selling compute for rather less than amazon, even though I had to rent space and power from a fancy pants datacenter by the rack. I was essentially paying retail, and I only had 20-30Kw of capacity, max, and even at that scale, owning was cheaper than renting.
I can only imagine that at F50 scale, the economics tilt even further towards own, as you can much better amortize the labor than I could
(and it's not like the datacenter loses money, either. Renting datacenter space is like renting commercial realestate. When your lease is up, the renewal rate goes up to whatever competing space costs + what it costs you to move. Owning is so much better in that respect.)
I mean, sure, automating things is great, and running your own hardware cloud-style is great for a lot of workloads... but that doesn't have a lot to do with the decision to rent rather than own your servers.
Cloud providers charge a lot compared to owning your own hardware, unless your load is extremely bursty.
The bandwidth prices are particularly eregrious. They are like what you would pay for bandwidth 10+ years ago.
I personally ran a small VPS company and managed to turn a (small) profit while selling compute for rather less than amazon, even though I had to rent space and power from a fancy pants datacenter by the rack. I was essentially paying retail, and I only had 20-30Kw of capacity, max, and even at that scale, owning was cheaper than renting.
I can only imagine that at F50 scale, the economics tilt even further towards own, as you can much better amortize the labor than I could
(and it's not like the datacenter loses money, either. Renting datacenter space is like renting commercial realestate. When your lease is up, the renewal rate goes up to whatever competing space costs + what it costs you to move. Owning is so much better in that respect.)
Amazon etc negotiate prices for large accounts so the price you pay will not be the same price that a Netflix pays.
Sure, and it's possible OP really is saving money by renting their hardware instead of owning. I don't know the details of their needs or of their costs or what cloud provider they are moving to or what deal they cut with that provider.
I'm just saying, there's currently a huge push to rent your essential services from service providers such as amazon, and from what I've seen, it's pretty easy to "move to the cloud" and then find that your total cost goes up pretty dramatically vs. owning hardware.
I mean, yea, "cloud" does make sense sometimes. But... it is sold as being cheaper, and quite often it is not.
Even if the rates look good up front, you've got the same problem that you have when leasing commercial realestate from professionals; they are professionals, which means that when it comes time to renew the lease, the rate will go up to match what renting something similar would cost from a competitor, plus slightly less than whatever they think it will cost you to move. If you can't switch cloud providers and your cloud provider knows it, you can count on your negotiated rates going up next contract renewal.
I'm just saying, there's currently a huge push to rent your essential services from service providers such as amazon, and from what I've seen, it's pretty easy to "move to the cloud" and then find that your total cost goes up pretty dramatically vs. owning hardware.
I mean, yea, "cloud" does make sense sometimes. But... it is sold as being cheaper, and quite often it is not.
Even if the rates look good up front, you've got the same problem that you have when leasing commercial realestate from professionals; they are professionals, which means that when it comes time to renew the lease, the rate will go up to match what renting something similar would cost from a competitor, plus slightly less than whatever they think it will cost you to move. If you can't switch cloud providers and your cloud provider knows it, you can count on your negotiated rates going up next contract renewal.
> it's possible OP really is saving money by renting their hardware instead of owning.
Is it, though? Other than the special case of Netflix, it strains credibility that any cloud provider would be willing to give enough of a discount to match, let alone beat, ones own hardware. How would they make money?
This can be especially true if one "unfairly" compares a cloud provider's relatively expensive hardware choices to ones own cheaper choices. For example, AWS with dual 10GE motherboards and CPUs that weren't the best performance-per-dollar when they bought them, if all you need is single gigabit (and don't need particularly high single-core CPU speed).
> the rate will go up to match what renting something similar would cost from a competitor, plus slightly less than whatever they think it will cost you to move.
Fortunately, there's an upper bound, which is the public price, which has, at least historically, been coming down. Also, AWS has the spot instance market, which can provide downward price pressure from even that ceiling.
Is it, though? Other than the special case of Netflix, it strains credibility that any cloud provider would be willing to give enough of a discount to match, let alone beat, ones own hardware. How would they make money?
This can be especially true if one "unfairly" compares a cloud provider's relatively expensive hardware choices to ones own cheaper choices. For example, AWS with dual 10GE motherboards and CPUs that weren't the best performance-per-dollar when they bought them, if all you need is single gigabit (and don't need particularly high single-core CPU speed).
> the rate will go up to match what renting something similar would cost from a competitor, plus slightly less than whatever they think it will cost you to move.
Fortunately, there's an upper bound, which is the public price, which has, at least historically, been coming down. Also, AWS has the spot instance market, which can provide downward price pressure from even that ceiling.
>Is it, though? Other than the special case of Netflix, it strains credibility that any cloud provider would be willing to give enough of a discount to match, let alone beat, ones own hardware. How would they make money?
Right now, as I understand it, Uber is losing money driving me to work every day, and I'm not really sure how they will ever make a profit off of that. if they double their prices? I'm gonna buy a car, drive my own self to work, and use uber once a week when I go to the bar. I don't really see how buying fake market share (rather,market share that only exists when you are dumping) helps them when it comes time to not dump anymore.
Point being, there are plenty of people who are willing to sell two dollar bills for a dollar when the economy is hot. (I'm not suggesting amazon does this... but they certainly could. It's possible. )
Even in more rational markets, loss leaders are a big thing, and in leasing situations, where it's hard to move once you are in? loss leaders are a really common sales tactic. How often have you gotten "first month free" rent deals on one year leases, that work out to below market for the first year (because of the free first month) but above market after that, even if they don't actually raise the rent?
Datacenter leases often have a similar 'ramp up' period, where you pay less at the beginning of a contract than towards the end, even not counting the fact that they jack up your bill by however much they think it will cost you to leave.
I mean, these all can be really good deals if you know that it's easy for you to move later,and it's certainly possible (though maybe not easy) to setup your infrastructure so you can switch cloud providers with little cost.
>Fortunately, there's an upper bound, which is the public price
This is true...but that's a pretty high upper bound, when you compare to owning hardware/leasing pipe and power at all but the smallest scale.
>which has, at least historically, been coming down.
Historically, the cost of compute has come down, though slower than the cost of hardware (at least until lately) the cost of bandwidth has come down... much more slowly.
>Also, AWS has the spot instance market, which can provide downward price pressure from even that ceiling.
I think spot instances are a difference in kind (and actually a really great deal, if you have a workload that can be interrupted like that) - but most workloads I have worked with, especially in corp, don't really work that way.
Right now, as I understand it, Uber is losing money driving me to work every day, and I'm not really sure how they will ever make a profit off of that. if they double their prices? I'm gonna buy a car, drive my own self to work, and use uber once a week when I go to the bar. I don't really see how buying fake market share (rather,market share that only exists when you are dumping) helps them when it comes time to not dump anymore.
Point being, there are plenty of people who are willing to sell two dollar bills for a dollar when the economy is hot. (I'm not suggesting amazon does this... but they certainly could. It's possible. )
Even in more rational markets, loss leaders are a big thing, and in leasing situations, where it's hard to move once you are in? loss leaders are a really common sales tactic. How often have you gotten "first month free" rent deals on one year leases, that work out to below market for the first year (because of the free first month) but above market after that, even if they don't actually raise the rent?
Datacenter leases often have a similar 'ramp up' period, where you pay less at the beginning of a contract than towards the end, even not counting the fact that they jack up your bill by however much they think it will cost you to leave.
I mean, these all can be really good deals if you know that it's easy for you to move later,and it's certainly possible (though maybe not easy) to setup your infrastructure so you can switch cloud providers with little cost.
>Fortunately, there's an upper bound, which is the public price
This is true...but that's a pretty high upper bound, when you compare to owning hardware/leasing pipe and power at all but the smallest scale.
>which has, at least historically, been coming down.
Historically, the cost of compute has come down, though slower than the cost of hardware (at least until lately) the cost of bandwidth has come down... much more slowly.
>Also, AWS has the spot instance market, which can provide downward price pressure from even that ceiling.
I think spot instances are a difference in kind (and actually a really great deal, if you have a workload that can be interrupted like that) - but most workloads I have worked with, especially in corp, don't really work that way.
I think most of us here understand the concept of loss leaders and of sacrificing profits (even operating at a loss) for market share.
My point isn't that it doesn't happen, nor that it couldn't happen in this space. Rather, my point is, why would they bother (be willing)?
> there are plenty of people who are willing to sell two dollar bills for a dollar when the economy is hot. (I'm not suggesting amazon does this... but they certainly could. It's possible. )
In fact, we pretty much know, because Amazon has revealed it, that AWS is making money hand-over-fist, and they do this while having sizeable (existing) market share.
Would the other providers compete on (non-public) price just to gain (new) market share? It's possible, but I maintain it's an extraordinary proposition, considering Microsoft's overall software ecosystem integration advantage with Azure (much better lock-in) and Google's various attempts at features, including AI hardware.
Of course, this is all fairly academic, because..
> This is true...but that's a pretty high upper bound, when you compare to owning hardware/leasing pipe and power at all but the smallest scale.
The OP admitted that the cost savings is based on public pricing, so the decision makers don't actually care about the hugely inflated cost of cloud infrastructure! (Or, as in my personal experience, they refuse to believe it, which makes job hunting particularly surreal). That further supports my belief that the providers have no motivation to negotiate price.
> Historically, the cost of compute has come down, though slower than the cost of hardware (at least until lately)
(Did you mean the parenthetical to apply to this or to the statement about bandwidth?)
Is there anywhere that tracks this? I've only ever done that-point-in-time pricing analysis, never historical tracking. I tend to wonder how much of the difference is due to Amazon's specific choice of hardware for each generation. That is, would the trend look different if you limited the comparison to the cost of hardware with only the same specs as theirs (which are, of course, not necessarily, or even ever, the best bang-for-the-buck)?
> I think spot instances are a difference in kind (and actually a really great deal, if you have a workload that can be interrupted like that) - but most workloads I have worked with, especially in corp, don't really work that way.
Agreed, and they require additional engineering work (and forethought) to take advantage of properly. I just like to mention them to avoid any cries of "but cloud doesn't have to cost full price" when venturing into the territory of how expensive it is. (Also, with AWS, previous generation, and reserved instances, though knowled of the latter is implied with the spot market).
My point isn't that it doesn't happen, nor that it couldn't happen in this space. Rather, my point is, why would they bother (be willing)?
> there are plenty of people who are willing to sell two dollar bills for a dollar when the economy is hot. (I'm not suggesting amazon does this... but they certainly could. It's possible. )
In fact, we pretty much know, because Amazon has revealed it, that AWS is making money hand-over-fist, and they do this while having sizeable (existing) market share.
Would the other providers compete on (non-public) price just to gain (new) market share? It's possible, but I maintain it's an extraordinary proposition, considering Microsoft's overall software ecosystem integration advantage with Azure (much better lock-in) and Google's various attempts at features, including AI hardware.
Of course, this is all fairly academic, because..
> This is true...but that's a pretty high upper bound, when you compare to owning hardware/leasing pipe and power at all but the smallest scale.
The OP admitted that the cost savings is based on public pricing, so the decision makers don't actually care about the hugely inflated cost of cloud infrastructure! (Or, as in my personal experience, they refuse to believe it, which makes job hunting particularly surreal). That further supports my belief that the providers have no motivation to negotiate price.
> Historically, the cost of compute has come down, though slower than the cost of hardware (at least until lately)
(Did you mean the parenthetical to apply to this or to the statement about bandwidth?)
Is there anywhere that tracks this? I've only ever done that-point-in-time pricing analysis, never historical tracking. I tend to wonder how much of the difference is due to Amazon's specific choice of hardware for each generation. That is, would the trend look different if you limited the comparison to the cost of hardware with only the same specs as theirs (which are, of course, not necessarily, or even ever, the best bang-for-the-buck)?
> I think spot instances are a difference in kind (and actually a really great deal, if you have a workload that can be interrupted like that) - but most workloads I have worked with, especially in corp, don't really work that way.
Agreed, and they require additional engineering work (and forethought) to take advantage of properly. I just like to mention them to avoid any cries of "but cloud doesn't have to cost full price" when venturing into the territory of how expensive it is. (Also, with AWS, previous generation, and reserved instances, though knowled of the latter is implied with the spot market).
>Would the other providers compete on (non-public) price just to gain (new) market share? It's possible, but I maintain it's an extraordinary proposition, considering Microsoft's overall software ecosystem integration advantage with Azure (much better lock-in) and Google's various attempts at features, including AI hardware.
The better their lock-in, the more sense it makes for them to lose money at first, if that's what it takes to get you into bondage. (assuming they have a way of raising the prices later, or of dropping their costs later) I'm just saying that it would totally make sense to do this if it got you more business. Maybe they only do it for especially juicy customers? I mean, yeah, I'm reaching for some way that it might conceivably be cheaper to rent servers when you have already done the work of setting up your own datacenter, and it's hard.
(now, I do understand that most in house datacenters have ridiculous policies that managers would be happy to pay 5x to get around. But if you are at f50 scale, you can setup your own "internal cloud" with amazon like policy affordances.)
In the aughts, the standard thing to do when starting a VPS provider was to come in as cheap as you possibly could, get as many customers as you could, and then just leave your prices the same as the hardware (and thus power) costs fell out from under you. I mean, I don't know anyone who lost money on their marginal customer, but plenty started out with margins that wouldn't have been any good long-term.
That said, I don't know of any cloud infrastructure providers who seem to be losing money on their marginal customer, and as you point out, there's very good evidence that AWS makes a killing. My impression is that it's an industry with pretty good margins, once you scale enough to amortize your labor costs, which is harder than it looks. I mean, in those days, hardware(and thus power) costs were dropping dramatically, labor costs were not, and even if you have a scalable system where one person can handle a lot of servers, you need a minimum size team, which gets expensive fast.
>(Did you mean the parenthetical to apply to this or to the statement about bandwidth?)
the recent change is that hardware has gotten... more expensive in the last little bit. not just ddr4 (but mostly ddr4... and that's most of your cost, a lot of the time, as that's the thing you least want to oversubscribe) To me? this is shocking. Honestly, I haven't really been keeping up with bandwidth prices since I sold my thing, so I wouldn't know as much about that side of things.
>Is there anywhere that tracks this? I've only ever done that-point-in-time pricing analysis, never historical tracking.
pcpartpicker has nice historical graphs for hardware pricing that are publicly accessible (but that don't go back so far) One of my super early projects was a web price scraper for books... I wish I had kept up with that project and expanded it to computer parts, because there's a lot of data that I don't know how to get anymore.
https://pcpartpicker.com/trends/price/memory/
>I tend to wonder how much of the difference is due to Amazon's specific choice of hardware for each generation. That is, would the trend look different if you limited the comparison to the cost of hardware with only the same specs as theirs (which are, of course, not necessarily, or even ever, the best bang-for-the-buck)?
From what I've seen, the x86 server market is pretty efficient. Yes, yes, the professionals will take you if you let them, but it's pretty easy to get to within 10% of the tray price even on small purchases, and it's hard to get much below tray prices, even on large purchases - (dramxchange is a pretty good resource if you want to know about 'tray price' for ram) I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it mostly even uses the same chipsets, and most of the cost is the cpu/ram, so there's really not a lot of difference in who you buy from, as long as you get quotes from more than one vendor.
The times I've seen people get really ripped off, they got sold different types of hardware; like one place I worked was getting sold blade servers using FBDIMMs. Blade servers for "efficency" - of course, there was only enough power for one blade chassis per rack. And FBDIMMs used like 3x the power of low voltage reg. ecc ddr2. I pointed out the existence of the low power xeons that took reg. ecc ddr2 rather than FBDIMMs, and ran enough benchmarks to show that they were about as good as the regular xeons of similar speed... and used a lot less power. I'm not aware of any similar traps in the market right now, but I am a little out of the loop.
My personal theory as to the lack of cost concern in "cloud" is that times are hot. When times are hot? you jump. make hay while the sun is shining. If spinning up cloud servers is faster than developing your datacenter infrastructure? who cares if it costs 5x as much. Setting up openstack is a huge pain in the ass, and you're going to have growing pains for the next six months, if your team is really good.
My personal theory is that this trend will reverse with the next downturn. Meanwhile, I'm working for other people, which is pretty nice in a lot of ways, even though it does feel sad that all the time I spent in managing x86 hardware is something people just don't care about anymore.
The better their lock-in, the more sense it makes for them to lose money at first, if that's what it takes to get you into bondage. (assuming they have a way of raising the prices later, or of dropping their costs later) I'm just saying that it would totally make sense to do this if it got you more business. Maybe they only do it for especially juicy customers? I mean, yeah, I'm reaching for some way that it might conceivably be cheaper to rent servers when you have already done the work of setting up your own datacenter, and it's hard.
(now, I do understand that most in house datacenters have ridiculous policies that managers would be happy to pay 5x to get around. But if you are at f50 scale, you can setup your own "internal cloud" with amazon like policy affordances.)
In the aughts, the standard thing to do when starting a VPS provider was to come in as cheap as you possibly could, get as many customers as you could, and then just leave your prices the same as the hardware (and thus power) costs fell out from under you. I mean, I don't know anyone who lost money on their marginal customer, but plenty started out with margins that wouldn't have been any good long-term.
That said, I don't know of any cloud infrastructure providers who seem to be losing money on their marginal customer, and as you point out, there's very good evidence that AWS makes a killing. My impression is that it's an industry with pretty good margins, once you scale enough to amortize your labor costs, which is harder than it looks. I mean, in those days, hardware(and thus power) costs were dropping dramatically, labor costs were not, and even if you have a scalable system where one person can handle a lot of servers, you need a minimum size team, which gets expensive fast.
>(Did you mean the parenthetical to apply to this or to the statement about bandwidth?)
the recent change is that hardware has gotten... more expensive in the last little bit. not just ddr4 (but mostly ddr4... and that's most of your cost, a lot of the time, as that's the thing you least want to oversubscribe) To me? this is shocking. Honestly, I haven't really been keeping up with bandwidth prices since I sold my thing, so I wouldn't know as much about that side of things.
>Is there anywhere that tracks this? I've only ever done that-point-in-time pricing analysis, never historical tracking.
pcpartpicker has nice historical graphs for hardware pricing that are publicly accessible (but that don't go back so far) One of my super early projects was a web price scraper for books... I wish I had kept up with that project and expanded it to computer parts, because there's a lot of data that I don't know how to get anymore.
https://pcpartpicker.com/trends/price/memory/
>I tend to wonder how much of the difference is due to Amazon's specific choice of hardware for each generation. That is, would the trend look different if you limited the comparison to the cost of hardware with only the same specs as theirs (which are, of course, not necessarily, or even ever, the best bang-for-the-buck)?
From what I've seen, the x86 server market is pretty efficient. Yes, yes, the professionals will take you if you let them, but it's pretty easy to get to within 10% of the tray price even on small purchases, and it's hard to get much below tray prices, even on large purchases - (dramxchange is a pretty good resource if you want to know about 'tray price' for ram) I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it mostly even uses the same chipsets, and most of the cost is the cpu/ram, so there's really not a lot of difference in who you buy from, as long as you get quotes from more than one vendor.
The times I've seen people get really ripped off, they got sold different types of hardware; like one place I worked was getting sold blade servers using FBDIMMs. Blade servers for "efficency" - of course, there was only enough power for one blade chassis per rack. And FBDIMMs used like 3x the power of low voltage reg. ecc ddr2. I pointed out the existence of the low power xeons that took reg. ecc ddr2 rather than FBDIMMs, and ran enough benchmarks to show that they were about as good as the regular xeons of similar speed... and used a lot less power. I'm not aware of any similar traps in the market right now, but I am a little out of the loop.
My personal theory as to the lack of cost concern in "cloud" is that times are hot. When times are hot? you jump. make hay while the sun is shining. If spinning up cloud servers is faster than developing your datacenter infrastructure? who cares if it costs 5x as much. Setting up openstack is a huge pain in the ass, and you're going to have growing pains for the next six months, if your team is really good.
My personal theory is that this trend will reverse with the next downturn. Meanwhile, I'm working for other people, which is pretty nice in a lot of ways, even though it does feel sad that all the time I spent in managing x86 hardware is something people just don't care about anymore.
> nice historical graphs for hardware pricing that are publicly accessible
Even then, that's only half the comparison, though. Do you know of anyone keeping historical AWS prices? That would seem more likely, actually, than a reliable pc-parts database.
> I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it mostly even uses the same chipsets, and most of the cost is the cpu/ram, so there's really not a lot of difference in who you buy from, as long as you get quotes from more than one vendor.
Noooo.. no you too! :)
I find that this is true, but only if the workload is primarily CPU- and RAM-bound. My sense is that somewhere between the dot-com boom and now (500x increase in speed?) CPUs got fast enough, and software got parallel enough, that the challenge in scaling shifted CPU capacity to I/O. (More RAM always seems to help, but I agree there's not much differentiation there).
As such, at any given point in time, each vendor's offering at a particular (otherwise competitive) price point might not make sense. The mobo might not have enough PCIe slots with enough lanes (or too many on too few slots). The onboard 10GE might use a chipset that performs poorly under OpenBSD. The chassis might hold comically few disks (ahem, Dell) or have no SAS expander backplane option.
The other differentiator, is at the high end, for >2S systems. It's arguably no longer "commodity" hardware, but the parts are more similar than different, and there's no lock-in if one can wait at least the full release cycle. Still, it's rare that anyone is brave enough to buck the trend and pay the 2x-4x price premium to scale "up" before embarking on the engineering nightmare that is scaling "out" (which I'm not convinced has any less of a hardware price premium, just that it can grow without bound).
> Blade servers for "efficency"
I've never understood that mass delusion, either, though at least not there do exist high-power-density datacenters that can house them. I assume someone managed to spread the myth that it's space that's expensive at a datacenter, rather than power and cooling.
> I'm not aware of any similar traps in the market right now
I'm not, either, at this very moment, but every time I've looked, out of curiosity, there's something like that. All the vendors will have at least one product that seems objectively worse in every way to the older products, including being more proprietary, that I wonder if it isn't some custom design that flopped that they're trying to dump.
>My personal theory as to the lack of cost concern in "cloud" is that times are hot.
Agreed. Easy money, be it from VCs, low interest rates, and/or a booming economy (perhaps fueled by the first two) helps explain it.
> Setting up openstack is a huge pain in the ass,
Is that really the go-to option? I'd think there are much lighter-weight alternatives. I'd have thought provisioning the base infrastructure were much less a PITA than writing the "infrastructure as code" which one has to do even for public cloud.
> growing pains for the next six months,
Is it really that long? Or is that just the perception?
> if your team is really good.
Maybe that's the problem. Good help is hard to find?
Still, what about 1-2 years down the line of making hay at those 5x costs? There are plenty of companies like that and, at some point, it has to at least be slightly tempting to go through the effort, including hiring. Maybe the money really is too easy.
> My personal theory is that this trend will reverse with the next downturn.
I've been thinking the same thing, but I'm wondering what will happen with all that excess capacity that cloud providers are suddenly left holding.
> Meanwhile, I'm working for other people, which is pretty nice in a lot of ways, even though it does feel sad that all the time I spent in managing x86 hardware is something people just don't care about anymore.
I've always worked for other people (which also makes me worry about too strong a downturn), and the hardware management has only been a part of what I do.
I'm not so much sad at the apathy toward one of my skills. It's more that I'm so limited in how much performance I can squeeze out of the infrastructure or how low I can keep the cost. It makes for weird tension.
Even then, that's only half the comparison, though. Do you know of anyone keeping historical AWS prices? That would seem more likely, actually, than a reliable pc-parts database.
> I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it mostly even uses the same chipsets, and most of the cost is the cpu/ram, so there's really not a lot of difference in who you buy from, as long as you get quotes from more than one vendor.
Noooo.. no you too! :)
I find that this is true, but only if the workload is primarily CPU- and RAM-bound. My sense is that somewhere between the dot-com boom and now (500x increase in speed?) CPUs got fast enough, and software got parallel enough, that the challenge in scaling shifted CPU capacity to I/O. (More RAM always seems to help, but I agree there's not much differentiation there).
As such, at any given point in time, each vendor's offering at a particular (otherwise competitive) price point might not make sense. The mobo might not have enough PCIe slots with enough lanes (or too many on too few slots). The onboard 10GE might use a chipset that performs poorly under OpenBSD. The chassis might hold comically few disks (ahem, Dell) or have no SAS expander backplane option.
The other differentiator, is at the high end, for >2S systems. It's arguably no longer "commodity" hardware, but the parts are more similar than different, and there's no lock-in if one can wait at least the full release cycle. Still, it's rare that anyone is brave enough to buck the trend and pay the 2x-4x price premium to scale "up" before embarking on the engineering nightmare that is scaling "out" (which I'm not convinced has any less of a hardware price premium, just that it can grow without bound).
> Blade servers for "efficency"
I've never understood that mass delusion, either, though at least not there do exist high-power-density datacenters that can house them. I assume someone managed to spread the myth that it's space that's expensive at a datacenter, rather than power and cooling.
> I'm not aware of any similar traps in the market right now
I'm not, either, at this very moment, but every time I've looked, out of curiosity, there's something like that. All the vendors will have at least one product that seems objectively worse in every way to the older products, including being more proprietary, that I wonder if it isn't some custom design that flopped that they're trying to dump.
>My personal theory as to the lack of cost concern in "cloud" is that times are hot.
Agreed. Easy money, be it from VCs, low interest rates, and/or a booming economy (perhaps fueled by the first two) helps explain it.
> Setting up openstack is a huge pain in the ass,
Is that really the go-to option? I'd think there are much lighter-weight alternatives. I'd have thought provisioning the base infrastructure were much less a PITA than writing the "infrastructure as code" which one has to do even for public cloud.
> growing pains for the next six months,
Is it really that long? Or is that just the perception?
> if your team is really good.
Maybe that's the problem. Good help is hard to find?
Still, what about 1-2 years down the line of making hay at those 5x costs? There are plenty of companies like that and, at some point, it has to at least be slightly tempting to go through the effort, including hiring. Maybe the money really is too easy.
> My personal theory is that this trend will reverse with the next downturn.
I've been thinking the same thing, but I'm wondering what will happen with all that excess capacity that cloud providers are suddenly left holding.
> Meanwhile, I'm working for other people, which is pretty nice in a lot of ways, even though it does feel sad that all the time I spent in managing x86 hardware is something people just don't care about anymore.
I've always worked for other people (which also makes me worry about too strong a downturn), and the hardware management has only been a part of what I do.
I'm not so much sad at the apathy toward one of my skills. It's more that I'm so limited in how much performance I can squeeze out of the infrastructure or how low I can keep the cost. It makes for weird tension.
Your employees need skills that are useful and in demand. They need those going forward, whether you close your data centers or not.
I'd suggest you be upfront with them. Use this as a pro-active way to show you want them around, but it's not business-as-usual. The good ones will see this as an opportunity for the long term, while others may view this as a time to move on.
To reinforce plans, show them what skills you need after the move. This will give a north star to motivate those who want career mobility.
I'd suggest you be upfront with them. Use this as a pro-active way to show you want them around, but it's not business-as-usual. The good ones will see this as an opportunity for the long term, while others may view this as a time to move on.
To reinforce plans, show them what skills you need after the move. This will give a north star to motivate those who want career mobility.
The implementation of hybrid cloud support is the key thing to consider pursuing both from a strategic and technical perspective. I have provided complex hosting for large corporations for more years than I like to remember and it is the only thing of lasting and permanent value for a few reasons:
Having a completely migratable cloud setup, between any cloud provider, or back to your own environment (OpenStack, etc), is the single most important long term investment to consider for your stack.
Your situation sounds very advantageous in having the expertise ready to containerize machines, and dockerize applications that may not be. This too, is a very marketable skill for people who may be leaving.
Historically, connectivity has driven location of data. Popularity has oscillated between on-premise vs client-server setups (cloud). Currently, the cloud is in vogue, for many great reasons. The pendulum though seems to have historically swung both ways.
While network speed will continue to increase, and computing power will continue to increase, other factors may play into why a corporation wishes to keep it's data in-house, be it political, security, industry practices, or a sudden wish to change directions due to a failure too many of a cloud provider.
The cloud providers do a very good job of making it easy to get sucked into their world.
It would be interesting if you followed up here with the decision you made and why. Not enough thought seems to be out in the open about these types of decisions, let alone at your decisions.
Having a completely migratable cloud setup, between any cloud provider, or back to your own environment (OpenStack, etc), is the single most important long term investment to consider for your stack.
Your situation sounds very advantageous in having the expertise ready to containerize machines, and dockerize applications that may not be. This too, is a very marketable skill for people who may be leaving.
Historically, connectivity has driven location of data. Popularity has oscillated between on-premise vs client-server setups (cloud). Currently, the cloud is in vogue, for many great reasons. The pendulum though seems to have historically swung both ways.
While network speed will continue to increase, and computing power will continue to increase, other factors may play into why a corporation wishes to keep it's data in-house, be it political, security, industry practices, or a sudden wish to change directions due to a failure too many of a cloud provider.
The cloud providers do a very good job of making it easy to get sucked into their world.
It would be interesting if you followed up here with the decision you made and why. Not enough thought seems to be out in the open about these types of decisions, let alone at your decisions.
This sounds like a very exciting time, the work is changing, the very economics of the business are changing. Remember to lead throughout this transition. Communicate well, frame the transition in a positive way and take care of people. Do not allow your exuberance at the potential cost savings blind you to the pitfalls in a project of this magnitude. The transition will be harder than you think, take longer and cost more. That being said, it’s probably worth doing. Along the way you must remember that your employees are a super important part of your business so treat them accordingly.
Offer training. The automation work is more gratifying and valuable anyway. You’ll be offering them a step forward. Some will take it, some will move on. Even those who move on may find that the change catapults them forward as well.
I trained a colleague in cloud automation in two months. Admittedly he is very capable and committed, but you’ll find that in some of your people too. He’s now worth more than anyone I could have hired. You have the opportunity to elevate your existing employees significantly. Embrace that opportunity.
PM me if you’d like to see the training materials I’ve put together.
Offer training. The automation work is more gratifying and valuable anyway. You’ll be offering them a step forward. Some will take it, some will move on. Even those who move on may find that the change catapults them forward as well.
I trained a colleague in cloud automation in two months. Admittedly he is very capable and committed, but you’ll find that in some of your people too. He’s now worth more than anyone I could have hired. You have the opportunity to elevate your existing employees significantly. Embrace that opportunity.
PM me if you’d like to see the training materials I’ve put together.
I would love to see those training materials! I don't see any contact info, but maybe you could shoot me an email to the email in my profile?
You can reach out privately to other companies and leaders you know if they may have a position for them. HR can also help find positions for people, that’s part of their job.
What you can’t do make a decision that’s good for a very small number of people at the expense of everyone else in the company if you’re a leader.
What you can’t do make a decision that’s good for a very small number of people at the expense of everyone else in the company if you’re a leader.
You’re not going to be able to just flip a switch and do that migration instantly. You have time to retrain people and some people will adapt better than others.
Can you last long term if you don’t move? If there are huge savings to be made others in the same space might jump on the opportunity.
It’s a horrible thing to have to let people go but the alternative may be far worse.
It’s a horrible thing to have to let people go but the alternative may be far worse.
I will assume that at the current state you are able to pay both the datacenter costs and people costs since it seems that is what you are currently doing.
With that assumption in mind, if you do migrate and realize that you now don’t need 80% of the people you needed before, it could mean one of two things:
- you are now able to pocket whatever savings you were able get from the migration and either save it or invest in something to potentially keep that money in the economy, just in a different way
- you now have 80% of your people free to do whatever other things you didn’t do because they had to maintain datacenters. That is huge in terms of efficiency gains and allows you to redirect that money to increase the value provided by your company either by repurposing the current workforce or getting rid of them and bringing people you could previously not bring on board due to existing costs
Either way I believe that everybody is better off After the changes.
Efficiency gains such as the one you are describing is what makes it possible for economies to grow and keep moving forward.
Can you imagine if you had to handle cows, pigs and farms to have a BLT? It’s very good that such things are outsourced because that allows for improvement that would otherwise not be possible.
With that assumption in mind, if you do migrate and realize that you now don’t need 80% of the people you needed before, it could mean one of two things:
- you are now able to pocket whatever savings you were able get from the migration and either save it or invest in something to potentially keep that money in the economy, just in a different way
- you now have 80% of your people free to do whatever other things you didn’t do because they had to maintain datacenters. That is huge in terms of efficiency gains and allows you to redirect that money to increase the value provided by your company either by repurposing the current workforce or getting rid of them and bringing people you could previously not bring on board due to existing costs
Either way I believe that everybody is better off After the changes.
Efficiency gains such as the one you are describing is what makes it possible for economies to grow and keep moving forward.
Can you imagine if you had to handle cows, pigs and farms to have a BLT? It’s very good that such things are outsourced because that allows for improvement that would otherwise not be possible.
Odds are high that the anticipated cost savings sits largely on the folks they intend to sack, rather than on any power/real estate/bandwidth/hardware savings.
Actually the cost savings hinge on the massive cost savings from moving to a commercial cloud, laying off workforce would save more, but would prefer to save a bunch by moving to cloud and keep the workforce for other things.
That's extremely surprising.
I see two possible explanations (both of which may apply):
(1) You're getting the cloud servers for a remarkable discount, which, as other commenters mentioned, may not last if you don't put in the engineering effort to be vendor-agnostic (though I'm not sure I agree it's as huge a cost as otherwise implied).
(2) You're grossly over-paying for your current environment. Given what I know about the purchasing habits over merely large (nowhere near the size of F50, mind you) companies and, more importantly, the pricing of "enterprise" hardware and support, this seems likely.
If the second is true, then I expect you could gain the same savings, if not more, by implementing your own "private cloud" by emulating the purchasing and ops habits of larger-scale startups or established tech companies (but not yet so huge as to be too custom) who run their own hardware.
You'd still have to retrain and put an end to manual configuration, but I expect that would ultimately be welcomed. Some might be able to keep their jobs almost as-is.
What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. You'd probably have to hire someone who knows how to be frugal like that from an engineering/ops standpoint for architecture/purchasing and is willing to work for such a large company and isn't already employed by FAAMG.
I see two possible explanations (both of which may apply):
(1) You're getting the cloud servers for a remarkable discount, which, as other commenters mentioned, may not last if you don't put in the engineering effort to be vendor-agnostic (though I'm not sure I agree it's as huge a cost as otherwise implied).
(2) You're grossly over-paying for your current environment. Given what I know about the purchasing habits over merely large (nowhere near the size of F50, mind you) companies and, more importantly, the pricing of "enterprise" hardware and support, this seems likely.
If the second is true, then I expect you could gain the same savings, if not more, by implementing your own "private cloud" by emulating the purchasing and ops habits of larger-scale startups or established tech companies (but not yet so huge as to be too custom) who run their own hardware.
You'd still have to retrain and put an end to manual configuration, but I expect that would ultimately be welcomed. Some might be able to keep their jobs almost as-is.
What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. You'd probably have to hire someone who knows how to be frugal like that from an engineering/ops standpoint for architecture/purchasing and is willing to work for such a large company and isn't already employed by FAAMG.
Maybe so, I haven't made this move before, which is part of the reason I'm asking here, as unlikely as it may be that "someone who is in this position would be asking on HN." Sorry for being vague on all of this. Regardless:
1.) No specific discount, this is standard, public pricing on major commercial clouds, with some tricks applied like reserved instances, which is part of the reason I am not worried about low prices first and then ballooning pricing later,
2.) Our operations and maintenance and sustaining are in fact more expensive that anyone would reasonably expect, I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices, I've walked the data center floors and coded with the ops teams.
It is #2 that is really hitting us. I have personally moved several apps as a PoC and you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs and autoscaling.
I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. Thank you for the response.
1.) No specific discount, this is standard, public pricing on major commercial clouds, with some tricks applied like reserved instances, which is part of the reason I am not worried about low prices first and then ballooning pricing later,
2.) Our operations and maintenance and sustaining are in fact more expensive that anyone would reasonably expect, I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices, I've walked the data center floors and coded with the ops teams.
It is #2 that is really hitting us. I have personally moved several apps as a PoC and you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs and autoscaling.
I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. Thank you for the response.
I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying.
The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/co...
Let me guess... you're using kit from the vendor at the bottom of that list? :)
No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.
The commentator your/this reply is to points out an interesting aspect:
> What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect.
This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.
Well... the landscape is indeed changing, as clarified by the 9-year old article above.
So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.
(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)
And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.
If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses).
I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is.
Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing...
> I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally.
I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on.
There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here.
This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.)
Cannot keep writing as I must go, may continue later
The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/co...
Let me guess... you're using kit from the vendor at the bottom of that list? :)
No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.
The commentator your/this reply is to points out an interesting aspect:
> What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect.
This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.
Well... the landscape is indeed changing, as clarified by the 9-year old article above.
So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.
(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)
And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.
If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses).
I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is.
Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing...
> I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally.
I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on.
There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here.
This was what I was getting at with the "the offer will be what the buying team wants to hear" - at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.)
Cannot keep writing as I must go, may continue later
> Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards
I suspect it's this kind of over-reach that ends up scaring decision-makers off from any in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions.
Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this?
I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s.
What we don't know, however, is if those backplanes were actually a commodity yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats).
To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.
I suspect it's this kind of over-reach that ends up scaring decision-makers off from any in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions.
Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this?
I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s.
What we don't know, however, is if those backplanes were actually a commodity yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats).
To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.
> Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard.
Hmmm. Good point. (Maybe I scared OP away, heh.)
> Is there an example of any large company, other than Google, who has mastered this?
Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s (2015).
Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable].
Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.)
As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on https://about.gitlab.com/ (the one from Forrester) doesn't show GitHub :D
Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well.
I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.
Hmmm. Good point. (Maybe I scared OP away, heh.)
> Is there an example of any large company, other than Google, who has mastered this?
Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s (2015).
Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable].
Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.)
As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on https://about.gitlab.com/ (the one from Forrester) doesn't show GitHub :D
Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well.
I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.
> I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta
I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all).
Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options.
My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available.
> Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior
That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower-end/frugal market.
> I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively
Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly massive scale.
Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a smaller VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company.
> (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc
That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price.
At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale.
Networking gear isn't quite commoditized, but we're close, with open-source/SDN type platforms.
One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along).
[1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.
I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all).
Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options.
My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available.
> Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior
That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower-end/frugal market.
> I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively
Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly massive scale.
Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a smaller VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company.
> (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc
That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price.
At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale.
Networking gear isn't quite commoditized, but we're close, with open-source/SDN type platforms.
One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along).
[1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.
> I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices
I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative.
I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly.
Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any).
The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway.
> you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs
I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap.
VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true).
> and autoscaling.
This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well.
Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings.
The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking.
Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill).
[1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.
I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative.
I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly.
Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any).
The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway.
> you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs
I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap.
VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true).
> and autoscaling.
This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well.
Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings.
The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking.
Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill).
[1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.
You really should contact me in confidence. We’ve done almost exactly this and the data I have is compelling.
Moving to the cloud doesn't save money for an established business which knows their datacenter needs. Cloud providers pay for the same hardware, staff to manage that hardware, and then add a layer of profit on top. Going to the cloud only makes sense if you believe you can fire people.
And realistically, you need most of those people to maintain and configure your cloud service.
It's hard to imagine someone Fortune 50 scale gaining any cost efficiency from going to a cloud service, is this a temporary pricing promotion you're looking to take advantage of or have you done all your homework on the savings? How confident are you on what your cloud expenses will look like?
And realistically, you need most of those people to maintain and configure your cloud service.
It's hard to imagine someone Fortune 50 scale gaining any cost efficiency from going to a cloud service, is this a temporary pricing promotion you're looking to take advantage of or have you done all your homework on the savings? How confident are you on what your cloud expenses will look like?
I have to assume the massive savings are from a top tier deal to secure a f50 as a client. You know once they move they aren’t going to move again.
Google, AWS, Azure would all love a f50 client and I don’t think they’d care too much about breaking even on the costs of the service or even losing some money because of the amount of social capital they earn from the “partnership”.
Google, AWS, Azure would all love a f50 client and I don’t think they’d care too much about breaking even on the costs of the service or even losing some money because of the amount of social capital they earn from the “partnership”.
They are all about profits from inertia. I’m seeing organizations migrate to the cloud lured by low initial upfront cost that increase once they have completed their migration. When the real bills come in people do hand waving and say the increased cost are negated by “agility.” And don’t even think about migrating out of a cloud provider...you will have to beg for permission.
There are a lot of benefits to migrating to the cloud, in my experience, cost isn’t one of them. Unless of course you are mucking around with accounting tricks by cutting employees and moving that cost to services or cost of goods sold, I’m not a expert in this, but it works and will save you money because you can control these expenses more easily and because of how payroll vs services are accounted for.
There are a lot of benefits to migrating to the cloud, in my experience, cost isn’t one of them. Unless of course you are mucking around with accounting tricks by cutting employees and moving that cost to services or cost of goods sold, I’m not a expert in this, but it works and will save you money because you can control these expenses more easily and because of how payroll vs services are accounted for.
Nobody's going to do the scale of an F50 for free. Any sort of good deal is going to have an expiration date, or it won't cover expansion that will almost certainly be needed, potentially almost immediately if all of their needs aren't immediately apparent.
This is one of the other risks of moving to a cloud provider: You add a new dependency on a single company. If you don't have a multi-cloud strategy that allows you to force Google, Amazon, and Microsoft to fight for your business each time you expand, you're going to eventually end up paying top dollar.
This is one of the other risks of moving to a cloud provider: You add a new dependency on a single company. If you don't have a multi-cloud strategy that allows you to force Google, Amazon, and Microsoft to fight for your business each time you expand, you're going to eventually end up paying top dollar.
If you plan on being „cloud agnostic“, not using the services they offer like databases, you might as well stay in your own DC.
I agree with my sibling-commenter that this is a bit of a stretch, and I also think that merely not using a cloud provider's integrated database offering (e.g. RDS instead of running ones own on EC2 with or without EBS) doesn't invalidate all the benefits.
However, you do bring up an important point, which is a criticism I've always had for the term "infrastructure as code": Unless the underlying hardware is identical in performance, that code isn't really describing/commanding anything deterministic.
This is especially true for databases or anything I/O-sensitive like message queues or stream processing.
It would certainly make "cloud agnosticism" a much less trivial exercise.
At the risk of sounding ad-hominem, I generally find that it is software engineers whose experience is primarily at higher levels of abstraction who under-estimate how much it matter (or, conversely, over-estimate how much software or "code" can just solve such problems).
See also https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
However, you do bring up an important point, which is a criticism I've always had for the term "infrastructure as code": Unless the underlying hardware is identical in performance, that code isn't really describing/commanding anything deterministic.
This is especially true for databases or anything I/O-sensitive like message queues or stream processing.
It would certainly make "cloud agnosticism" a much less trivial exercise.
At the risk of sounding ad-hominem, I generally find that it is software engineers whose experience is primarily at higher levels of abstraction who under-estimate how much it matter (or, conversely, over-estimate how much software or "code" can just solve such problems).
See also https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
This is a non sequitur. You can, and should, leverage the flexibility of cloud computing without being too locked down to a single vendor.
That’s one of the things tools like Terraform, Kubernetes or Mesos were designed for.
That’s one of the things tools like Terraform, Kubernetes or Mesos were designed for.
Terraform doesn’t buy you anything. It supports multiple vendors, but you have to re-write all the code when switching.
If the lowest common denominator is K8S/Mesos, while not using any of the cloud services, you’re doing it wrong.
If the lowest common denominator is K8S/Mesos, while not using any of the cloud services, you’re doing it wrong.
Which would do you have to re-write? Your deployment code or your application code. If it’s the latter you’re not using the cloud correctly. If it’s the former, re-writing deployment scripts is much easy than physically migrating a DC which is think is the real comparison.
I have built multi-cloud, totally cloud-agnostic deployments at my previous company with terraform, for really interesting reasons. At least for tf even if you are spinning up a VM and installing something like postgres on it (cloud agnostic, no managed services), rewriting is required because the order of operations can be different in infrastructure building, different modules, different vars, etc. You can use your code for one cloud as a template for the new one, but you do need to maintain a set of scripts for each cloud in my experience.
I thought this was why we moved away from mainframe computing to general purpose hardware...now the best practice is to go the opposite direction, this time we are calling shared compute resources “cloud.”
Some cloud providers will be better than others. AWS is still very expensive for I/O intensive workloads. The tendency to revert to mainframe area commitments because a vendor provides a short term discount without factoring in true cost required to maintain the same QoS provided by on-prem is common for executives.
A new layer of abstraction is a major benefit of the cloud.
Some cloud providers will be better than others. AWS is still very expensive for I/O intensive workloads. The tendency to revert to mainframe area commitments because a vendor provides a short term discount without factoring in true cost required to maintain the same QoS provided by on-prem is common for executives.
A new layer of abstraction is a major benefit of the cloud.
How much actual benchmarking have you been able to complete for your applications?
Have you already virtualized your own server and applications in the DC you have?
Have you already virtualized your own server and applications in the DC you have?
[deleted]
They need cloud certifications. They should see this as a fabulous opportunity to get paid to do cloud training, hands-on work. If you become a cloud architect or a cloud security expert, whether it is at your company or somewhere else, there will always be great opportunity. If they refuse to learn to serve a changing market, that is where they will run into problems. You can only ride a single skill for so long. If you stayed in the horse-shoe business after about 1904, life was not great.
I'd recommend CNCF's training for Kubernetes: https://www.cncf.io/certification/training/
We offer a free edX course and then a paid one that prepares you for the Certified Kubernetes Administrator exam.
Disclosure: I am executive director of the CNCF and helped create these courses.
P.S. No job will be around forever. You are not doing you or them any favors by trying to maintain them in inefficient roles. But you can invest the resources to bring them with you to the new world.
We offer a free edX course and then a paid one that prepares you for the Certified Kubernetes Administrator exam.
Disclosure: I am executive director of the CNCF and helped create these courses.
P.S. No job will be around forever. You are not doing you or them any favors by trying to maintain them in inefficient roles. But you can invest the resources to bring them with you to the new world.
If your business has competitors, and they move to the cloud, conceivably they will be able to lower prices, and/or provide more service/features because of cloud scale. In most instances, the less your competitors charge, the more customers flock to your competitors. It is not personal, but economic in my opinion.
Our business is an old, traditional business that happens to have lots of data centers for legacy reasons. Even though we spend $BBB on data centers, they're not the core of the business and a very small part of overall revenue. Realistically, even if we didn't move from data centers we would be good as long as the core non-tech business functioned.
As an economic principal what you suggest is not the best use of inventor assets and compounds when you factor in employee efficiency/return per employee. If you invested your hard earned money into this business you would want cost efficiencies and high productivity. Company executives have a fiduciary responsibility to do so in the US. If you're running a business and are not seeking cost efficiency but instead are more interested in employee satisfaction, likely you should be in HR and let the CEO handle day to day operations and the bottom line, so you can contribute to the company by driving employee retention, satisfaction or some metric that reflects positively on the bottom line. It's OK to the a CEO. It's OK to be in HR. It's not OK to be a CEO in HR. It's not OK to be a HR as CEO.
Point well taken. But as many have mentioned in this thread, it is very hard to hire good people and the people I have already have deep knowledge of the systems and business and are good at what they do. Just from that efficiency perspective it makes sense to retrain per fiduciary responsibility. The fact that I can keep lots and lots of people working instead of terminating is an awesome bonus that I feel great about.
Are you sure you’re spending triple-digit Billions in dollars for datacenters? Or are you just trolling?
While automation is removing some jobs, we are far from a fully automated society. I obviously don't know about the specific job markets where your data centers are, but in general, the ones having the hardest time finding a job aren't the ones with years of work experience in running data centers.
They might have to retrain and if they live in a small town they might have to move/commute to find something new however. This can be hard enough for some people that they won't be happy about it.
In the end, I think it's hard to argue that we should keep being inefficient just to save jobs. History haven't been nice to that kind of companies.
They might have to retrain and if they live in a small town they might have to move/commute to find something new however. This can be hard enough for some people that they won't be happy about it.
In the end, I think it's hard to argue that we should keep being inefficient just to save jobs. History haven't been nice to that kind of companies.
Retrain them in AWS, devops, or something that has similar need, versus hiring from outside.
Best way to retrain without having to pay AWS a huge fee for all the certs?
I really think the worst thing you could do is continue to run the business in an inefficient manner. Improving technology and as a result efficiency is the reason we have advanced so much as a species.
However, you don't need to be callous in your pursuit of efficiency. You should try to give these people an opportunity to update their skills and get a new position. You might not be in a position to do that however.
However, you don't need to be callous in your pursuit of efficiency. You should try to give these people an opportunity to update their skills and get a new position. You might not be in a position to do that however.
Just retrain them.
They're going to have tangential skills to the new platform.
Get your network team to learn all about VPCs and networks and security groups and peering and VPNs, train them to be your in-house experts to deal with changes and expansions and to help trouble-shoot issues.
Chances are you're still going to have systems, your systems engineers can be repurposed to better tie in your monitoring tools, your security tools, etc. You're going to need authentication and identity management, and probably have internal people handling it.
If they're motivated to keep their jobs, all it should cost you is a $5-20K/head in training.
That's peanuts compared to the turmoil of everyone thinking they're going to lose their jobs, or the costs of hiring new people w/ pre-existing cloud skills. It is cheaper than the stop-gap of paying outside consultants till you can bring in the interior staff. Also, these people _know_ your existing systems. They know your business logic, they know what needs to be up and they know what can't break on Thanksgiving weekend. That is valuable knowledge that is a hidden cost in rehiring the work force.
You're going to find new challenges in the new system. If your employees are willing to transition, then transition them. It is your best bet.
If you have a subset of employees that can't/refuse to transition, put them on performance improvement plans. No rumor mill - if you're on a PiP you need to shape up or get out; if you're not on a PiP keep doing well, your job is safe.
They're going to have tangential skills to the new platform.
Get your network team to learn all about VPCs and networks and security groups and peering and VPNs, train them to be your in-house experts to deal with changes and expansions and to help trouble-shoot issues.
Chances are you're still going to have systems, your systems engineers can be repurposed to better tie in your monitoring tools, your security tools, etc. You're going to need authentication and identity management, and probably have internal people handling it.
If they're motivated to keep their jobs, all it should cost you is a $5-20K/head in training.
That's peanuts compared to the turmoil of everyone thinking they're going to lose their jobs, or the costs of hiring new people w/ pre-existing cloud skills. It is cheaper than the stop-gap of paying outside consultants till you can bring in the interior staff. Also, these people _know_ your existing systems. They know your business logic, they know what needs to be up and they know what can't break on Thanksgiving weekend. That is valuable knowledge that is a hidden cost in rehiring the work force.
You're going to find new challenges in the new system. If your employees are willing to transition, then transition them. It is your best bet.
If you have a subset of employees that can't/refuse to transition, put them on performance improvement plans. No rumor mill - if you're on a PiP you need to shape up or get out; if you're not on a PiP keep doing well, your job is safe.
I can't agree more about retraining these guys. Hiring cloud engineers with good skills is extremely difficult
But is OP really going to have the same cost savings if they retain everyone? I am assuming labour is a huge part in that figure.
The ideal situation is to be in a growing company, and to move employees from shrinking areas into growing areas.
For example, if the company needs 8 fewer people to install and set up servers, but 10 more people to work on robust automated code deployments to support business growth, and both jobs need Unix skills.
Hence, the setting-up-servers department budget shows cost savings, the overall technology budget grows by 2 employees instead of by 10, and business growth means overall profit increases.
For example, if the company needs 8 fewer people to install and set up servers, but 10 more people to work on robust automated code deployments to support business growth, and both jobs need Unix skills.
Hence, the setting-up-servers department budget shows cost savings, the overall technology budget grows by 2 employees instead of by 10, and business growth means overall profit increases.
The cost saving from going cloud in a large part comes from utilization. 1 box that you can keep busy at 100% by clever workload timing is cheaper than 10 boxes that are idle 90% of the time that you just need to spike on. Or capacity that you can give back when you don’t need it. Rack space is expensive, power and cooling are expensive.
The job of the people is to translate the business requirements into stuff actually happening, and that doesn’t go away with cloud. If anything you need to do more of it (someone has to take care of auto-scaling, a thing that wasn’t possible in your legacy DC, for example). The worst thing you can do is sack the people then realize you need consultants at 5x the cost to do anything now...
The job of the people is to translate the business requirements into stuff actually happening, and that doesn’t go away with cloud. If anything you need to do more of it (someone has to take care of auto-scaling, a thing that wasn’t possible in your legacy DC, for example). The worst thing you can do is sack the people then realize you need consultants at 5x the cost to do anything now...
This claim of massive savings is ludicrous for an f50 in the first place. It’s doubtful their cost is primarily maintaining and admining fleets of x64 servers, which is what the “cloud” is optimized for. These companies generally have massive costs from legacy mainframe, iseries, mvs etc systems. The ops creds are suspect to say the least.
Your comment would be more palatable if you gave the OP benefit of the doubt. It seems reasonable to assume that someone who is in the position to affect such change, and is willing to ask HN for advice, likely has actual cost savings estimates.
Perhaps saying something like, "Are you sure the cost savings are really there, and you won't face unexpected hidden costs in the future? I have experience with X,Y,Z, and we found that while the cloud is optimized for x64 deployment, we had hidden costs A, B, C, crop up after 9 months."
As it is, it's only added noise and made you look bad.
Perhaps saying something like, "Are you sure the cost savings are really there, and you won't face unexpected hidden costs in the future? I have experience with X,Y,Z, and we found that while the cloud is optimized for x64 deployment, we had hidden costs A, B, C, crop up after 9 months."
As it is, it's only added noise and made you look bad.
Your comment would be more palatable if you gave the parent poster the benefit of the doubt.
Perhaps using http://www.paulgraham.com/disagree.html and avoid dismissing the content because you don't like the way they said it (DH2, responding to tone).
Perhaps using http://www.paulgraham.com/disagree.html and avoid dismissing the content because you don't like the way they said it (DH2, responding to tone).
He mentioned 90% of their datacenter operations. I assume the remaining 10% would be the assorted legacy every large company has of SPARC, PA-RISC, POWER, i and zSeries accretes over time.
Those will remain where they are and, eventually be ported or replaced (or kept alive - IBM mainframes have been legacy-friendly for over 50 years now) by newer systems as they are phased out.
Those will remain where they are and, eventually be ported or replaced (or kept alive - IBM mainframes have been legacy-friendly for over 50 years now) by newer systems as they are phased out.
How many are them? (10? 100?)
Are some of them close to retirement age?
Are some of them close to retirement age?
Thousands, many close to retirement age. Think a company like UPS/Fedex that isn't at its core a tech company but with tech datacenters for logistics and billions in revenue.
Thousands? I really find that hard to believe.
Im not sure why I was down-voted. Even if you close data centers and move to the cloud, you still need personnel to manage the infrastructure.
Indeed. TBH the post reads like advertising for the alleged savings of going "cloud" vs on-premise.
If serious, consider offering an early retirement package and see who bites.
hotdog97(1)
Are you sure the savings will be real? I've read that real savings results from using the cloud for bursts while keeping the base load on-prem. (Never done this personally so can't speak from experience.)
Once you let those people to you will be losing their knowledge in this area. Not sure if it matters.
Once you let those people to you will be losing their knowledge in this area. Not sure if it matters.
Since you're thinking towards the future, think also to the time when you will probably have to move an awful lot of your stuff back in-house, which might be much sooner than you'd expect. Plenty of companies which outsourced in a big way some years back have already been through a similar cycle.
A former Fortune 500 employer of mine went through the outsourcing boomerang. Now I understand that they have big plans to move everything to the cloud within the next two years. Knowing their systems like I do, and knowing that the CEO who is driving this is fairly noobish (note that I said CEO rather than CIO), I don't think this plan is going to work out too well for them. But given that since I left they've had to file bankruptcy, and in doing so involuntarily dropped to about half of what they were before (employees, assets, etc.), I guess this is just par for the course for them now.
A former Fortune 500 employer of mine went through the outsourcing boomerang. Now I understand that they have big plans to move everything to the cloud within the next two years. Knowing their systems like I do, and knowing that the CEO who is driving this is fairly noobish (note that I said CEO rather than CIO), I don't think this plan is going to work out too well for them. But given that since I left they've had to file bankruptcy, and in doing so involuntarily dropped to about half of what they were before (employees, assets, etc.), I guess this is just par for the course for them now.
Good luck being at someone else's mercy when the problems that are being held together by duct tape come to a head.
At the same time, I rather negotiate with a service provider than employees who have implemented their position into the system.
Your competency is eventually going to do the same and save money and have advantages over you if you don't take advantage of the awesome cost savings. Then you may cease being competitive and would have to close the data centers anyway.
As others said, you could retrain them.
As others said, you could retrain them.
I've never managed more people than myself, BUT don't underestimate how difficult it is to find people in IT. It's really, really hard, and even harder to find people willing to stay longer than 2 years.
Loyal tech workers are invaluable right now. But then again I'm not the one who has to make that choice.
Loyal tech workers are invaluable right now. But then again I'm not the one who has to make that choice.
Embracing automation and moving to cloud systems means layoffs for a dying/stagnating company and an opportunity engine for an innovating one. Our startup does GPU visual graph analytics for boosting visibility, and we see a world of a difference in velocity etc when we engage with a company that has cloud data lakes etc than one that doesn't. And, that includes F50s and federal agencies: super normal!
Technology employees should embrace new tools; that's part of the job that ensures compounding effects over time.
Technology employees should embrace new tools; that's part of the job that ensures compounding effects over time.
You obviously have a good heart for asking the question!
I have a feeling that whatever decisions you finally take would take into consideration the wellbeing of your employees.
I would assume that the folks who are doing the legacy stuff have the necessary aptitude to pick up the skills that are needed to automate for the future. Can their existing data center skills be translated into learning about AWS/Azure, VMs, Containerization and the necessary scripting and DSLs around those toolsets?
Obviously there will be reductions in headcount but give them the option to retool for the future. Even if they are not all retained but if you retrain them then they will still have options elsewhere with relevant skillsets.
You are finally running a business after all where revenue and profit are the drivers.
Good luck and I would say that your employees are luckier than those who consider them as mere resources and not people with lives and loved ones to take care of.
I have a feeling that whatever decisions you finally take would take into consideration the wellbeing of your employees.
I would assume that the folks who are doing the legacy stuff have the necessary aptitude to pick up the skills that are needed to automate for the future. Can their existing data center skills be translated into learning about AWS/Azure, VMs, Containerization and the necessary scripting and DSLs around those toolsets?
Obviously there will be reductions in headcount but give them the option to retool for the future. Even if they are not all retained but if you retrain them then they will still have options elsewhere with relevant skillsets.
You are finally running a business after all where revenue and profit are the drivers.
Good luck and I would say that your employees are luckier than those who consider them as mere resources and not people with lives and loved ones to take care of.
Just do it. Most old school sysadmins want to write their Perl and sh scripts not terraform and cloud formation scripts.
I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and SRE before most of the admins. Most of them left to go safe companies where they can stick to their old ways.
I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and SRE before most of the admins. Most of them left to go safe companies where they can stick to their old ways.
I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and SRE before most of the admins.
Yes, devs are often enthusiastic because they think they can ditch those boring old operations people who always stop them having fun.
Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff...
The truth is the need for ops people doesn't evaporate when you go cloud, be wary of anyone who says it does. This is the same dynamic that played out with MongoDB.
Yes, devs are often enthusiastic because they think they can ditch those boring old operations people who always stop them having fun.
Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff...
The truth is the need for ops people doesn't evaporate when you go cloud, be wary of anyone who says it does. This is the same dynamic that played out with MongoDB.
> Yes, devs are often enthusiastic because they think they can ditch those boring old operations people who always stop them having fun.
I think it's more subtle than that. (Also, I think the parent was trolling, but you knew that). It's that there's a certain class of programmers (and/or managers with exclusively programming backgrounds) that truly believe all (technology) problems can be solved with software.
If one assumes that belief, then the hostility to someone who believes differently is arguably inevitable.
It also explains why, to some, "Devops" is code for "wannabe dev" or some other coding-first role.
>Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff... >The truth is the need for ops people doesn't evaporate when you go cloud
That explains the dozens of seemingly-identical job postings from tech startups for "Devops" engineers with both experience in the above latest tools and operational experience. Fortunately, they only ask for 1-3 years, which is realistic.
> This is the same dynamic that played out with MongoDB.
Ultimately, though, did it matter? If we sometimes got fail whales instead of 140 characters, we still didn't get flying cars.
Alternatively, for whatever reason (freely flowing VC? low interest rates?), there's too little incentive for low cost or high reliability ops right now.
I think it's more subtle than that. (Also, I think the parent was trolling, but you knew that). It's that there's a certain class of programmers (and/or managers with exclusively programming backgrounds) that truly believe all (technology) problems can be solved with software.
If one assumes that belief, then the hostility to someone who believes differently is arguably inevitable.
It also explains why, to some, "Devops" is code for "wannabe dev" or some other coding-first role.
>Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff... >The truth is the need for ops people doesn't evaporate when you go cloud
That explains the dozens of seemingly-identical job postings from tech startups for "Devops" engineers with both experience in the above latest tools and operational experience. Fortunately, they only ask for 1-3 years, which is realistic.
> This is the same dynamic that played out with MongoDB.
Ultimately, though, did it matter? If we sometimes got fail whales instead of 140 characters, we still didn't get flying cars.
Alternatively, for whatever reason (freely flowing VC? low interest rates?), there's too little incentive for low cost or high reliability ops right now.
The new reality is that the barriers are gone. Ops must dev, dev must ops.
Who can get away with a job that just requires them to open a Java IDE? Those jobs are just as extinct as a pure ops role.
Who can get away with a job that just requires them to open a Java IDE? Those jobs are just as extinct as a pure ops role.
With MongoDB the devs said "we don't need a DBA anymore". But what they failed to understand was that the DBA was not the person who knew all the database command syntax, but rather the person who was responsible for the data - its availability, security, and so on. And getting rid of that person didn't get rid of the reason the job existed. I dare say that came as a bit of a shock. Well, serves them right for stabbing a fellow Worker in the back! And it is playing out exactly the same way in Cloud. All the responsibilities still exist, apart from the most basic ones such as "sweep the floor of the datacentre". Now who's going to take them?
I was an admin that learned to be a dev specifically so I could work on cloud stuff. I almost doubled my salary in 3 years.
That's the part that amazes me, just how much more companies pay for the cloud stuff, even on the labor side, yet startup executives (some of whom I've personally interviewed with) keep insisting it's cheaper.
If you are in a position to make decisions for that many people, is it a safe assumption that you have a fairly robust knowledge of how to enact organizational change? Because based on my limited knowledge, major change like this has processes to be worked through which will help answers some key questions -- what is the current and future state, who will gain/lose, what will your communication plan and transition plan be, etc.
You have much more knowledge to give those answers than we do. What to tell employees, and who can/not be saved from losing their job depends on those answers.
I will say that open communication, with complete answers of what people should expect, will help the change to happen without losing the trust of those who do get to stick around. You will have some harsh truths to share. If I worked for you, I would want to hear them as soon as there are enough answers to not leave me in limbo on how it affects me, personally. Not sooner, not later.
You have much more knowledge to give those answers than we do. What to tell employees, and who can/not be saved from losing their job depends on those answers.
I will say that open communication, with complete answers of what people should expect, will help the change to happen without losing the trust of those who do get to stick around. You will have some harsh truths to share. If I worked for you, I would want to hear them as soon as there are enough answers to not leave me in limbo on how it affects me, personally. Not sooner, not later.
I would second this comment and add that proper change management is needed to ensure a smooth transition. As soon as the word is out, if you don’t have a plan and/or don’t communicate in the right way, your people will start sending out resumes and that can wreck havoc in your business.
As you might expect, replacing top notch techies is quite hard. So managing the transition and making sure those that won’t be kept are better off is not only a good heartened approach but important to the success of the project.
Management consultants usually get a bed rep but that’s something they’re pretty good at. If you’re an F50, you have the budget to hire a top one.
As you might expect, replacing top notch techies is quite hard. So managing the transition and making sure those that won’t be kept are better off is not only a good heartened approach but important to the success of the project.
Management consultants usually get a bed rep but that’s something they’re pretty good at. If you’re an F50, you have the budget to hire a top one.
I have some experience in this area. My thoughts:
Retraining can be done as part of the project, but you need to figure out what the project and sub projects looks like first. Then you an know roughly how you’ll move people around at various phases. It takes time to build new systems and time to train. Line those up and you can take advantage of them.
Be careful with costs. They will blow out if you aren’t mindful.
It’s going to be messy for a LONG time, which causes a huge amount of pain in the org. Be ready for this.
Moving to cloud brings a whole new set of problems you haven’t thought of yet. For example, cleanup becomes more critical (old resources cost money, but can’t be “seen” because they’re not physical and just a hidden line item somewhere).
There’s more. My email is in my profile if you wish to reach out.
Retraining can be done as part of the project, but you need to figure out what the project and sub projects looks like first. Then you an know roughly how you’ll move people around at various phases. It takes time to build new systems and time to train. Line those up and you can take advantage of them.
Be careful with costs. They will blow out if you aren’t mindful.
It’s going to be messy for a LONG time, which causes a huge amount of pain in the org. Be ready for this.
Moving to cloud brings a whole new set of problems you haven’t thought of yet. For example, cleanup becomes more critical (old resources cost money, but can’t be “seen” because they’re not physical and just a hidden line item somewhere).
There’s more. My email is in my profile if you wish to reach out.
Cleanup is even a problem in smallish startups with some dozens off devs. I cannot imagine the amount of security groups a F50 must have after some years
What type of commercial providers are you saving money by moving to?
This happened to my friend in his 50s. He was a storage specialist working for a bank. Since they outsourced to IBM a few years ago he's been stacking shelves in a supermarket.
If you let yourself get that specialized then you're always going to be one or two innovations away from a layoff. I kind of feel that anyone who inhabits a niche like that for a long time is going to be in danger of becoming a victim of their own comfort seeking. Our industry is not kind to those who don't continually challenge themselves and learn new things. As a counter example to your friend's situation, I'll be turning 58 this november, and I'm an SRE/devops engineer working on kubernetes-based infrastructure in google cloud. Five or six years ago I was a back end .NET guy, a few years before that I was doing ASP web apps. Before that C++ and COM/DCOM stuff, and before that Orbix/CORBA, and before that... you get the picture. The old saw is that nothing stays the same, so if you're not moving with the world you can be sure it's moving past you.
I imagine it is going to take you a LONG time to do this. Some apps will migrate easily, others not so much. If you want to look after your people build a program of work in place that helps you realise immediate benefit while giving you the change to reskill people as you migrate to newer tech.
Jobs in the private sector do not come with lifetime guarantees. If you have new jobs you could train them to do, great. Otherwise, give the people you will lay off notice and severance, and let them figure out what to do.
To any companies wondering why their employees jump ship at the drop of a hat, or why there's just no loyalty anymore, this attitude is it.
Sure, but it's a cycle that is easy to fall into and hard to break now that we're here. Distrust begets distrust.
Then those with the power, the employers, can work to earn that trust.
This can't be over-emphasized, that, especially with no union, the power dynamic is immensely unbalanced. Even just the information assymetry is huge, since, despite recent innovations like Glassdoor, workers know very little about even each other's circumstances.
The parent's comment use of the word "cycle" implies, by association with phrases like "cycle of violence", that there might be some equality or that both parties may be victims to some degree.
In the face of such assymmetry, such a claim would be extraordinary, requiring a strong argument, more than just an implication.
The parent's comment use of the word "cycle" implies, by association with phrases like "cycle of violence", that there might be some equality or that both parties may be victims to some degree.
In the face of such assymmetry, such a claim would be extraordinary, requiring a strong argument, more than just an implication.
No. As an engineer myself, this is exactly what I'm working to avoid. I know we could terminate, but these people have built careers and have deep knowledge of the systems and business, I'm not letting them go without at least giving them the choice to retrain.
Out of curiosity, what does F50 refer to here? Google doesn’t help, nor does https://en.m.wikipedia.org/wiki/F50 .
It refers to the 50 biggest companies in the https://en.wikipedia.org/wiki/Fortune_500
Fortune 50, I assume.
You should do the move. Sysadmin is now a niche discipline, people work in devops now. The tech economy is good and their skills are getting more obsolete.
Just start slowly migrating things, and train them in devops skills. Then, even if you have a reduction in a year, they will have a year worth of modern skills in EC2 devops and the like.
It's going to happen, so better it happens now than in the future.
Just start slowly migrating things, and train them in devops skills. Then, even if you have a reduction in a year, they will have a year worth of modern skills in EC2 devops and the like.
It's going to happen, so better it happens now than in the future.
Kudos for keeping your sights on helping your team! I'm a big fan of holding on to a group of people that understands your business (I'd rather retrain a technology than instill passion and domain experience).
Not 100% relevant (and not my better writing) but here are some of the things I've seen companies do in terms of moving people around from NOCs that might give you some ideas https://www.pagerduty.com/blog/future-of-noc/
If you'd like a soundingboard to brainstorm with feel free to ping me at [email protected]
Not 100% relevant (and not my better writing) but here are some of the things I've seen companies do in terms of moving people around from NOCs that might give you some ideas https://www.pagerduty.com/blog/future-of-noc/
If you'd like a soundingboard to brainstorm with feel free to ping me at [email protected]
Since it sounds like you're going to save a bunch of money, I suggest letting them go and using that money to provide very generous severance packages (months of salary at least). Then they can choose to retrain themselves, take some time off, whatever. It works for Netflix.
I would build your own cloud. You have customers. Don’t give them away.
Maybe this is unrealistic somehow, but what about paying the difference in cost directly to the affected employees?
There are a lot of benefits to migrating to the cloud, in my experience, cost isn’t one of them. Unless of course you are mucking around with accounting tricks by cutting employees and moving that cost to services or cost of goods sold, I’m not a expert in this, but it works and will save you money because you can control these expenses more easily and because of how payroll vs services are accounted for.
You should aim to keep 50% of your workforce. 25% won’t want to change 10-25% won’t be able to. I would invest in training on automation and multicloud / hybrid cloud management. I recently worked on a large scale migration and you will definitely discover that the cloud wont work for a lot of your high performance computing, or it will be cost prohibitive. You also need to take into consideration intellectual property, does the company really want all of its IP on another conglomerates infrastructure? But most importantly you need to avoid lock-in to one provider at all cost, training your trusted resources who understand the business requirements on hybrid cloud or multi cloud implementation will really enable cost savings because you will have leverage during price negotiations. If you get locked into one provider you are going to get royally screwed on long term cost, seen it happen.
You should aim to keep 50% of your workforce. 25% won’t want to change 10-25% won’t be able to. I would invest in training on automation and multicloud / hybrid cloud management. I recently worked on a large scale migration and you will definitely discover that the cloud wont work for a lot of your high performance computing, or it will be cost prohibitive. You also need to take into consideration intellectual property, does the company really want all of its IP on another conglomerates infrastructure? But most importantly you need to avoid lock-in to one provider at all cost, training your trusted resources who understand the business requirements on hybrid cloud or multi cloud implementation will really enable cost savings because you will have leverage during price negotiations. If you get locked into one provider you are going to get royally screwed on long term cost, seen it happen.
Disclosure: I work for Google Cloud (and I assume/hope someone at Google is competing for your business).
Some of those folks happily do updated things, but others (like people that rack servers) won’t have an obvious new role in a cloud focused world.
My (now) colleague at Nic Harteau gave a talk at our Leaders’ Circle event talking about precisely this in Spotify’s move to Cloud: how could he keep the infrastructure team as they moved to cloud? As it turned out, most of them happily adjusted what they were doing because the mission was the same: serve the product teams / groups that need infrastructure from them. Sure, now you don’t configure a physical firewall, but there are still firewalls in cloud. In a sense, they’re even more important!
I believe Nic said only one person from the group left. The engineer was focused on building their own SDN stack on-prem. That job, and other “things the provider completely handles” do go away. But that person likely would have been massively useful in helping guide how Spotify should use Google Cloud’s own SDN setup. There are lots of ways to architect how services interact in clouds, and a lot of it comes down to funny networking.
So I’d say more than you might think, but please don’t mislead them.
Some of those folks happily do updated things, but others (like people that rack servers) won’t have an obvious new role in a cloud focused world.
My (now) colleague at Nic Harteau gave a talk at our Leaders’ Circle event talking about precisely this in Spotify’s move to Cloud: how could he keep the infrastructure team as they moved to cloud? As it turned out, most of them happily adjusted what they were doing because the mission was the same: serve the product teams / groups that need infrastructure from them. Sure, now you don’t configure a physical firewall, but there are still firewalls in cloud. In a sense, they’re even more important!
I believe Nic said only one person from the group left. The engineer was focused on building their own SDN stack on-prem. That job, and other “things the provider completely handles” do go away. But that person likely would have been massively useful in helping guide how Spotify should use Google Cloud’s own SDN setup. There are lots of ways to architect how services interact in clouds, and a lot of it comes down to funny networking.
So I’d say more than you might think, but please don’t mislead them.
Why not automate your stuff at your current datacenter?
First path is to retrain them. Getting more cost efficient can coincide with overall corporate growth.
Some people prefer to remain in their specialty or struggle to make the leap. In that case if they have tech skills, someone will take them. It may require a move, but they will land. It's one thing if you're firing people in the dark days of a deep recession. (Folks let go in 2001 had to leave California) It's another thing if you're letting people go into a market of 4% unemployment. If they can't get jobs with similar pay, then you were overpaying them.
Some people prefer to remain in their specialty or struggle to make the leap. In that case if they have tech skills, someone will take them. It may require a move, but they will land. It's one thing if you're firing people in the dark days of a deep recession. (Folks let go in 2001 had to leave California) It's another thing if you're letting people go into a market of 4% unemployment. If they can't get jobs with similar pay, then you were overpaying them.
Tech jobs exist because of automation and they eventually go away because automation. But every dollar saved in salary is available to spend on hiring people to do other jobs. It works out in the long run, but that doesn't make it easy on those who have to transition.
I think it would be useful if employers eliminating jobs could have sort of a reverse auction where other employers offer comparable jobs in exchange for a portion of the savings. And the workers could either take their severance or accept one of those offers with no interruption in their benefits or work.
I think it would be useful if employers eliminating jobs could have sort of a reverse auction where other employers offer comparable jobs in exchange for a portion of the savings. And the workers could either take their severance or accept one of those offers with no interruption in their benefits or work.
Cloud/Comerical <> automation.
I have been burned in the past by "commercial", thankfully I wasn't the guy who had to go to the board hat in hand and say "this isn't in our control"...
Every comercial provider is going to do everything in their power to "lock you in" they are going to put up features that are "better enough" that you migrate to them and then they own you.
Automate, reduce your workforce as needed (it will be less than your plan) and then look at the commercial provider as a second step, look at them in relation to a real DR plan. You might not like what you see for day to day operations but having them on stand by in the even of an emergency could make you a hero and present a massive cost savings already.
I have been burned in the past by "commercial", thankfully I wasn't the guy who had to go to the board hat in hand and say "this isn't in our control"...
Every comercial provider is going to do everything in their power to "lock you in" they are going to put up features that are "better enough" that you migrate to them and then they own you.
Automate, reduce your workforce as needed (it will be less than your plan) and then look at the commercial provider as a second step, look at them in relation to a real DR plan. You might not like what you see for day to day operations but having them on stand by in the even of an emergency could make you a hero and present a massive cost savings already.
[deleted]
Saying first what I want you to see the most: please add a throwaway email to your bio profile (and maybe add a new comment with it so more see it). I guarantee people will want to say hi privately - not [just] vendors but random people who have on-the-ground advice.
--
Hopefully you still see this even though it's been a few hours.
Glad I went through the thread: there was quite a bit of extra narrative and detail in your comment replies.
> Our business is an old, traditional business that happens to have lots of data centers for legacy reasons. Even though we spend $BBB on data centers, they're not the core of the business and a very small part of overall revenue. Realistically, even if we didn't move from data centers we would be good as long as the core non-tech business functioned.
> Thousands [of engineers], many close to retirement age. Think a company like UPS/Fedex that isn't at its core a tech company but with tech datacenters for logistics and billions in revenue.
> [T]he cost savings hinge on the massive cost savings from moving to a commercial cloud, laying off workforce would save more, but would prefer to save a bunch by moving to cloud and keep the workforce for other things.
--
Ok, here's my take.
I wonder what your [job] position is, heh. Maybe someone with a CTO's ear, or maybe you're an unusually empathetic CTO.
In any case you're clearly at the point where you've had the internal discussions, the trigger has been assembled and is ready to pull, and you and/or the team (committee?) has probably already been groomed by all the gigantic providers' sales teams, which of course pull out all the stops in your case. If I understand these things correctly, if you haven't yet had an amazing lunch or two on one or more vendors' dimes, you will :) (or at least the high-level people will, would be nice if you get included).
Sales is important. But always remember, particularly for large potential accounts like yours, the sales numbers will always be what the high-level execs want them to be, and the sales teams will be spinning the story the execs want to hear. The on-the-ground experience is always going to be rough. You've probably experienced similar transitions, and that's what's informing your skepticism some of your team won't have jobs. Nice you're giving your people some thought.
On the one hand, it doesn't really matter who you sign up with, as vendor ceases to matter somewhat short of the $B mark.
On the other hand, try to mitigate the risk of discovering that a vendor can't provide a given feature at some eleventh hour, or you may end up in a confused tangle of AWS and GCE deployments (because the other major vendor will of course have the missing feature, and the fragmentation will begin).
How I'd achieve this: every time you go "ok, so this is our implementation blueprint", start the rounds of internal technical enumeration over again. Spend time with the engineers to understand pain points. Talk to as many people on the ground as possible, especially the forgotten people in the corners. One-on-ones, or small groups of people who are known to gel well with each other (no impedance mismatches, work well together), could be good for extricating annoying gotchas. This could look like devs aimlessly paging through source code and saying what they think (this could catch tiny things well), or it might be more structured with general architectural discussion.
Do NOT have the cloud provider vendors do this work (of talking to your existing team); I wouldn't be surprised if they offer to do it, but if they do, all their feedback ultimately goes to the sales guys (via an internal engineer digestion process), and not directly to you.
Well, the analysis team will probably generate a high-level report that contains all the information they've discovered (this is probably legally mandated for full disclosure about what they now know), but all the on-the-ground details about inefficiencies, vendor-specific pain points and so forth will only be derived by vendor-internal engineering debriefs, and you'll naturally never get any of this particularly valuable information - unfortunately this approach is within the vendors' business focus, as inefficiency means more compute resources used ($), and pain points mean more vendor-infrastructure maintenance ($). You could respin this statement to say that the vendor who makes the most attractive offer is the one who knows their system is the worst fit for your system, but I don't know :)
Instead, if you don't have the internal resources (or freshness/objectivity) to put together a bird's-eye view, get an outside consultancy to help, directing them to investigate your current architecture top-to-bottom-and-back-to-top-again, and advise on ways the system can be rearchitected for efficiency and optimization within modern cloud infrastructure. (They'd then be the ones doing the one-on-ones and so forth I mentioned above.)
Another thing that may be extremely useful would be to take the "we're moving to the cloud traction" and carefully smudge out (some of) the dividing line between "we're overhauling everything to migrate it and make it work most efficiently within $vendor" and "we're rearchitecting major components so they work more efficiently overall" so the line between these two sentiments is blurred, and both things wind up happening in lockstep. Obviously the two ideas are intrinsically distinct and this can only be taken so far - "rewrite everything from scratch to get it running in the cloud" wouldn't pass any accounting costing or time-boxing :) - but you can probably push this far enough to extract some usefulness from it (eg, you probably have an idea how long upper level will be willing to wait for this transition to finish - 2 years? 1 year? 6 months? - and you might be able to squeeze some interesting and useful tidy-up operations into that window).
Blurring the lines between these two things will play to your existing devs' strengths, as rewriting/restructuring is easier on the brain than the "replace engine in flight" of picking up all the existing state, punting it into to the cloud ad-hoc, and running after all the resulting fires that get created and scrambling to get it all working perfectly.
It won't be a perfect transition if you're really a $BBB operation, and restructuring will be the easiest way to cohesively get the existing devs into a flexible frame of mind and ready to deal with whatever curveballs get thrown their way.
This is one way you could efficiently achieve keeping a lot of your existing team.
Plus, restructuring means everything can slowly be built up onto the new system; picking everything up at once and expecting it to land on its feet probably won't go very well.
Depending on how things go, if you bring in new techs to assist with understanding modern best practice/hitting the ground running, I cannot recommend enough pitting them against your current team when doing onboarding/interviewing, as they'd be an EXCELLENT catalyst for judging candidates' character.
These are a couple of lateral ways the current workforce could be creatively employed during this time. I reckon careful management would see everyone kept on until retirement.
The current team knows all the systems inside out. Letting them go would make everything a thousand times worse.
Also, another reason I thought it would be cool to put an email is because your post makes for a perhaps-unwitting but nonetheless effective and interesting recruitment drive; I'm aware of very few (okay, zero) high-level people at equally high-level companies who really care about their employees. In my case this is because I don't have any connections, but the same probably applies to many others.
I don't know why https://news.ycombinator.com/item?id=16283252 stood out to me a few months ago, but your post resonates similarly. Probably because it incorporates a similar human response too. (Note that I have no association with the potential companies listed and have never talked to any of the users in that thread, I just thought it was interesting)
--
Hopefully you still see this even though it's been a few hours.
Glad I went through the thread: there was quite a bit of extra narrative and detail in your comment replies.
> Our business is an old, traditional business that happens to have lots of data centers for legacy reasons. Even though we spend $BBB on data centers, they're not the core of the business and a very small part of overall revenue. Realistically, even if we didn't move from data centers we would be good as long as the core non-tech business functioned.
> Thousands [of engineers], many close to retirement age. Think a company like UPS/Fedex that isn't at its core a tech company but with tech datacenters for logistics and billions in revenue.
> [T]he cost savings hinge on the massive cost savings from moving to a commercial cloud, laying off workforce would save more, but would prefer to save a bunch by moving to cloud and keep the workforce for other things.
--
Ok, here's my take.
I wonder what your [job] position is, heh. Maybe someone with a CTO's ear, or maybe you're an unusually empathetic CTO.
In any case you're clearly at the point where you've had the internal discussions, the trigger has been assembled and is ready to pull, and you and/or the team (committee?) has probably already been groomed by all the gigantic providers' sales teams, which of course pull out all the stops in your case. If I understand these things correctly, if you haven't yet had an amazing lunch or two on one or more vendors' dimes, you will :) (or at least the high-level people will, would be nice if you get included).
Sales is important. But always remember, particularly for large potential accounts like yours, the sales numbers will always be what the high-level execs want them to be, and the sales teams will be spinning the story the execs want to hear. The on-the-ground experience is always going to be rough. You've probably experienced similar transitions, and that's what's informing your skepticism some of your team won't have jobs. Nice you're giving your people some thought.
On the one hand, it doesn't really matter who you sign up with, as vendor ceases to matter somewhat short of the $B mark.
On the other hand, try to mitigate the risk of discovering that a vendor can't provide a given feature at some eleventh hour, or you may end up in a confused tangle of AWS and GCE deployments (because the other major vendor will of course have the missing feature, and the fragmentation will begin).
How I'd achieve this: every time you go "ok, so this is our implementation blueprint", start the rounds of internal technical enumeration over again. Spend time with the engineers to understand pain points. Talk to as many people on the ground as possible, especially the forgotten people in the corners. One-on-ones, or small groups of people who are known to gel well with each other (no impedance mismatches, work well together), could be good for extricating annoying gotchas. This could look like devs aimlessly paging through source code and saying what they think (this could catch tiny things well), or it might be more structured with general architectural discussion.
Do NOT have the cloud provider vendors do this work (of talking to your existing team); I wouldn't be surprised if they offer to do it, but if they do, all their feedback ultimately goes to the sales guys (via an internal engineer digestion process), and not directly to you.
Well, the analysis team will probably generate a high-level report that contains all the information they've discovered (this is probably legally mandated for full disclosure about what they now know), but all the on-the-ground details about inefficiencies, vendor-specific pain points and so forth will only be derived by vendor-internal engineering debriefs, and you'll naturally never get any of this particularly valuable information - unfortunately this approach is within the vendors' business focus, as inefficiency means more compute resources used ($), and pain points mean more vendor-infrastructure maintenance ($). You could respin this statement to say that the vendor who makes the most attractive offer is the one who knows their system is the worst fit for your system, but I don't know :)
Instead, if you don't have the internal resources (or freshness/objectivity) to put together a bird's-eye view, get an outside consultancy to help, directing them to investigate your current architecture top-to-bottom-and-back-to-top-again, and advise on ways the system can be rearchitected for efficiency and optimization within modern cloud infrastructure. (They'd then be the ones doing the one-on-ones and so forth I mentioned above.)
Another thing that may be extremely useful would be to take the "we're moving to the cloud traction" and carefully smudge out (some of) the dividing line between "we're overhauling everything to migrate it and make it work most efficiently within $vendor" and "we're rearchitecting major components so they work more efficiently overall" so the line between these two sentiments is blurred, and both things wind up happening in lockstep. Obviously the two ideas are intrinsically distinct and this can only be taken so far - "rewrite everything from scratch to get it running in the cloud" wouldn't pass any accounting costing or time-boxing :) - but you can probably push this far enough to extract some usefulness from it (eg, you probably have an idea how long upper level will be willing to wait for this transition to finish - 2 years? 1 year? 6 months? - and you might be able to squeeze some interesting and useful tidy-up operations into that window).
Blurring the lines between these two things will play to your existing devs' strengths, as rewriting/restructuring is easier on the brain than the "replace engine in flight" of picking up all the existing state, punting it into to the cloud ad-hoc, and running after all the resulting fires that get created and scrambling to get it all working perfectly.
It won't be a perfect transition if you're really a $BBB operation, and restructuring will be the easiest way to cohesively get the existing devs into a flexible frame of mind and ready to deal with whatever curveballs get thrown their way.
This is one way you could efficiently achieve keeping a lot of your existing team.
Plus, restructuring means everything can slowly be built up onto the new system; picking everything up at once and expecting it to land on its feet probably won't go very well.
Depending on how things go, if you bring in new techs to assist with understanding modern best practice/hitting the ground running, I cannot recommend enough pitting them against your current team when doing onboarding/interviewing, as they'd be an EXCELLENT catalyst for judging candidates' character.
These are a couple of lateral ways the current workforce could be creatively employed during this time. I reckon careful management would see everyone kept on until retirement.
The current team knows all the systems inside out. Letting them go would make everything a thousand times worse.
Also, another reason I thought it would be cool to put an email is because your post makes for a perhaps-unwitting but nonetheless effective and interesting recruitment drive; I'm aware of very few (okay, zero) high-level people at equally high-level companies who really care about their employees. In my case this is because I don't have any connections, but the same probably applies to many others.
I don't know why https://news.ycombinator.com/item?id=16283252 stood out to me a few months ago, but your post resonates similarly. Probably because it incorporates a similar human response too. (Note that I have no association with the potential companies listed and have never talked to any of the users in that thread, I just thought it was interesting)
These are really helpful insights. Thank you. Position wise yes, high level newly-minted CTO-ish-type office sounds about right, sorry to be vague. Decision is made, implementation and vendor selection is the hard part. I have already moved many applications as proof of concept and the savings seen is one driver of this change, and me being the one that moved them is one reason I get to be involved/doing/advising/engineering this. We are not FAANG and not located in cool locales so harder to attract great talent.
The cloud vendors are in fact already offering to take on a lot of this work of transitioning, which I am avoiding to avoid lock in at all costs.
And for others reading it's not UPS/Fedex just to throw that out there, I saw another thread on UPS yesterday trying to move huge organization's ops to 21st century and was trying to think of examples of what I assumed were large legacy companies that are not core tech and those popped to mind. That thread also served as a catalyst as I'm also wrestling with moving a huge orgs ops to the 21st century.
I have taken your advice on email: [email protected]
The cloud vendors are in fact already offering to take on a lot of this work of transitioning, which I am avoiding to avoid lock in at all costs.
And for others reading it's not UPS/Fedex just to throw that out there, I saw another thread on UPS yesterday trying to move huge organization's ops to 21st century and was trying to think of examples of what I assumed were large legacy companies that are not core tech and those popped to mind. That thread also served as a catalyst as I'm also wrestling with moving a huge orgs ops to the 21st century.
I have taken your advice on email: [email protected]
> Decision is made, implementation and vendor selection is the hard part.
>The cloud vendors are in fact already offering to take on a lot of this work of transitioning, which I am avoiding to avoid lock in at all costs.
This gives you a distinct advantage, in that you could, subsequently, move back and be your own "private cloud" vendor for even further cost savings.
> We are not FAANG and not located in cool locales so harder to attract great talent.
You may find that becomes less true as housing prices in cool locales continue to rise. Alternatively, you could open offices in those locales, since, if your infrastructure is going to be cloud-based (public, private, or hybrid), it's less important that the staff be co-located with it.
There's also the notion of a remote-first engineering culture that I'm sure you can find plenty of evangelism for, just by searching.
This gives you a distinct advantage, in that you could, subsequently, move back and be your own "private cloud" vendor for even further cost savings.
> We are not FAANG and not located in cool locales so harder to attract great talent.
You may find that becomes less true as housing prices in cool locales continue to rise. Alternatively, you could open offices in those locales, since, if your infrastructure is going to be cloud-based (public, private, or hybrid), it's less important that the staff be co-located with it.
There's also the notion of a remote-first engineering culture that I'm sure you can find plenty of evangelism for, just by searching.
Of the apps you have already moved, are any of them in the range of poorly documented, duct taped together, you know, the nightmare apps that I’m sure you have there? Just to be sure you’ve established both ends of the costs-to-savings spectrum. It may be the most cost effective approach is to outsource some apps and keep others on premise.
Wow, I must admit that my original comment was written from the position of "now how would this kind of thing happen in real life" and involved a rather heavy-handed dose of armchair speculation. The various details I mentioned (CTO-type position; that the decision is made; that the vendors have offered to help) were all "well it would probably happen this way" postulations. Ha. (Did any lunches happen yet?)
Vagueness is fine and important. Oh, and congrats on the newly-minted position :)
TIL what FAANG is ("Facebook, Apple, Amazon, Netflix and Google"). I see. Well, maybe posting in the monthly "Who is hiring?" threads could be helpful. (Reading through previous hiring threads is probably the best way to get an idea of what+how to post: https://hn.algolia.com/?query="ask%20hn:%20who%20is%20hiring...)
Who is hiring? is all semi-anonymously done, of course; what goes into the post is rough location, perks, job focus (an interesting question), benefits, etc. (Heh, I'd just link to this thread in the "benefits" section - then applicants would know they were applying to somewhere that looks like it actually respects what puts the intelligence behind "artificial intelligence" :) there's not enough of that out there IIUC, see eg http://rachelbythebay.com/w/2018/04/17/company/).
> The cloud vendors are in fact already offering to take on a lot of this work of transitioning, which I am avoiding to avoid lock in at all costs.
Well, for what it's worth, it is a necessary step. a) you're switching sysadmin+ops+networking+infra teams, so the replacement team does kind of need to get an idea what they're up against; and b) there's a reasonable chance that your deployment will give $vendor a run for its money and require small bits of build-out (probably edge-case handling) to more smoothly support your use case(s). This was why I mentioned the analysis teams would be taking their insights back to HQ for debriefing (cue internal mad scramble :P), but didn't manage to articulate that in my previous post.
Hmmmm, on that note, that gives me an idea: maybe port one or two of your craziest components - the ones that are the hardest to wield and integrate and _will_ have problems - to both vendors, and see what the customer support response is like. Ah, wait, that won't work, you won't be able to hide which account you're calling with problems about and the sales teams will make sure support pulls out all the stops. Of course once you actually sign up with a $vendor, you're a captive audience and will be subject to all the bureaucratic rainbow tape.
Ah, so this is one motivation to sign up with both vendors: to avoid lockin so they still have to work hard for your business.
So THAT'S why vendors work so hard for exclusivity. Hah! I suddenly understand a new reason why there's such a big reason to get vendor logos onto webpages, it's not just advertising for the vendor but because it almost certainly represents lockin - because it would be politically incorrect for a newly-cloud-ified company to say "we went with GCE and AWS!", wouldn't it? That makes it look like one provider isn't enough - when in reality, one vendor would probably provide all requirements, but once they're "it", they've got the contract and can stop working. (Something something psuedo-strawman argument...)
Hmmm. So that's a bit of the risk balance: go with one vendor and risk lockin and being a captive customer, or go with both, get slightly shunned ("we don't want your logo on our webpage, and we don't want you putting our logo and $competitor's on yours") and have a bit more control because both vendors are competing. (So then the shunning would be mixed with wooing. That would be interesting to watch. Why am I imagining buying popcorn...)
(This stream of consciousness approach was unintentional but I don't think I can express any better by refactoring the presentation)
Hm, another thing I thought of: if your existing workforce is big enough (at least 50-100 people? {EDIT: oh, thousands, this would work amazingly then}) and everyone's fairly isolated/segregated, you could take advantage of the segregation by getting everyone {or maybe many} to explain their systems to everyone else {or many others} - maybe explain to multiple people with decreasing levels of familiarity.
This would mostly be an exercise in "rubber duck debugging", with the decreasing levels of familiarity forcing people to articulate better.
This approach kills two birds with one stone: not only does it mean people have properly articulated their systems (and found lots of little edge cases), at the end of such a venture, everyone would know a lot lot more about everyone else's stacks. This in turn would have two benefits, a) people would better understand what the components they manage are talking to, and b) overall agility would increase as 1) remote teams could take simple steps to fix remote fires and 2) devs would better empathise with what they were interacting with, which will produce a more cohesive system overall.
Now I'll go reply in the thread where you explain current costs.
Wonder if you've gotten any emails from anyone yet :) I'll try and follow up w/ a ping hopefully soonish (might take me longer than I'd prefer but it'll happen).
(Also, the F5 key on my laptop is far too easy to hit... when this happens, I quickly open Chrome's task manager, SIGSTOP the renderer PID for the tab I'm in, gdb -p <pid>, generate-core-file, then grep my post back out. This is the 2nd time I've done this and it works absolutely great because HN uses <textarea>s :D. CTRL+W is harder and either requires a hasty drop into hibernation, or SIGSTOPping all the renderers then generating+grepping core files one at a time. Good luck if the ^W killed the renderer and the memory is now unallocated. Goes and installs form saver extension)
Vagueness is fine and important. Oh, and congrats on the newly-minted position :)
TIL what FAANG is ("Facebook, Apple, Amazon, Netflix and Google"). I see. Well, maybe posting in the monthly "Who is hiring?" threads could be helpful. (Reading through previous hiring threads is probably the best way to get an idea of what+how to post: https://hn.algolia.com/?query="ask%20hn:%20who%20is%20hiring...)
Who is hiring? is all semi-anonymously done, of course; what goes into the post is rough location, perks, job focus (an interesting question), benefits, etc. (Heh, I'd just link to this thread in the "benefits" section - then applicants would know they were applying to somewhere that looks like it actually respects what puts the intelligence behind "artificial intelligence" :) there's not enough of that out there IIUC, see eg http://rachelbythebay.com/w/2018/04/17/company/).
> The cloud vendors are in fact already offering to take on a lot of this work of transitioning, which I am avoiding to avoid lock in at all costs.
Well, for what it's worth, it is a necessary step. a) you're switching sysadmin+ops+networking+infra teams, so the replacement team does kind of need to get an idea what they're up against; and b) there's a reasonable chance that your deployment will give $vendor a run for its money and require small bits of build-out (probably edge-case handling) to more smoothly support your use case(s). This was why I mentioned the analysis teams would be taking their insights back to HQ for debriefing (cue internal mad scramble :P), but didn't manage to articulate that in my previous post.
Hmmmm, on that note, that gives me an idea: maybe port one or two of your craziest components - the ones that are the hardest to wield and integrate and _will_ have problems - to both vendors, and see what the customer support response is like. Ah, wait, that won't work, you won't be able to hide which account you're calling with problems about and the sales teams will make sure support pulls out all the stops. Of course once you actually sign up with a $vendor, you're a captive audience and will be subject to all the bureaucratic rainbow tape.
Ah, so this is one motivation to sign up with both vendors: to avoid lockin so they still have to work hard for your business.
So THAT'S why vendors work so hard for exclusivity. Hah! I suddenly understand a new reason why there's such a big reason to get vendor logos onto webpages, it's not just advertising for the vendor but because it almost certainly represents lockin - because it would be politically incorrect for a newly-cloud-ified company to say "we went with GCE and AWS!", wouldn't it? That makes it look like one provider isn't enough - when in reality, one vendor would probably provide all requirements, but once they're "it", they've got the contract and can stop working. (Something something psuedo-strawman argument...)
Hmmm. So that's a bit of the risk balance: go with one vendor and risk lockin and being a captive customer, or go with both, get slightly shunned ("we don't want your logo on our webpage, and we don't want you putting our logo and $competitor's on yours") and have a bit more control because both vendors are competing. (So then the shunning would be mixed with wooing. That would be interesting to watch. Why am I imagining buying popcorn...)
(This stream of consciousness approach was unintentional but I don't think I can express any better by refactoring the presentation)
Hm, another thing I thought of: if your existing workforce is big enough (at least 50-100 people? {EDIT: oh, thousands, this would work amazingly then}) and everyone's fairly isolated/segregated, you could take advantage of the segregation by getting everyone {or maybe many} to explain their systems to everyone else {or many others} - maybe explain to multiple people with decreasing levels of familiarity.
This would mostly be an exercise in "rubber duck debugging", with the decreasing levels of familiarity forcing people to articulate better.
This approach kills two birds with one stone: not only does it mean people have properly articulated their systems (and found lots of little edge cases), at the end of such a venture, everyone would know a lot lot more about everyone else's stacks. This in turn would have two benefits, a) people would better understand what the components they manage are talking to, and b) overall agility would increase as 1) remote teams could take simple steps to fix remote fires and 2) devs would better empathise with what they were interacting with, which will produce a more cohesive system overall.
Now I'll go reply in the thread where you explain current costs.
Wonder if you've gotten any emails from anyone yet :) I'll try and follow up w/ a ping hopefully soonish (might take me longer than I'd prefer but it'll happen).
(Also, the F5 key on my laptop is far too easy to hit... when this happens, I quickly open Chrome's task manager, SIGSTOP the renderer PID for the tab I'm in, gdb -p <pid>, generate-core-file, then grep my post back out. This is the 2nd time I've done this and it works absolutely great because HN uses <textarea>s :D. CTRL+W is harder and either requires a hasty drop into hibernation, or SIGSTOPping all the renderers then generating+grepping core files one at a time. Good luck if the ^W killed the renderer and the memory is now unallocated. Goes and installs form saver extension)
Your people shouldn't be doing manual config regardless of if you're on-prem or cloud. If they are, you're going to have a tough transition to cloud, and you're still going to need a lot more of those people than you think you will.
Hate to do this, but be mindful people bearish on the benefits of moving in the first place will be downvoted crazily, potentially by people who have an economic incentive to convince you this is a good idea.
Hire for attitude, Train for skills - If you've good dedicated people that you can trust, I'm sure they can be trained for other positions.
Good people are always hard to come by, and they will be even more loyal if they will see that the company doesn't ditch them.
We’re in the process of moving our infrastructure to Azure. Being the public sector we take things relatively slow, but we’ve already fully embraced Azure ad as well as office 365, one drive for business, planner and all those other things.
The next step is our servers, they are already virtual and hosted in our own cloud, but moving our database cluster to a commercial cloud is still a big step politically.
Anyway, you retrain your employees. You may need fewer in the long run, but finding people who are actually professional at stuff like setting up on-premise security inside Azure hasn’t been easy for us. We’ve had 3 very highly regarded consultant agencies to help us, and they quite honestly didn’t know anything but tech-book Azure. More than once we’ve had to send consultants home because it became obvious they knew less about this stuff than our people.
Azure has also opened up for a range of automation through services, but that’s not really automatic either. We’ve had to retrain our people in powershell, and now we’re running a lot of stuff through the orchestra service. So right now, we’ve actually not needed fewer people, just better trained ones.
I image you could replace Azure with Aws or any such service and it be the same story.
The next step is our servers, they are already virtual and hosted in our own cloud, but moving our database cluster to a commercial cloud is still a big step politically.
Anyway, you retrain your employees. You may need fewer in the long run, but finding people who are actually professional at stuff like setting up on-premise security inside Azure hasn’t been easy for us. We’ve had 3 very highly regarded consultant agencies to help us, and they quite honestly didn’t know anything but tech-book Azure. More than once we’ve had to send consultants home because it became obvious they knew less about this stuff than our people.
Azure has also opened up for a range of automation through services, but that’s not really automatic either. We’ve had to retrain our people in powershell, and now we’re running a lot of stuff through the orchestra service. So right now, we’ve actually not needed fewer people, just better trained ones.
I image you could replace Azure with Aws or any such service and it be the same story.
If you don't mind me asking - why Azure specifically?
We’re the public sector, we operate more than 500 it systems, some of which won’t be leaving the windows platform.
We run office 365, Skype for business and handle our Security through af and adfs for 10.000 employees.
So we’re already heavily invested in Microsoft as far as what our IT staff is trained in, and they prefer Azure to say AWS often for non-technical reasons, but that sort of motivation is still important from a management point of view, even if you can run windows in AWS.
That’s one reason. The other factors is that we’re in the EU and Microsoft were the first to implement our laws, without fighting them. Unlike AWS I can physically visit our Azure servers, though this requires you to throw a lot of money at Microsoft. Last but not least, Microsoft is a trusted partner that we’ve worked with for 25 years, Amazon is not.
There weren’t any other options than AWS and Microsoft, because we’re in the eu and in the public sector, and AWS wasn’t under the privacy laws at the time we started.
A lot of our neighbors are choosing AWS though, so it’s certainly viable today.
We run office 365, Skype for business and handle our Security through af and adfs for 10.000 employees.
So we’re already heavily invested in Microsoft as far as what our IT staff is trained in, and they prefer Azure to say AWS often for non-technical reasons, but that sort of motivation is still important from a management point of view, even if you can run windows in AWS.
That’s one reason. The other factors is that we’re in the EU and Microsoft were the first to implement our laws, without fighting them. Unlike AWS I can physically visit our Azure servers, though this requires you to throw a lot of money at Microsoft. Last but not least, Microsoft is a trusted partner that we’ve worked with for 25 years, Amazon is not.
There weren’t any other options than AWS and Microsoft, because we’re in the eu and in the public sector, and AWS wasn’t under the privacy laws at the time we started.
A lot of our neighbors are choosing AWS though, so it’s certainly viable today.
Microsoft throws the moneys at you and "owns" places. They marched into our company as if they owned the place. They have all the selling power and all the money to "convince" you. And yes, contracts are time limited so the big squeeze will be coming eventually.
I would guess they are a Microsoft Shop.
Azure seems to have a compelling platform for Microsoft Shops. I don't see the allure for a Linux shop. Sure you can run Linux on Azure, just like you can run Windows on AWS... but I don't think either is optimal.
Azure seems to have a compelling platform for Microsoft Shops. I don't see the allure for a Linux shop. Sure you can run Linux on Azure, just like you can run Windows on AWS... but I don't think either is optimal.
You're taking the marketing approach as opposed to coming up with real solutions -- think critically about the business need, investigate the cost savings and make a transition plan for those who actually add value to the organization.
1) Consider if your organization has a future strategic value for having the org capability in house (is this datacenter simply a necessary evil or does your organization deploy new technologies to new locations for any part of its business)?
If so, you may be outsourcing a core part of your organization's capability (consider it). If not (e.g. these are all simply HR and other general business systems [ or services which do not depend on anything physical], you should proceed to evaluating costs
2) By the thought that you would outsource the majority of jobs means that you probably shouldn't have had the employees in the first place (in the Fortune 50 I know, we mostly used contractors with a quarter dozen employees). Hence I would evaluate if there are other business units which might benefit from their skill (power related, analyst related et al) and offer training per the previous notes.
A good decision to be a part of and my hope is that your answers here inspire the right questions (to allow for the right transition plan for your organization and company as a whole).
1) Consider if your organization has a future strategic value for having the org capability in house (is this datacenter simply a necessary evil or does your organization deploy new technologies to new locations for any part of its business)?
If so, you may be outsourcing a core part of your organization's capability (consider it). If not (e.g. these are all simply HR and other general business systems [ or services which do not depend on anything physical], you should proceed to evaluating costs
2) By the thought that you would outsource the majority of jobs means that you probably shouldn't have had the employees in the first place (in the Fortune 50 I know, we mostly used contractors with a quarter dozen employees). Hence I would evaluate if there are other business units which might benefit from their skill (power related, analyst related et al) and offer training per the previous notes.
A good decision to be a part of and my hope is that your answers here inspire the right questions (to allow for the right transition plan for your organization and company as a whole).
My rule of thumb: invest 30% of the cost savings into training your people on skills you need now but also with a view to future plans. Invest in preparing your people for new ways like DevOps, microservices. With their experience and knowledge they can save you even more looking forward.
And even if you don’t need them all, by training them you give them a better chance when finding new jobs. A small investment can go a long way. Also helps your companies reputation when people don’t simply get removed.
And even if you don’t need them all, by training them you give them a better chance when finding new jobs. A small investment can go a long way. Also helps your companies reputation when people don’t simply get removed.
When Netflix moved from the datacenter to the cloud, there were a whole bunch of really good people with skills we didn't need anymore -- sysadmins, DBAs, datacenter techs, etc.
Netflix went to them and said, "do you want to learn some new skills that apply to our new cloud based infra?" Some said no, and were given big severance checks and glowing recommendations. Others said yes, and were retrained.
And some were retrained but it didn't work out, it turned out they weren't interested in the new work, or just weren't needed, and so they too were given big severance checks and glowing recommendations, and went to look for new jobs using both their old and new skills.
It's nice that you're worried about them, but the best thing you can do for them is offer them the chance to learn new skills, either through new jobs or paying for them to learn new skills.
Netflix went to them and said, "do you want to learn some new skills that apply to our new cloud based infra?" Some said no, and were given big severance checks and glowing recommendations. Others said yes, and were retrained.
And some were retrained but it didn't work out, it turned out they weren't interested in the new work, or just weren't needed, and so they too were given big severance checks and glowing recommendations, and went to look for new jobs using both their old and new skills.
It's nice that you're worried about them, but the best thing you can do for them is offer them the chance to learn new skills, either through new jobs or paying for them to learn new skills.
This seems best - offer them the choice.
Don't go "learn to code cloud formation in 30 days or lose your job with no severance".
Don't go "learn to code cloud formation in 30 days or lose your job with no severance".
Progress
If you don't do the move someone else will eventually given the monetary incentive. This isn't something that can be stopped.
Better that it happens under the watch of someone that clearly cares
Better that it happens under the watch of someone that clearly cares
> theyre doing stuff like manual config, and future state is everythign automated as much as possible in the cloud and shutter the data centers
Ok, who let the C'level out of his cage? Quick, convene a meeting about meetings and distract him with new synergistic products that mesh the work output of the teams into a new daily reporting product with shiny graphs for him to make informed decisions about, preferably by proposing an "cloud only" data scientist.
But seriously. This comment alone shows this person doesn't have a clue wtf they are talking about. If your guys were so focused on manual config that was a management problem, and isn't an inherent fact about the current state of data centers. So you decided to move data centers, you just moved the jobs somewhere else, to the new provider (Let me guess, amazon, microsoft, or google...)
Say it with me: "There is no such thing as "the cloud"."; Say it again: "There is no such thing as "the cloud"."
Your shit still runs on servers, that at the end of the day someone manages the systems of. So they manage the systems at a lower level that is transparent to you as the modern datacenter is able to hotload blocks, devices, vm's, containers, etc across a swath of machines, clusters, etc.
At the end of the day, you made a business decision that decided managing your own on prem wasn't for you. For some companies, that can be the right strategic move, but as the salty ol bofh who has seen it happen, it's not all puppy dogs and rainbows and there are many pros and many cons to such a move... with "the boss" usually completely underestimating the cons.
Next up: the boss asks how to ease the transition of the low level programmers because he's hiring a new wave of nodejs/meanstack devs because "there's so many of them he can pay them less than the greybeards" and "they are young and know all the cool new devops agile cloud cycle methodologies" and "c++ is dying".
Whatever f50 this is I'd longshort the fuck out of it. I bet it's in a downslide for other reasons and IT is the easy dept to squeeze politically while ignoring the root causal issues that are probably management related.
tldr; Data centers aren't going anywhere, your shit still runs on servers, people still design build and rack boxen, design, fine tune, and diagnose networks, munge and wrangle data, etc, etc, and the skills they have are still in high demand the world over. Disagree? Tell me how many data centers as a percentage that have fully automated robotized racking?
Ok, who let the C'level out of his cage? Quick, convene a meeting about meetings and distract him with new synergistic products that mesh the work output of the teams into a new daily reporting product with shiny graphs for him to make informed decisions about, preferably by proposing an "cloud only" data scientist.
But seriously. This comment alone shows this person doesn't have a clue wtf they are talking about. If your guys were so focused on manual config that was a management problem, and isn't an inherent fact about the current state of data centers. So you decided to move data centers, you just moved the jobs somewhere else, to the new provider (Let me guess, amazon, microsoft, or google...)
Say it with me: "There is no such thing as "the cloud"."; Say it again: "There is no such thing as "the cloud"."
Your shit still runs on servers, that at the end of the day someone manages the systems of. So they manage the systems at a lower level that is transparent to you as the modern datacenter is able to hotload blocks, devices, vm's, containers, etc across a swath of machines, clusters, etc.
At the end of the day, you made a business decision that decided managing your own on prem wasn't for you. For some companies, that can be the right strategic move, but as the salty ol bofh who has seen it happen, it's not all puppy dogs and rainbows and there are many pros and many cons to such a move... with "the boss" usually completely underestimating the cons.
Next up: the boss asks how to ease the transition of the low level programmers because he's hiring a new wave of nodejs/meanstack devs because "there's so many of them he can pay them less than the greybeards" and "they are young and know all the cool new devops agile cloud cycle methodologies" and "c++ is dying".
Whatever f50 this is I'd longshort the fuck out of it. I bet it's in a downslide for other reasons and IT is the easy dept to squeeze politically while ignoring the root causal issues that are probably management related.
tldr; Data centers aren't going anywhere, your shit still runs on servers, people still design build and rack boxen, design, fine tune, and diagnose networks, munge and wrangle data, etc, etc, and the skills they have are still in high demand the world over. Disagree? Tell me how many data centers as a percentage that have fully automated robotized racking?
You can't post attacks like this to HN, and abusing the site this way will eventually get your main account banned as well.
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
There’s actually a huge difference in performing their job in a DC/NOC versus on Azure or other cloud providers.
You really have this completely wrong. I am a systems/network engineer and am newer to the business side. Asking here to get some upsight into the people side. Thank you for pointing out that things run on servers, that was news to me.
Sarcasm is rarely well-received here. Also, I think anyone reading any of the rest of the thread would see that you do have an understand of the issues, so no need to defend yourself.
Conversely, the parent commenter uses hyperbole to the point of appearing not to have that understanding.
For example, even without "fully robiticized" datacenters, there are remarkable economies of scale before that, including ordering servers by the rack (or maybe even row at a time), pre-assembled by a vendor.
Things really can be different, even though they can be the same.. even both at the same time!
Conversely, the parent commenter uses hyperbole to the point of appearing not to have that understanding.
For example, even without "fully robiticized" datacenters, there are remarkable economies of scale before that, including ordering servers by the rack (or maybe even row at a time), pre-assembled by a vendor.
Things really can be different, even though they can be the same.. even both at the same time!
I have a chance to basically migrate 90% of my F50's data centers to commercial providers. The cost savings are awesome, but what about the people currently doing legacy stuff? theyre doing stuff like manual config, and future state is everythign automated as much as possible in the cloud and shutter the data centers.
I think the harsh truth is that many wont have jobs post move, but maybe I'm being cynical. maybe theres a chance to retrain the employees, but even doing that automation generally reduces the workforce.
whats your take? what do I tell current employees? how many can I realistically be able to save from being jobless?